Job Summary
To conceptualize, design and deliver product / sustenance delivery through the team as per defined scope,by adopting industry-leading practices, thus contributing significantly to the advancement and success of the project.
Key Responsibilities
Position Summary Design, build, and operationalize enterprise-grade AI/ML and agentic AI solutions on the Databricks Lakehouse platform. Drive end-to-end lifecycle ownership—from model development and RAG pipelines to deployment, monitoring, and governance assuring scalability, security, and business value realization across Data & Analytics use cases. Develop end-to-end agentic workflows using open-source LLMs hosted on the Databricks platform. Build the user interface using Databricks Apps or similar tool, ensuring the entire ecosystem adheres to strict enterprise security standards and provides highly accurate, context-aware responses. Key Responsibilities Agent Development: Build and orchestrate autonomous AI agents with multi-step reasoning, tool usage, and workflow chaining using frameworks like LangChain, CrewAI, AutoGen, Semantic Kernel, or LlamaIndex. LLM Integration & Optimization: Deploy, fine-tune, and serve open-source LLMs (e.g., Llama 3) using Databricks Model Serving; optimize latency, throughput, and cost. RAG & Knowledge Systems: Design advanced RAG pipelines leveraging vector search, embeddings, semantic ranking, and enterprise data sources (structured + unstructured). Context Engineering: Develop prompt strategies, memory frameworks, and metadata tagging to improve contextual accuracy and response quality. UI & Experience Design: Build intuitive AI-driven applications using Databricks Apps (Streamlit/Dash) or modern web frameworks to enable business consumption. Data Engineering for AI: Build reliable data pipelines (batch & streaming) supporting training, inference, and feature generation using Delta Lake. Security & Governance: Implement enterprise-grade controls using Unity Catalog (row/column-level security, lineage, auditability) aligned with compliance standards. LLM Guardrails & Responsible AI: Implement guardrails (e.g., NeMo Guardrails) for prompt injection prevention, hallucination mitigation, and safe output handling. MLOps & AIOps: Establish CI/CD pipelines for AI models and agents, including versioning, monitoring, drift detection, observability, and incident response. Performance & Cos