Data Bricks Engineer

Job Description –Databricks Platform Agentic Build Engineering Lead

Bulgaria - Remote

Role snapshot

We are looking for a hands-on Databricks engineering lead who has already built and run AI agents in production, to lead the build of agentic operations on Databricks on AWS Lakehouse platform.

About the role

Client is transforming it’s Databricks on AWS Lakehouse platform operations support from manual ticket handling to governed, agent-driven operations.

You will lead the engineering of that Agents Build journey on Databricks, from AI-assisted entry points through to production agents that monitor, remediate and fulfil requests.

You will set the technical direction, build the core components yourself, and guide team of platform engineers.

The scope has three layers:

• AI-assisted entry points: conversational intake of incidents and service requests via ServiceNow forms, a natural-language data catalogue and access-request app, and an Ops Control Tower giving humans evidence, approval and a kill switch on every agent action.
• AI platform foundations: a governed SOP and skill repository (versioned, changed via pull request), a central MCP server and tool registry with per-SOP allow-lists, and cost controls tracking token cost per resolved ticket (EUR/ticket).
• Agentic Ops build: a Platform Monitoring & Remediation Agent (detect, diagnose, remediate on unified observability) and a Platform Service Request Fulfilment Agent (end-to-end request handling following approved SOPs).

Key responsibilities

Technical leadership

• Own the target architecture for agentic operations on the Databricks Lakehouse and drive it from design to production.
• Lead and mentor platform engineers; set coding, review, testing and release standards.
• Work with cross functional teams – Data Product teams, Client Inhouse Architects, Security and FinOps to agree scope, guardrails and priorities.

AI-assisted entry points

• Build natural-language intake that turns user requests into structured ServiceNow incidents and service requests.
• Build a conversational data-discovery and access-request experience on top of Unity Catalog metadata.
• Deliver an Ops Control Tower (e.g. Databricks Apps) showing per-task evidence: root cause, checks and actions taken, with approve / stop controls and platform health and cost side by side.

AI platform foundations

• Design a governed SOP and skill library: templates, versioning, role-based edit rights, changes via pull request and owner approval.
• Build and run a central MCP registry: approved MCP servers and tools for Databricks and source systems, with tool allow-lists per SOP.
• Implement token and compute cost tracking per agent and per resolved ticket, with budgets and alerts.

Agentic Ops build

• Build a Monitoring & Remediation Agent on unified observability (Databricks system tables, AWS infrastructure, Power BI capacity): detect, diagnose, remediate.
• Implement tiered autonomy: Tier 1 autonomous fix, Tier 2 AI-assisted, Tier 3 human-led, with every action through registered MCP tools and visible in the Control Tower.
• Build a Service Request Fulfilment Agent that picks up requests from the intake channels and fulfils them by following approved SOPs, with human approval where required.
• Set up agent evaluation, tracing, safety guardrails and incident runbooks; measure resolution rate, MTTR and cost per ticket.

Must-have skills and experience

• 10 years in data engineering or data platforms, with 3 years as tech lead or lead engineer.
• 5 years hands-on Databricks: Unity Catalog, Workflows/Jobs, Delta Lake, cluster and SQL warehouse administration, system tables, Databricks Asset Bundles, REST API and SDK.
• Proven delivery of at least one LLM agent or agentic workflow in production (not only a proof of concept), with tool calling and human-in-the-loop approval.
• Hands-on with agent frameworks and tooling: Mosaic AI Agent Framework, LangChain/LangGraph, or similar; Model Context Protocol (MCP) servers and clients.
• Agent quality and safety: evaluation sets, tracing (e.g. MLflow Tracing), guardrails, prompt and tool permissioning.
• Strong Python and SQL; Git-based workflows, pull-request reviews, CI/CD (GitHub Actions, Azure DevOps or GitLab).
• AWS infrastructure for Databricks: IAM, networking, S3, CloudWatch; Infrastructure as Code with Terraform.
• Observability and platform operations: monitoring, alerting, incident and root-cause analysis, runbooks/SOPs, SRE practices.
• Integration with ITSM, ideally ServiceNow (incidents, service requests, APIs).
• Fluent English and confidence presenting to senior stakeholders.

Nice-to-have skills

• Databricks certifications: Data Engineer Professional, Generative AI Engineer Associate, or Platform Administrator.
• Databricks Apps, Mosaic AI Model Serving, Vector Search, Genie or AI/BI.
• LLM FinOps: token-cost attribution, model routing, caching, budgets per use case.
• Power BI / Fabric capacity monitoring and administration.
• Experience building SOP-driven automation or runbook automation (e.g. AIOps, auto-remediation).
• Data governance and access-request workflows on Unity Catalog.