Asrah - Senior Data Engineer - Python/Azure Databricks

Role : Senior Data Engineer - Azure, Databricks & GenAI

About the Role :

We are looking for a Senior Data Engineer to design, build, and operate an enterprise-scale data platform on Microsoft Azure and Databricks, with strong hands-on experience in Generative AI, LLMs, and RAG-based applications.

The role will own the end-to-end lifecycle of data pipelines and AI-powered applications, working closely with engineering, product, and business teams to build scalable, reliable, and production-ready data and AI solutions.

Key Responsibilities :

• Design, develop, and maintain scalable data pipelines and data processing solutions using Azure and Databricks.
• Build and optimize data engineering workflows using PySpark and Python.
• Develop and manage enterprise data platforms, ensuring scalability, reliability, security, and performance.
• Design and implement ETL/ELT pipelines for structured and unstructured data.
• Build GenAI/LLM-powered applications, including RAG (Retrieval-Augmented Generation) solutions.
• Work with LLMs, embeddings, vector databases, prompt engineering, and retrieval pipelines.
• Integrate LLM capabilities with enterprise data and business applications.
• Develop, deploy, monitor, and optimize AI-powered applications in production.
• Implement data quality, validation, monitoring, and governance practices across pipelines.
• Optimize Databricks workloads and Spark jobs for performance and cost efficiency.
• Collaborate with cross-functional teams to translate business requirements into scalable data and AI solutions.
• Troubleshoot production issues and ensure high availability and reliability of data and AI workloads.
• Stay updated with emerging technologies across Data Engineering, GenAI, LLMs, and Azure.

Required Skills &

Experience :

• Strong hands-on experience in Data Engineering and building enterprise-scale data platforms.
• Strong proficiency in Python and PySpark.
• Hands-on experience with Azure data services and Databricks.
• Experience designing and developing ETL/ELT data pipelines.
• Strong understanding of data engineering concepts, distributed computing, and data architecture.
• Hands-on experience with Generative AI, LLMs, and RAG architectures.
• Experience working with LLM application development, embeddings, vector search, and retrieval pipelines.
• Understanding of data modeling, data quality, and data governance.
• Experience with cloud-based data platforms and production-grade engineering practices.
• Strong problem-solving and analytical skills.
• Ability to work independently and collaborate effectively with technical and business stakeholders.

Preferred Skills :

• Experience with Azure services such as Azure Data Factory, Azure Data Lake, Azure OpenAI, and Azure AI services.
• Experience with MLflow and Databricks-based AI/ML workflows.
• Experience with vector databases and semantic search.
• Knowledge of LLM frameworks and orchestration tools.
• Experience deploying AI applications using APIs and microservices.
• Understanding of CI/CD, DevOps, monitoring, and cloud security.

Education :

• B.Tech / B.E. in Computer Science, Information Technology, Engineering, or a related field.