Skip to main content

Azure Data Engineer

Technology
Indore, India4 days ago
Full time

Job description

Role: Senior Data Engineer

Years of Experience: 5-8 Years

Location: Ahmedabad, Pune, Indore

Notice Period: Immediate to 30 Days Max

Must Skill: Python, SQL, Azure Databricks, ADF, ETL, Pyspark,

Role Summary

Build and operate scalable, reliable data pipelines on Azure. Develop batch and streaming ingestion, transform data using Databricks (PySpark/SQL), enforce data quality, and publish curated datasets for analytics and ML.

Key Responsibilities

  • Design, build, and maintain ETL/ELT pipelines in Azure Data Factory and Databricks across Bronze Silver Gold layers.
  • Implement Delta Lake best practices (ACID, schema evolution, MERGE/upsert, time travel, Z-ORDER).
  • Write performant PySpark and SQL; tune jobs (partitioning, caching, join strategies).
  • Create reusable components; manage code in Git; contribute to CI/CD pipelines (Azure DevOps/GitHub Actions/Jenkins).
  • Apply data quality checks (Great Expectations or custom validations), monitoring, drift detection, and alerting.
  • Model data for analytics (star/dimensional); publish to Synapse/Snowflake/SQL Server.
  • Uphold governance and security (Purview/Unity Catalog lineage, RBAC, tagging, encryption, PII handling).
  • Author documentation/runbooks; support production incidents and root-cause analysis; suggest cost/performance improvements.

Must-Have (Mandatory)

  • Data Engineering & Pipelines
  • Hands-on experience building production pipelines with Azure Data Factory and Databricks (PySpark/SQL).
  • Working knowledge of Medallion Architecture and Delta Lake (schema evolution, ACID).
  • Programming & Automation
  • Strong Python (pandas/PySpark) and SQL.
  • Practical Git workflow; experience integrating pipelines into CI/CD (Azure DevOps/GitHub Actions/Jenkins).
  • Familiarity with packaging reusable code (e.g., Python wheels) and configuration-driven jobs.
  • Data Modeling & Warehousing
  • Solid grasp of dimensional modeling/star schemas; experience with Synapse, Snowflake, or SQL Server.
  • Data Quality & Monitoring
  • Implemented validation checks and alerts; exposure to drift detection and pipeline observability.
  • Cloud Platforms (Azure preferred)
  • ADLS Gen2, Key Vault, Databricks, ADF basics (linked services, datasets, triggers), environment promotion.
  • Data Governance & Security
  • Experience with metadata/lineage (Purview/Unity Catalog), RBAC, secrets management, and secure data sharing.
  • Understanding of PII/PHI handling and encryption at rest/in transit.
  • Collaboration
  • Clear communication, documentation discipline, Agile ways of working, and code reviews.

Good-to-Have

  • Databricks Asset Bundles (DAB) for environment promotion/infra-as-code style deployments.
  • Streaming/real-time: Kafka/Event Hubs; CDC tools (e.g., Debezium, ADF/Synapse CDC).
  • MLOps touchpoints: MLflow tracking/registry, feature tables, basic model-inference pipelines.
  • Power BI exposure for publishing curated tables and building operational KPIs.
  • DataOps practices: automated testing, data contracts, lineage-aware deployments, cost optimization on Azure.
  • Certifications: Microsoft Certified Azure Data Engineer Associate (DP-203) or equivalent.

Qualifications

  • 46 years of professional experience in data engineering (or equivalent project depth).
  • Bachelor’s/Master’s in CS/IT/Engineering or related field (or equivalent practical experience).
Role: Data Engineer, Industry Type: Telecom / ISP, Department: Data Science & Analytics, Employment Type: Full Time, Permanent Role Category: Data Science & Machine Learning Education UG: B.Tech / B.E. in Any Specialization Key Skills Skills highlighted with ‘‘ are preferred keyskills PythonAzure Data FactoryData ModelingSQLAzure Databricks Data Engineering
Keywords
PythonAzure Data FactoryData ModelingSQLAzure DatabricksData Engineering

Interested in this role?