Skip to main content

Title: Azure Databricks Developer

Technology
Birlasoft
2 months ago
Full timeOn-site

Job description

Area(s) of ResponsibilityKey Responsibilities

  1. Data Engineering Execution

    • Build and maintain ingestion frameworks (ADF / Databricks / Spark)
    • Implement Bronze -> Silver transformations aligned to architecture
    • Architect data quality checks, schema validation, and contract rules
    • Build and optimize robust, high-throughput ELT/ETL pipelines, enabling ingestion, transformation, and curation of structured, semi-structured, and unstructured data.
    • Integrate data from multiple on-premise and cloud-based systems, APIs, and third-party sources.
    • Implement complex transformations using PySpark, ensuring performance efficiency and code modularity.
    • Build orchestration workflows in ADF, including pipelines, triggers, linked services, integration runtimes, and parameterized datasets.
    • Familiar with using Databricks Genie.
    • Build: Dimensional models (star/snowflake schemas), Fact tables, dimensions, surrogate keys, SCD handling
    • Translate Silver datasets into: Analytics-ready models, Consistent KPI definitions, and business logic
    • Ensure: Consistency across domains (common dimensions, conformed models), Reusability and scalability of models
  2. Databricks & PySpark Engineering
    • Develop scalable transformation scripts using PySpark on Databricks, applying advanced optimizations like caching, partitioning, and Delta Lake capabilities.
    • Implement Delta Lake features-ACID transactions, schema enforcement, schema evolution, and time travel-across the data lifecycle.
    • Perform performance tuning, handling bottlenecks related to cluster configuration, shuffle operations, joins, and parallelization.
    • Collaborate with platform teams to manage Databricks clusters, jobs, notebooks, and CI/CD integrations.
Locations
  • Country/Region: IN
  • Requisition ID: 38967
  • Work Model:
  • Position Type:
  • Salary Range:
  • Location: INDIA - PUNE - BIRLASOFT OFFICE - HINJAWADI
Keywords
OrchestrationSCHEMAApache SparkBusiness logicScalabilityPartitionCluster analysisAcid ProCI/CDReusabilityData managementData clusterDisk partitioning

Interested in this role?