Dipanjan Das

Performance-driven and technically robust Data Engineer with nearly 4 years of expertise designing, implementing, and optimizing enterprise-grade, cloud-native data platforms across the CPG and Retail sectors utilizing Microsoft Azure and Databricks. Proven track record architecting robust Data Lakehouse systems leveraging the Medallion Architecture to deliver scalable, production-ready analytical data products. Expert in developing metadata-driven ingestion frameworks, engineering optimized PySpark pipelines, enforcing data governance via Unity Catalog, and deploying automated CI/CD workflows via Azure DevOps to maximize system observability and pipeline throughput.

Data Engineer - Tata Consultancy Services - Kolkata, India

(2022-08)

Client: Diageo Analytics, Retail

• Designed and developed a scalable, metadata-driven framework using Azure Data Factory and ADLS Gen2 to automate batch and streaming data ingestion from diverse enterprise sources, improving pipeline reliability by 30%.
• Designed and delivered an end-to-end Data Lakehouse solution using Databricks Medallion Architecture (Bronze, Silver, Gold), orchestrating automated processing across 10+ fact and dimension tables.
• Engineered high-performance PySpark jobs in Databricks to implement SCD Type 1 and Type 2 (Slowly Changing Dimensions) processing utilizing Delta Lake merge operations, ensuring historic data tracking and transactional data integrity across dimensions.
• Built high-performance distributed data workflows utilizing PySpark, Databricks Workflows, and Unity Catalog for secure governance, leveraging broadcast joins, partitioning strategies, and the Catalyst Optimizer to maximize throughput.
• Developed complex analytical transformation modules using Advanced SQL, CTEs, and Window Functions across Dimensional and 3NF models, improving downstream query performance for data scientists and business consultants by 20%+.
• Spearheaded development of 20+ enterprise Power BI dashboards and data models for global stakeholders, converting business KPIs into structured DAX measures and optimized Power Query models.

Data Analytics and Integration Intern - Cognizant Technology Solutions - Kolkata, India

(2022-02 - 2022-08)

• Developed ETL integration components for the 'Sales Insights ETL Pipeline' proof-of-concept, processing files using Python and SQL to extract, transform, and load clean structured datasets into target warehouse schemas.
• Collaborated closely with cross-functional engineering and application teams to map out technical business requirements and debug data pipelines, refining overall data ingestion quality.

Bachelor of Technology - Electronics and Communication Engineering - Academy of Technology (2018-07 - 2022-07) Hooghly, India