Ronit Dutta

Data Engineer with 3+ years of experience, recognized for strong problem-solving, stakeholder communication, and ownership of end-to-end data solutions. Experienced in collaborating with business and leading teams to convert requirements into scalable, high-quality data pipelines. Technically proficient in Azure Data Factory, Databricks, and Snowflake, with hands-on expertise in SQL, Python and Spark.

Solid background in Big Data & Hadoop ecosystem technologies like Hive and Oozie, with a proven focus on performance optimization, data reliability, and production-grade analytics solutions.

Data Engineer at Tata Consultancy Services (2023-08 – Present)

• Designed and built generic, reusable, metadata driven pipelines in Azure Data Factory (ADF) to ingest data from diverse sources including APIs, Azure Blob Storage, ADLS, AWS S3, PostgreSQL, and GCP.
• Orchestrated Change Data Capture (CDC) pipelines in Azure Data Factory (ADF) to capture incremental data loads and support near real-time analytics.
• Managed secure secrets and credentials using Azure Key Vault integrated with Databricks and ADF via managed identities.
• Designed and developed data pipelines in PySpark based on Medallion Architecture (Bronze–Silver–Gold layers) leveraging Unity Catalog tables in Databricks for governance and security.
• Gained strong expertise in Delta Lake architecture, including ACID transactions, time travel, and file optimization techniques for large-scale data processing.
• Optimized Spark jobs through partition tuning, broadcast joins, caching strategies, and adaptive query execution, reducing pipeline runtime by ~20-30%.
• Implemented production-grade orchestration using Databricks Workflows to manage ETL pipelines with task dependencies, automated retries, alerting, and parameterized notebook execution, improving pipeline reliability and operational visibility.
• Developed scalable data ingestion and transformation pipelines across ADLS, S3, APIs, and relational sources, handling data validation and transformation requirements.
• Worked with Serverless SQL Warehouse, Delta Live Tables (DLT), and Unity Catalog for secure, scalable data processing.
• Implemented Snowpark scripts to process semi structured data (JSON, CSV) with multiple encoding parameters, ensuring scalability and reliability in Snowflake.
• Developed and optimized Stored Procedures in Snowflake to handle schema evolution, mismatches, and automated error handling during ingestion.
• Modeled analytical datasets using fact and dimension tables (star schema) to enable efficient querying and reporting.
• Built end-to-end Hadoop batch ingestion pipelines using SFTP, Unix shell scripting, and watermark-based validation to ensure file integrity, data completeness, and quality before processing.
• Developed and orchestrated scalable workflows with Oozie and Spark on HDFS to transform, validate, and load curated datasets into Hive with idempotent incremental processing.
• Automated delta consolidation, watermark tracking, and archival strategies to preserve historical data, optimize storage, and ensure reliable downstream analytics.
• Deployed and version-controlled ADF pipelines and Databricks notebooks using Azure DevOps and GitHub Actions, supporting reliable, auditable releases across environments.

B. Tech in Computer Science & Engineering – Academy of Technology (2019-07 – 2023-07)