Karan Varade

Results-driven Data Engineer with proven experience in designing, developing, and optimizing data pipelines and analytics solutions using modern cloud and data stack technologies. Strong expertise in SQL, AWS, Databricks, Snowflake, and DBT for building scalable, high-performance data workflows. Adept at integrating data across multi-cloud environments and ensuring high data quality, governance, and performance tuning for analytical use cases.

DATA ENGINEER at ZS ASSOCIATES (2024-06 – Present)

• Migrated over 5 TB of data from Hive to Snowflake, resulting in a 35% reduction in storage costs and improving query performance by up to 60%, significantly accelerating data access for analytics teams.
• Designed and implemented end-to-end ETL pipelines using Databricks, AWS (EMR, EC2), and Azure Repos, enabling real-time data processing for a live executive dashboard.
• Cleaned and transformed 100M+ records using advanced SQL techniques, improving data quality and ensuring datasets were analysis-ready, contributing to more accurate and actionable insights.
• Leveraged AWS EMR and EC2 instances for scalable and distributed data processing, optimizing large-scale data transformations and calculations.
• Implemented data transformation and modeling workflows in DBT, creating reusable models and automating data validation, which improved pipeline reliability and reduced manual rework by 30.
• Worked cross-functionally with stakeholders to define requirements for data processing, analysis, and reporting, ensuring alignment with business goals.
• Utilized EMR and Hive for additional data processing and analytics tasks, enhancing overall data capabilities.

DATA ENGINEER at WIPRO (2021-03 – 2023-10)

• Worked on a project implementing real-time data synchronization using AWS DMS, RDS, Glue, PySpark and Lambda.
• Collaborated with senior engineers to design and build an end-to-end data pipeline using S3 triggers, Lambda functions and Glue workflows, ensuring efficient data ingestion and transformation.
• Developed PySpark scripts to perform data transformations, cleansing, and aggregation for CDC processes, enhancing data quality and optimizing performance.
• Built and optimized Delta Lake tables in Databricks to support ACID transactions, incremental processing, and improved data reliability.
• Configured and monitored AWS DMS tasks to capture and replicate incremental changes from source databases to target databases.
• Assisted in implementing event-driven data processing by leveraging S3 event notifications to trigger Lambda functions, enabling near real-time data processing and Glue job execution.
• Conducted performance tuning and optimization of ETL processes for improved efficiency and reduced latency.
• Maintained project documentation, including data pipelines, architectural designs, and implementation details, promoting knowledge sharing and team collaboration.

B.Tech in Electronics and Telecommunication Engineering – DR. BABASAHEB AMBEDKAR TECHNOLOGICAL UNIVERSITY, LONERE (2020-10)