Chaithanya Pilli

Data Engineer (ETL) with 4 years of experience building scalable data pipelines on Azure. Improved data processing performance by optimizing PySpark jobs. Experienced in migrating on-premise systems to ADLS Gen2 and developing data solutions in the banking domain.

Proficient in PySpark, Databricks, Azure Data Factory, and SQL, with a strong focus on data quality. Experienced in data validation, reconciliation, workflow orchestration, and implementing data quality frameworks to ensure reliable and consistent data pipelines across enterprise platforms and Azure Data Lake Storage (ADLS Gen 2).

Data Engineer at Tata Consultancy Services (2022-05 – Present)

Banking Domain | Azure Cloud Platform | Deposits, Point of Sale, Payments Services

• Worked on end-to-end ADF pipelines migrating banking data from on-premise RDBMS, flat files, CSV, JSON, and API sources into ADLS Gen2, standardizing all datasets into Delta Snappy Parquet format for optimized downstream consumption.
• Built watermark-based incremental pipelines to extract and load new records from RDBMS into ADLS Gen2, and worked on current and historical views built on top of base tables to support latest-state and full-history data access patterns.
• Developed stored procedures for incremental data extraction from RDBMS, reducing redundant processing and ensuring consistent data delivery to downstream consumers.
• Worked on Deposits and Point-of-Sale (POS) transaction ingestion workflows.
• Worked on migration of peer-to-peer payment ETL consumption service workflows involving POS and Cards transactions and validation.
• Orchestrated metadata-driven ADF pipeline execution via Azure Functions, reading pipeline configuration from Azure Storage and exposing HTTP endpoints to dynamically trigger ingestion and consumption workflows.
• Developed PySpark and SQL-based transformation logic for large-scale batch processing of banking transaction data, preparing datasets for downstream analytical consumption.
• Developed data cleansing and schema alignment logic to resolve format mismatches, encoding inconsistencies (French characters), whitespace anomalies, and structural variations across heterogeneous banking source systems into standardized ADLS-ready datasets.
• Implemented a pipeline-level data quality framework covering null validation, duplicate detection, schema consistency checks, and source-to-target reconciliation - ensuring completeness and integrity of critical financial records across the pipelines.
• Worked on ADF monitoring and alerting mechanisms to track SLA compliance for banking data loads.
• Implemented Role-Based Access Control (RBAC) with Restricted, Super, and Elite user tiers, enforcing column-level security on sensitive financial datasets.
• Collaborated with cross-functional teams in Agile sprints (standups, planning, retrospectives) to deliver platform milestones on schedule.

Bachelor of Technology in Information Technology – CMR Technical Campus (2021)

– Sri Chaitanya Junior College (2017)

– Victory High School (2015)