Nishath Anjum Shaik

Data Engineer with 4 years of experience in designing, building, and optimizing scalable data pipelines on Azure. Strong expertise in Azure Databricks, PySpark, SQL, and Delta Lake for end-to-end ETL development. Experienced in implementing Medallion Architecture for efficient and structured data processing. Skilled in large-scale data processing and distributed computing using Spark.

Experience in optimizing SQL queries and Spark jobs to improve performance and scalability. Proficient in data modeling, data validation, and ensuring high data quality for analytics and reporting. Familiar with Azure Data Factory (ADF) for pipeline orchestration and scheduling data workflows.

Hands-on experience in workflow orchestration, pipeline automation, and scheduling using Databricks jobs. Strong understanding of data governance and cloud storage (ADLS Gen2). Good analytical, problem-solving, and communication skills with the ability to work independently and collaborate effectively in team environments.

Data Engineer at CISCO SYSTEMS (INDIA) PRIVATE LIMITED (2022-09 – Present)

Working as a Data Engineer for Cisco Systems, contributing to the development and optimization of a large-scale retail analytics platform that integrates data from multiple sources to deliver insights on inventory, sales, and supply chain operations

• Implemented ETL pipelines using PySpark in Azure Databricks to ingest data from MySQL and other RDBMS sources into ADLS
• Imported large volumes of data from multiple sources into ADLS and processed them using Databricks
• Involved in developing and optimizing Spark and SQL queries for data transformations, aggregations, and filtering
• Created and managed Delta tables, including data loading, upserts, and query development
• Applied Spark transformations for data cleansing, deduplication, and data movement across Medallion Architecture layers
• Scheduled and monitored pipelines using Databricks Workflows, handling job failures and reruns
• Optimized Spark jobs by handling data skewness, tuning joins, and improving overall performance
• Used partitioning and caching techniques to enhance processing efficiency
• Built custom transformation logic using Python (loops, conditions, functions) for dynamic data transformations
• Performed data validation, reconciliation, and data profiling to ensure high data quality
• Handled multiple data formats such as CSV, Parquet, and JSON during ingestion and processing
• Debugged and resolved pipeline failures and performance issues in production environments
• Developed reusable notebooks and modular code for efficient pipeline development
• Executed incremental data loading and upsert operations using Delta Lake
• Applied window functions and advanced SQL techniques for complex business transformations
• Ensured data consistency and integrity across Bronze, Silver, and Gold layers
• Established logging and error-handling mechanisms for better pipeline monitoring and traceability

(2022)