Shrujani D

14 years of IT Experience in analysis, design, development, implementation, administration and troubleshooting of Data warehouse applications. Hands-on Experience in Service Oriented Architecture (SOA), Event Driven Architecture, Distributed Application Architecture and Software as Service (SAS).

Implemented Azure Key

Vault to securely store and manage database credentials, API keys, access tokens, and sensitive configuration values across enterprise data engineering pipelines.

Integrated Azure Key

Vault with Azure Data Factory and Azure Databricks to retrieve secrets at runtime and eliminate hardcoded credentials in ETL/ELT workflows. Designed and developed scalable batch and streaming data pipelines using Apache Spark, PySpark, Scala, and Kafka to process high-volume structured and semi-structured data.

Implemented Spark Structured

Streaming applications to consume Kafka events, perform transformations, and deliver reliable downstream datasets with low latency. Designed and developed scalable data pipelines to support AI/ML workloads, enabling ingestion, transformation, and feature preparation from structured and semi-structured data sources. Designed and developed large-scale architectures using Python programming, Lambda functions Numphy and Pandas for enterprise-level data integration and data warehousing projects Developed and optimized SQL queries and stored procedures for data extraction and data loading using Apache Airflow, Kubernetes, Docker.

Extensive experience building scalable batch and streaming data pipelines using Databricks, Delta Lake, Apache Spark, Python, SQL, Apache Airflow, and DBT. Experienced in implementing CI/CD pipelines using GitHub Actions, Azure DevOps, and Git-based deployment strategies for automated testing and deployment.

Integrated Azure Key

Vault with Azure Data Factory and Azure Databricks to retrieve secrets securely at runtime and eliminate hardcoded credentials.

Configured Managed

Identity and Azure RBAC to enable secure, passwordless access to Key Vault while enforcing least-privilege permissions.

Implemented Azure Managed

Identity to enable secure, passwordless authentication between Azure Data Factory and Azure SQL Database.

Configured Azure Managed

Identity with Microsoft Entra ID to establish secure connectivity between Azure Databricks, Azure SQL Database, and other Azure data services.

Data Engineer - Walmart

(2025-04)

• Led migration of enterprise data pipelines from Automic Workload Automation to Apache Airflow, building scalable DAGs in Python with dynamic task generation, dependency resolution, and reusable components.
• Architected and developed distributed data processing pipelines using Scala (Apache Spark) on Azure Databricks, enabling efficient transformation and aggregation of large-scale marketplace payment datasets.
• Designed advanced Airflow orchestration patterns including backfills, SLA monitoring, retries, alerting, and parameterized DAGs, significantly improving pipeline reliability and observability.
• Optimized Spark jobs through partitioning, repartitioning, caching, join optimization, and shuffle tuning to improve processing performance.
• Developed reusable Python and Scala modules for data ingestion, transformation, validation, and error handling across multiple data engineering workflows.
• Configured Azure Key Vault with Managed Identity and Azure RBAC to enable secure, passwordless authentication and enforce least-privilege access controls.
• Utilized Azure Key Vault to manage credentials for Azure SQL Database, Azure Storage, and external API integrations, ensuring secure connectivity across data platforms.
• Leveraged Generative AI (GenAI) tools to automate SQL generation, data mapping documentation, ETL logic validation, and metadata extraction, improving development productivity.
• Created database users for Azure Managed Identity in Azure SQL Database and assigned appropriate permissions for secure data access.
• Integrated Azure Managed Identity into ETL/ELT pipelines to eliminate hardcoded database credentials and improve authentication security.
• Designed and implemented scalable enterprise data platform components using Azure Databricks, Delta Lake, Apache Airflow, and Python.
• Implemented CI/CD pipelines using GitHub Actions and Azure DevOps for automated deployment of Databricks notebooks, Airflow DAGs, and DBT projects.
• Developed high-performance ETL pipelines using Python and SQL within Azure Databricks, processing high-volume financial transaction data with strict accuracy and latency requirements.
• Designed and optimized complex SQL queries and data models in Teradata and Azure Databricks (Delta Lake), improving query efficiency and supporting downstream reporting use cases.
• Implemented scalable data solutions across cloud platforms, including Azure Databricks (ADLS, Delta Lake) and Google Cloud Platform (BigQuery, Cloud Storage), for data warehousing and processing.
• Performed end-to-end data discovery, schema analysis, and lineage tracking using Azure Databricks Unity Catalog and metadata tools to support audit, reconciliation, and compliance requirements.

Data Engineer - Cisco//Tech Mahindra San Jose - FTE - San Jose

(2019-05 - 2024-11)

• Experience with data processing and analytics using Apache Spark, Apache Flink, and other tools.
• Designed and implemented large-scale data integration architectures using Kubernetes, Docker, Apache Airflow for building pipelines with various data sources such as databases (e.g. Oracle, MySQL), cloud storage (e.g. AWS S3, Azure Blob), and APIs (e.g. RESTAPI, SOAP)
• Created Spark data frames using Spark SQL, Hadoop, Hive and prepared data for data analytics by storing it in AWS S3, Kubernetes, Docker
• Implemented secret rotation, expiration policies, and access permissions in Azure Key Vault to strengthen credential security and support compliance requirements.
• Integrated Azure Key Vault with Azure Monitor and Log Analytics to monitor access activities, troubleshoot authentication failures, and support production security investigations.
• Resolved performance challenges such as data skew, shuffle overhead, small files, and resource utilization while processing 1B+ records at scale.
• Implemented event-driven data ingestion using Kafka, Pub/Sub, or Azure Event Hubs to support near-real-time processing and downstream consumption.
• Developed Spark processing workflows on Dataproc and Azure Databricks for batch ETL, data cleansing, enrichment, and analytical data preparation.
• Implemented reliable production pipelines for 1B+ records with Airflow/Databricks Workflows, data-quality checks, monitoring, error handling, and automated retries.
• Implemented data governance and data quality solutions using Informatica's data governance and data quality tools, including data lineage, data traceability, and data auditing.
• Wrote unit tests for individual components and functions using frameworks like JUnit, PyUnit, and NUnit.
• Implemented metadata-driven ETL frameworks supporting schema evolution and automated onboarding and Built secure REST API integrations with enterprise applications.
• Developed and deployed a cloud-native application using AWS Kubernetes,Docker applications
• Designed and implemented a data lake using Apache Iceberg for a large-scale analytics platform.
• Managed Iceberg tables and built CI/CD pipelines using Python, Scala, Apache Airflow.
• Created Spark data frames using Spark SQL, Hadoop, Hive and prepared data for data analytics by storing it in AWS S3, Kubernetes, Docker.
• Writing complex scripts in Snowflake, GCP cloud data warehouse to business analysis and reporting.