Gopikrishna Chigurala

Results-driven Senior Data Engineer with 6 years of hands-on expertise architecting enterprise-scale Lakehouse and cloud data platform solutions. Deep specialization in Azure Databricks, Delta Lake, Azure Data Factory (ADF), PySpark, and Apache Airflow, with a strong track record of delivering metadata-driven, highly scalable ELT/ETL pipelines that reduce runtime significantly. Experienced across pharma commercial analytics and Pharma MDM/data convergence (HCP/HCO master data, standardization, data quality, hierarchies) for global Fortune 500 clients (J&J, Novartis, Unilever).

Expanding into Google Cloud Platform (BigQuery, Dataflow, Cloud Composer, Dataproc, Pub/Sub, Cloud Storage), applying proven Lakehouse, data warehousing, Airflow orchestration, and PySpark expertise to GCP-native data architectures.

Exploring

Gen AI and Prompt Engineering for next-generation data workflows.

Product Lead & Senior Data Engineer at Axtria India Pvt. Ltd. | Client: Johnson & Johnson (2024-05 – Present)

• Architected end-to-end Lakehouse data platform on Azure Databricks for the Fusion Forecast Data initiative, processing multi-terabyte pharma commercial datasets across Bronze → Silver → Gold layers using Delta Lake.
• Designed and productionised a metadata-driven ingestion framework in PySpark that dynamically configures source-to-target mappings, eliminating 80%+ of boilerplate pipeline code and enabling new source onboarding in hours instead of days.
• Supported Pharma MDM/data convergence by integrating and standardizing HCP/HCO and related master/reference data, applying data-quality validation and consistent downstream structures for commercial analytics.
• Built Databricks Autoloader (cloudFiles) pipelines for incremental, schema-evolving file ingestion from ADLS Gen2 (object-storage pattern equivalent to GCS), supporting near-real-time streaming and batch modes simultaneously.
• Implemented Delta Live Tables (DLT) for declarative data transformation with built-in DQ expectations and automatic lineage, significantly reducing incident resolution time.
• Configured Databricks Unity Catalog for fine-grained access control (IAM-style), column-level and row-level security, and data masking aligned with pharma compliance standards.
• Applied advanced Spark tuning (broadcast joins, AQE, Z-ordering, Delta caching, cluster autoscaling), achieving ~40% reduction in average pipeline runtimes on multi-terabyte datasets.
• Led CI/CD deployments of notebooks and DLT pipelines across DEV/UAT/PROD via Azure DevOps with Git-based branching strategy and automated test gates.
• Drove product roadmap as Product Lead: translated business requirements into technical user stories, managed sprint delivery, and coordinated across data science, BI, and platform teams.

Senior Data Engineer at Indegene India Pvt. Ltd. | Client: Novartis – IM SPL Data Convergence (2022-08 – 2024-05)

• Designed and delivered ELT pipelines in Matillion processing data through STG → INT → DWH layers from Snowflake, AWS S3, SFTP, and FTP sources for a global pharma data convergence initiative (cloud data warehouse ELT, comparable to BigQuery ELT patterns).
• Worked on master-data integration and data convergence for pharma customer/HCP-related datasets, supporting standardized and trusted data for downstream commercial analytics.
• Built and optimised complex Apache Airflow DAGs (the orchestration engine behind Cloud Composer) with SLA monitoring, sensor operators, and failure alerting for cross-cloud (AWS + Azure) ETL orchestration.
• Implemented Databricks-based PySpark transformation logic for APLD, Claims, TRx/NTRx, HCP, and Call Activity datasets feeding downstream commercial analytics.
• Developed and maintained Snowflake data marts with 3NF staging, star-schema dimensional models, and performance-tuned analytical queries.
• Integrated Airflow with AWS S3, Redshift, and RDS; implemented S3/ADLS file-arrival sensor triggers for event-driven pipeline execution.

Data Engineer | Dataiku Developer at Indegene India Pvt. Ltd. | Client: Novartis – IPMM & Phoenix Data Convergence (2022-01 – 2022-08)

• Developed Dataiku flows for multi-source data ingestion and transformation using Python and SQL recipes; automated pipeline triggers via Dataiku scenarios.
• Built data validation dashboards in Dataiku to surface dataset refresh anomalies to business stakeholders before downstream consumption.
• Engineered PySpark-based SCD Type 1 & Type 2 MERGE patterns in Databricks for reliable incremental loads with full audit history.
• Supported pharma master-data convergence through incremental processing, historical tracking, data-quality validation, and standardized downstream datasets.

Azure Data Engineer at Capgemini India Pvt. Ltd. | Client: Unilever – Customer Development & Supply Chain (2021-01 – 2021-12)

• Designed and managed ADF V2 pipelines with parameterised, metadata-driven patterns ingesting data from SAP HANA, Oracle, SQL Server, Teradata, and flat files (CSV, JSON, Parquet) into ADLS Gen2.
• Integrated Azure Key Vault for secrets management (Secret Manager equivalent) across all ADF Linked Services, eliminating hard-coded credentials from production pipelines.
• Worked with PolyBase and external tables in Azure Synapse Analytics for high-throughput data loading from ADLS Gen2 into the enterprise data warehouse.
• Built Power BI dashboards with advanced DAX measures and KPI cards for supply chain and customer development reporting, distributed to 500+ business users.

B.Tech in Electronics & Communication Engineering – JNTU Hyderabad (2011 – 2015)