Samardeep Yadav

Lead Data Engineer with 12+ years architecting enterprise data platforms across Azure, GCP, and Palantir Foundry. Proven track record building lakehouse architectures processing 10+ TB/day for banking, insurance, and real estate clients. Expertise spans platform design (Databricks, Synapse, BigQuery), governance (Purview, Unity Catalog), and team leadership.

Successfully led large-scale cloud migrations, delivered 40-60% performance optimizations, and built high-performing engineering teams across Fortune 500 clients.

Tech Lead & Data Architect at Tech Mahindra (2022-08 – Present)

Lead multi-cloud data strategy and architecture for financial services and insurance clients. Built and manage a team of 5 data engineers through hiring, onboarding, and continuous mentorship. Established code review standards and architectural best practices; present architecture roadmaps and delivery updates to business and engineering leadership.

• Architected target-state data warehouse topology to migrate legacy on-premise SQL Server systems to BigQuery, migrating 200+ tables representing 15+ TB of historical data.
• Designed schema-on-read ingestion framework to safely parse volatile Bloomberg market feeds with zero data loss, handling schema evolution automatically.
• Orchestrated end-to-end ingestion and transformation pipelines using Apache Airflow, managing complex DAG dependencies, automated retries, and SLA monitoring.
• Reduced query costs by 35% through BigQuery partitioning and clustering strategies, optimizing slot consumption for downstream reporting workloads.
• Implemented automated validation, reconciliation, and lineage tracking frameworks to enforce data quality and regulatory compliance post-migration.
• Delivered project 3 weeks ahead of schedule with comprehensive documentation and knowledge transfer.
• Led architecture for high-throughput Azure Lakehouse (ADF to ADLS to Databricks to Synapse) processing 8-10 TB/day using Medallion Architecture across Bronze, Silver, and Gold layers.
• Built production-grade PySpark pipelines in Databricks to transform denormalized source data into Kimball star schemas for insurance underwriting analytics.
• Designed semantic layer blueprints in Palantir Foundry Ontology to map raw data assets to business objects, enabling self-service analytics.
• Achieved 40-60% reduction in cluster processing overhead through Spark shuffle tuning, strategic caching, and partition pruning optimization.
• Delivered 80% performance improvement on critical reporting pipeline (2.5 hours to 30 minutes) by redesigning aggregation logic, adding pre-aggregation layers, and implementing Delta Lake caching.
• Implemented row-level and column-level security (RLS/CLS) with RBAC across the warehouse layer, ensuring GDPR and SOC2 compliance.
• Established Unity Catalog governance framework for centralized access control and audit logging.

Senior Azure Data Engineer / Platform Designer at Jones Lang LaSalle (JLL) (2020-01 – 2022-05)

• Co-architected cloud-native data platform (ADF to ADLS to Databricks to Synapse) scaling to 5-7 TB/day, supporting real estate valuation and portfolio analytics.
• Built metadata-driven ingestion frameworks that reduced manual operational intervention by 70% through dynamic pipeline generation and configuration-based execution.
• Designed reusable dimensional data models (Kimball methodology) that unified fragmented real estate business domains across 15+ regional offices.
• Integrated Snowflake data warehouse for specialized reporting workloads, building data models and optimizing queries for finance dashboards (3-month engagement).
• Established CI/CD pipelines in Azure DevOps using ARM templates to automate infrastructure deployment, improving platform stability by 40% and reducing deployment time from days to hours.
• Implemented monitoring and alerting with Azure Monitor and Log Analytics, reducing incident response time by 50%.

Data Engineer / Infrastructure Analyst at Maersk (2019-01 – 2019-12)

• Built Apache Spark pipelines processing 4-5 TB/day of international shipping and logistics data across 200+ ports and routes.
• Designed real-time data quality monitoring using Azure Log Analytics and Application Insights to detect anomalies and data quality issues early in the pipeline.
• Automated infrastructure deployment using ARM templates in Azure DevOps CI/CD pipelines, standardizing environment provisioning across dev, test, and production.
• Optimized data partitioning strategies in ADLS Gen2, reducing query times by 35% for vessel tracking and shipment analytics.

Technology Consultant at Deloitte (2017-06 – 2018-12)

• Built big data processing pipelines across 40+ corporate business verticals using Apache Spark and Hive on Hadoop ecosystem.
• Led migration from text-based formats to ORC storage, achieving 50% storage reduction and 3x query performance improvement.
• Developed data quality frameworks to validate data consistency across heterogeneous source systems.
• Collaborated with business analysts to translate complex business rules into scalable enterprise analytical pipelines.

Bachelor of Technology (B.Tech) in Computer Science & Engineering – Biju Patnaik University of Technology (BPUT)