Abhishek Reddy Manam

Data Engineer with 3+ years of experience designing, building, and optimizing scalable ETL/ELT pipelines and cloud data platforms across multi-cloud environments. Production experience on Azure (Data Factory, Synapse, ADLS Gen2, DevOps) with hands-on work across AWS (S3, Glue, EMR, Redshift, Lambda) and working knowledge of GCP (BigQuery, Dataproc, Cloud Composer), plus platform-independent expertise in Databricks, Apache Spark (PySpark), Delta Lake, Snowflake, dbt, and Apache Airflow that transfers directly across providers. Strong programming skills in Python and advanced SQL.

Builds medallion lakehouse architectures, Kimball dimensional models, and reusable metadata-driven ingestion frameworks that standardize delivery regardless of underlying cloud. Experienced in cloud migration and modernization, and in optimizing pipelines for performance, reliability, scalability, and cost. Implements data quality, governance, monitoring, and security best practices (dbt tests, Great Expectations, catalog lineage, RBAC least-privilege, column-level masking), and follows DevOps and CI/CD best practices with Git, Terraform, and automated testing.

Databricks Certified Data Engineer

Associate. M.S.

Information Systems

Technologies, Wilmington University.

Data Engineer at Digiuniv Technologies Private Limited (2023-06 – 2024-07)

• Designed, developed, and maintained scalable ETL/ELT data pipelines across cloud platforms — PySpark on Databricks over Delta Lake with medallion layers (Bronze/Silver/Gold) on cloud object storage, plus metadata-driven ingestion pipelines (40+ linked services) from relational databases, REST APIs, files, and cloud storage (85% deployment-effort reduction; source onboarding from 3 days to under 2 hours).
• Led cloud migration and modernization work — moved legacy on-premise and relational workloads to cloud lakehouse and warehouse architectures, redesigning ingestion, transformation, and storage patterns while preserving reconciliation and audit requirements.
• Built and optimized ingestion, transformation, and integration workflows using Apache Spark (PySpark) — DataFrames and Spark SQL transformations, MERGE-based CDC/SCD upserts, and schema evolution handling; Spark UI and DAG analysis (broadcast joins, partition sizing, skew resolution, spill remediation) delivered ~50% latency reduction on portable Spark workloads that run on Databricks, EMR, or Dataproc.
• Developed and maintained cloud data lakes and columnar data warehouses — object-storage lakehouse with Parquet/Delta curated layers, MPP warehouse development (distribution design, columnstore, CTAS, workload management), and Snowflake (COPY INTO, Snowpipe, Streams & Tasks, clustering keys) supporting reporting and analytics.
• Created batch and near-real-time processing solutions and orchestration — scheduled and event-driven workflows for daily and intra-day loads, incremental CDC processing, and Apache Airflow DAGs (8+ production DAGs with dependency chains, sensors, retries, and SLA callbacks with Slack/PagerDuty alerting) portable across managed Airflow services.
• Developed reusable, cloud-agnostic data engineering frameworks and automation — parameterized, metadata-driven pipeline templates, modular Python ETL components (OOP design, type hints, pytest), and dbt layered models (80+ models with macros and tests) that standardized delivery across teams and platforms.
• Implemented data quality, governance, monitoring, and security best practices — Great Expectations suites and dbt schema tests as CI quality gates (~40% quality-incident reduction), cross-source reconciliation, structured error handling with reject-row routing and automated alerting (~60% fewer manual interventions), catalog-based lineage and RBAC least-privilege, and column-level masking with encryption and audit logging.
• Troubleshot production issues, optimized cost and performance, and collaborated cross-functionally — structured incident triage with runbooks covering 15+ failure patterns (MTTR from 2 hours to 30 minutes) and elimination of 3 recurring failure classes (~40% of manual incident volume); CI/CD via Azure DevOps, GitHub Actions, and dbt Slim CI (70% build-time reduction) with Terraform/ARM IaC; delivered Power BI semantic models to 200+ business users; Agile-Scrum in Jira; mentored 2 junior engineers.

Associate Data Engineer at Blockysite (2022-03 – 2023-06)

• Built and maintained ETL/ELT pipelines using Databricks PySpark jobs on Delta Lake, cloud data lake storage, and Snowflake/Synapse SQL procedures; developed dbt models (staging/intermediate/mart, macros, tests, snapshots, incremental models) and Kimball dimensional models.
• Orchestrated workflows with Apache Airflow DAGs and Databricks Workflows; monitored daily data loads and resolved production incidents within SLAs; applied Spark and SQL performance tuning; delivered through GitHub Actions and Azure DevOps CI/CD with Git PR code reviews in Agile-Scrum sprints (Jira).

Data Engineer Intern at Blockysite (2021-06 – 2022-03)

• First production data engineering experience — Python and PySpark ETL development, Databricks notebook contributions, cloud pipeline development, SQL querying, data-lake basics, data quality checks, pytest unit testing, Git PR workflow, and production incident-triage participation.

M.S. in Information Systems Technologies – Wilmington University (2026-05)

B.Tech. in Computer Science & Engineering – Lovely Professional University