Divesh Reddy

AWS Data Engineer with 3+ years of experience designing and operating large-scale ETL/ELT pipelines on AWS using PySpark, Delta Lake, and a lakehouse architecture. Skilled in distributed Spark processing, advanced SQL, performance and cost optimization, and event-driven orchestration with Step Functions, Lambda, SQS, SNS, and EventBridge. Proven track record of delivering production-grade data platforms that power enterprise analytics, billing, and reporting at billions-of-records scale.

Software Engineer at Constella Intelligence (2023-04 – Present)

• Migrated 90 billion records (~80 TB) from legacy OpenSearch to a modern Delta Lake on S3, accelerating analytical query latency by 30% and enabling self-serve access for downstream business and reporting teams.
• Reduced AWS compute spend by 20% with zero downtime by re-architecting Spark ETL workloads from EMR to containerized ECS Fargate, improving fault isolation and independent service scaling.
• Delivered an hourly, fault-tolerant ingestion service handling high-volume message streams at 99.5% accuracy and <1% error rate, ensuring reliable downstream availability of billing and usage data.
• Improved Athena query performance and reduced scan costs by applying Z-Ordering, partitioning, and Glue Data Catalog optimizations across multi-TB Delta Lake tables.
• Implemented a medallion architecture for the billing data platform: raw application logs (JSON) landed in S3 as the Bronze layer; normalized, validated records were ingested into MySQL Aurora as the Silver layer; and daily MTD and monthly aggregated billing reports were generated as the Gold layer — providing clean, audit-ready data at each stage.
• Engineered resilient event-driven orchestration that automates large-scale pipeline execution, anomaly detection, and operational alerting — cutting manual intervention and improving SLA adherence on production workloads.
• Automated recurring billing and reconciliation reporting by consolidating data across four heterogeneous stores, eliminating manual effort and ensuring revenue accuracy.
• Collaborated cross-functionally with product, analytics, and customer-success teams to translate business requirements into scalable data products serving enterprise customers.

Backend Developer, Apprentice at Seneca Global (2023-02 – 2023-04)

• Completed hands-on backend training and developed a functional HR management prototype using FastAPI, SQLAlchemy, and PostgreSQL to demonstrate practical understanding of RESTful APIs and database design.

B.Tech in Computer Science & Engineering – Malla Reddy Engineering College (2019 – 2023)

Intermediate in MPC – Narayana Junior College (2017 – 2019)