Pallabi Das

Proficient Data Engineer with hands-on experience in cloud-based data solutions, distributed systems, and large-scale data pipeline management. Skilled in cloud infrastructure, data transformation, and performance optimization, delivering high-quality results in fast-paced environments.

GCP Data Engineer - Walmart - Dallas TX

(2023-05)

• Designed and implemented incremental data push projects, improving logging and performance and contributing to business profitability.
• Built and optimized Spark-based ETL pipelines to handle large-scale data ingestion and transformation using GCP Dataproc and BigQuery.
• Optimized PySpark jobs to improve data processing times in Kubernetes clusters.
• Implemented automation scripts using Apache Airflow, streamlining daily workflows and minimizing manual intervention.
• Collaborated with offshore teams to gather requirements and deliver data-driven solutions, ensuring successful implementation of business requirements.

AWS Data Developer - Fannie Mae - Sterling VA

(2022-03 - 2023-03)

• Engineered scalable ETL pipelines using AWS EMR and Spark, processing structured and semi-structured data daily from S3 and Redshift.
• Migrated high-volume data workloads from Teradata and Hadoop to Snowflake, improving query performance.
• Optimized PySpark jobs for Amazon EMR by employing partitioning, salting, and clustering techniques, resulting in faster execution times.
• Developed real-time data ingestion pipelines using Talend, ensuring seamless data integration between multiple sources and reducing latency.
• Created and managed interactive dashboards in Tableau for business stakeholders, enabling real-time visibility into key performance metrics.

GCP Data Engineer - PayPal - Austin TX

(2021-09 - 2022-01)

• Migrated legacy data pipelines from on-prem Hadoop to GCP's DataLake, cutting processing time.
• Built scalable data transformation jobs using BigQuery and DataFlow, processing large datasets for downstream analytics and reporting.
• Developed Kafka-based data streaming services, integrating with Talend to automate ingestion workflows and reduce data lag.
• Created complex SQL queries and views to support business reporting needs in Snowflake.
• Collaborated with cross-functional teams to optimize data models, increasing query efficiency.

M.S. - Computer Science - Chicago State University (2021-08)

B. Tech. - Electrical and Electronics Engineering - NSHM Knowledge Campus (2016-07)