Deepak S
Software Engineer
Hire this person
Send a job offer directly to this candidate
About
Data Engineer with 3+ years of experience in data ingestion, ETL/ELT, data transformation, data quality, web-scale data collection, and analytics-ready datasets using Python, SQL, PySpark, Apache Spark, Airflow, AWS, relational databases, and NoSQL. Built scalable ingestion and transformation workflows for large-volume e-commerce and market-intelligence datasets, including pipelines processing 100K+ records daily and workflows handling approximately 1TB of data. Developed cloud-oriented and automated data workflows using Scrapy, Zyte Scrapy Cloud, PySpark, SQL, and Apache Airflow while improving query performance, reliability, monitoring, validation, and downstream data readiness.
Experienced in data quality, schema validation, anomaly detection, reconciliation, database optimisation, orchestration, observability, and version-controlled engineering workflows.
Experience
Software Engineer – Data Engineering at Data Impact – NIQ (2024-05 – 2025-02)
- Designed scalable Python-based data ingestion pipelines using Scrapy and Zyte Scrapy Cloud, ingesting 100K+ records daily while maintaining high pipeline reliability.
- Collaborated with analytics and product teams to gather data requirements and onboarded 20+ e-commerce sources, increasing data coverage by 10%.
- Designed and maintained Apache Airflow ETL workflows for transforming and loading large datasets into MySQL and MongoDB; processed approximately 1TB of data and reduced processing time by 15%.
- Implemented scalable PySpark transformations and SQL-based processing for large datasets, preparing structured data for analytics consumption.
- Improved analytical query performance by 25% through indexing strategies, query optimisation, and schema improvements across relational and NoSQL databases.
- Built Grafana dashboards for pipeline performance, scrape success rates, system health, and alerting, reducing pipeline failures by 15%.
- Implemented schema enforcement, data validation, anomaly detection, and governance-oriented quality checks to improve dataset accuracy and reliability.
- Collaborated with data scientists and analytics teams using Git-based development and version-control workflows.
Data Quality Analyst at Turbolab Technologies (2023-09 – 2025-03)
- Developed and automated ETL workflows for extraction, transformation, and loading, reducing manual intervention and improving data workflow efficiency.
- Worked with SQL, MySQL, PostgreSQL, and data warehousing concepts to support complex queries, data validation, reconciliation, and performance-focused data operations.
- Built web-scraping and crawling workflows using Scrapy, Dragline, Selenium, Pyppeteer, and Playwright for structured and semi-structured data collection.
- Applied data cleansing, processing, validation, and automated reporting techniques to produce clean, consistent, and structured datasets.
- Performed REST API load and performance testing using Apache JMeter and prepared validation/reporting outputs.
- Used Git, GitHub, and GitLab for branching, merging, conflict resolution, repository management, and workflow automation.
Python Developer Intern at Techmindz (2022-08 – 2023-03)
- Developed web applications using Python and Django, supporting backend development and project delivery.
- Integrated frontend designs with Django templates and views and implemented database interactions and REST API functionality.
- Supported authentication, Django ORM, Git workflows, and deployment activities.
Education
Bachelor of Computer Applications in BCA – Bharathiar University (2019-05 – 2022-08)
