Kaustubh Pohane

Data Engineer with 3.5 years of hands-on experience building real-time and batch ETL pipelines using Apache Kafka, PySpark, and Databricks. Proven track record across IoT, retail, and airline domains — delivering 40% faster processing, 95% anomaly detection accuracy, and handling 20 GB+ of daily sensor data. End-to-end pipeline ownership from ingestion to analytics layer, with deep expertise in AWS S3, Airflow orchestration, PostgreSQL, and Snowflake.

Data Engineer at AUTONOMIX Solutions Pvt. Ltd. (2025-10 – Present)

• RetailOS – Unified Retail Data Platform: Built and owned end-to-end multi-platform data pipeline integrating Shopify, QuickBooks, and Square via Nango-proxied concurrent API workers, reducing data load time from hours to ~3 minutes
• RetailOS – Unified Retail Data Platform: Designed platform-isolated PostgreSQL schemas by mapping complex JSON payloads into structured layers, ensuring zero data bleed and full scalability across retail clients
• RetailOS – Unified Retail Data Platform: Architected raw storage layer on AWS S3 with optimized file formats for historical backup, enabling consistent, accurate data representation across all client-facing dashboards
• RetailOS – Unified Retail Data Platform: Sole owner of full backend data flow (Frontend → Service → Repository → DB), handling distributed data processing, architecture decisions, and end-to-end delivery independently
• AutoSage – AI-powered Internal Chatbot: Redesigned RAG pipeline architecture, reducing memory consumption by 35% and significantly improving document processing depth for multi-format ingestion (PDF, CSV, XLSX) into vector databases
• AutoSage – AI-powered Internal Chatbot: Led full codebase cleanup and refactor, improving system maintainability and ensuring seamless synchronization between all chatbot components
• AutoSage – AI-powered Internal Chatbot: Managed and mentored 2 interns across frontend and backend streams, coordinating delivery timelines, code reviews, and integration testing
• Airline sensor data capturing and transformation: Engineered a real-time ETL pipeline using Apache Kafka and Python, processing 20 GB of daily flight sensor data at 500 requests/minute via secure authenticated API
• Airline sensor data capturing and transformation: Built a 10-partition Kafka producer that reduced regional data skewness by 25%, ensuring efficient and balanced global data distribution
• Airline sensor data capturing and transformation: Accelerated PySpark transformation on Databricks by 30%, converting raw unstructured sensor data into analytics-ready structured datasets stored in Amazon S3
• Airline sensor data capturing and transformation: Documented full pipeline architecture and established data quality protocols in collaboration with analyst teams

Data Engineer at ICEICO Technologies Pvt. Ltd. (2023-01 – 2025-10)

• Operational data lot analysis - Batch processing: Engineered Apache Spark batch pipelines processing 1 TB of construction operational data across 25+ partitions, significantly improving throughput and processing efficiency
• Operational data lot analysis - Batch processing: Built Python scripts for API-driven data ingestion and text analysis on unstructured data, storing cleaned outputs in Snowflake and S3 for direct analyst access
• Operational data lot analysis - Batch processing: Applied SparkSQL for complex schema slicing and SQL query optimization, ensuring data consistency and producing client-ready analytical reports
• Operational data lot analysis - Batch processing: Established pipeline testing and troubleshooting protocols to uphold system reliability and reduce failure recovery time
• IoT Sensor Data processing and anomaly detection: Developed Kafka-based streaming pipelines ingesting 50+ real-time IoT sensor streams (temperature, humidity, airflow) from HVAC and climate control systems, maintaining 85%+ data readability
• IoT Sensor Data processing and anomaly detection: Processed 1+ GB/day of high-frequency sensor data using PySpark on Databricks, achieving 95% anomaly detection accuracy for predictive maintenance and operational resilience
• IoT Sensor Data processing and anomaly detection: Designed time-series schema with precise timestamp sequencing, reducing analytics retrieval time by up to 40%
• IoT Sensor Data processing and anomaly detection: Deployed Parquet-format transformed data to Amazon S3, creating a scalable data lake that directly supports predictive modeling workloads

Bachelor of engineering (B.E.) in Computer science and Engineering – Priyadarshini J.L. College of Engineering, Nagpur (2019 – 2023)

Higher secondary certificate (12th) in PCM Computer science branch – Santaji Mahavidyalaya, Nagpur (2017 – 2019)

Secondary school certificate (10th) – School of scholars, Wanadongri, Nagpur (2017)