Nimish Randive

Data Engineer with 5 years of extensive, hands-on experience across Big Data technologies, building large-scale data pipelines, ETL/ELT workflows, and cloud data warehouse solutions within fast-paced, product-driven organizations. Well-versed in Python, SQL, Spark, Hadoop, Kafka, and Airflow, with hands-on depth across GCP (BigQuery, Dataproc, Composer, GCS), data modeling, streaming pipelines, data quality, and Looker/LookML semantic layers that power self-serve analytics for business teams

Data Engineer at Lowe's (2024-08 – Present)

• Built and maintain end-to-end data pipelines on GCP (GCS, BigQuery, Dataproc) to consolidate and process multi-terabyte datasets from diverse source systems
• Designed and deployed a Looker semantic layer on BigQuery datasets, used by the backend team for API integration and self-serve analytics.
• Significantly reduced monthly GCP costs by 50% by migrating Spark Dataproc jobs to optimized BigQuery SQL, demonstrating cost efficiency and performance improvement.
• Reduced critical job runtimes by 30% by applying BigQuery partitioning and clustering strategies to high-volume analytical workloads.
• Automated a Python based cache service to refresh Looker cache after BigQuery table updates, eliminating query latency and improving application responsiveness.
• Consolidated data from multiple third-party APIs into one analytics platform, improving reporting consistency across teams.
• Built streaming data pipelines using Spark and Kafka to ingest and process data for analytical reporting in Looker.

Associate Data Engineer at Lowe's (2021-11 – 2024-08)

• Optimized API performance by refactoring a complex data model, achieving faster response times and improved page load performance.
• Implemented the Great Expectations framework to automate data quality checks on final Hive and Druid tables, ensuring data integrity and reliability for business consumption.
• Built a Spark-based pipeline to pull semi-structured operational data from MongoDB into Hadoop for analytical reporting.
• Refactored Hive jobs into Spark SQL applications, resulting in faster execution times and enhanced scalability for processing larger datasets.
• Improved the performance of critical data pipelines by implementing Spark Optimization techniques like caching and broadcast joins, leading to significant reductions in overall processing time.
• Migrated 25 Oozie workflows to Apache Airflow, resulting in improved workflow orchestration, monitoring, and scalability.

Post Graduate Diploma in Big Data Analytics – CDAC Pune (2021-05 – 2021-10)

Bachelor of Engineering in Electronics Engineering – DY Patil Institute of Technology (2017-07 – 2020-10)