Nilesh Mahajan

Data Engineer with 2.8 years of experience building and operating production-grade ETL/ELT pipelines and large-scale data infrastructure. Proficient in Python, SQL, and PySpark, with hands-on experience across Azure Data Factory and Azure Data Lake, workflow orchestration (Prefect, Airflow), and columnar warehousing at 500+ TiB scale. Strong foundation in data modelling, data quality, and data governance, with a track record of designing reliable pipelines for systems that needed fast, accurate data in real time.

Data Engineer at Minix Fintech Pvt Ltd (2023-10 – Present)

• Led a 5-member team to deliver an in-house research and trading platform (Datafeed, OMS, Strategy microservices) - owning architecture, integration contracts, code reviews, and production releases for 4–6 portfolio managers.

Awarded Best

Performer of the Year.

• Developed scrapers using Requests, UrlLib2, BeautifulSoup, and Scrapy to handle both static and JavaScript-rendered pages; leveraged Chrome DevTools (Network, Application, Performance tabs) to analyze website dynamics and uncover hidden API endpoints, session tokens, and cookie flows on sites with anti-scraping protections.
• Built a high-throughput UDP ingestion pipeline handling thousands of order messages per second across NSE and BSE, with TCP recovery for missed packets, completion validations, and generation of 30,000+ token-level structured records daily.
• Contributed to integrating and maintaining financial datasets from Bloomberg, Refinitiv (LSEG), and FactSet, and designed a scalable market data ingestion framework that improved data quality and processing efficiency across multiple providers.
• Reduced backtesting time by 50 percent (10 min → 5 min) through advanced ClickHouse query optimization and Polars-based data processing, directly improving research throughput for 4–6 portfolio managers.
• Orchestrated 25–30 production Prefect flows end-to-end across NSE, BSE, and MCX - covering DAG creation, scheduling, failure recovery, and Slack-based granular alerting that reduced mean time to detect pipeline failures from hours to seconds.

B.E in Mechanical Engineering – Government Engineering College, Daman (2023-07)