Neeraj

Detail-oriented Data Engineer, BI, and Data Analyst with 4 years of experience design, implementing robust data pipelines, real-time ingestion systems, and cloud-based data solutions. Proficient in building end-to-end ETL workflows and optimizing data processing using Python, Apache Spark. Skilled in handling large-scale data environments across various domains, with a strong focus on compliance with data governance.

Expertise in creating interactive dashboards to drive business intelligence and decision-making. Committed to leveraging advanced analytics, and big data frameworks to enhance operational efficiency and provide actionable insights. Adept at automating data pipelines ensuring continuous delivery and operational reliability.

Data Engineer - L1

(2025-01)

Intern Contract position involving ETL pipeline development, data processing, and cloud-based data solutions using Azure services.

• Conducted workshops, and requirement gathering sessions with stakeholders, and users.
• Created and implemented Synapse Pipelines using Linked Services, Datasets, and Pipelines to ETL data from various sources like Azure SQL, Blob Storage, ADLS, and developed Databricks ETL pipelines using notebooks, Spark Data Frames, Spark SQL, and Python scripting.
• Developed data processing tasks using PySpark such as reading data from external sources, merging data, performing data enrichment, and loading into target data destinations.
• Automated ADF pipelines using various triggers, created LogicApps task to send email notifications about pipelines run fails, monitored, and pipelines, triggers, linked services.
• Optimized Databricks, and Synapse performance using PySpark by tuning performance settings, such as batch interval time, level of parallelism, and memory configurations.
• Developed event-driven architectures using Event Grid, integrating with Cosmos DB to enable real-time data processing and seamless communication between distributed systems
• Developed and automated unit tests and participated in integration testing for ETL pipelines, ensuring data quality and robust error handling during production deployments.
• Implement version control systems and continuous integration/continuous deployment pipelines for data engineering projects to ensure proper collaboration & automated testing.

Data Developer - Truefy IT Consulting

(2023-06 - 2024-04)

Worked as Data Developer for MAX Healthcare client, collaborating with business stakeholders to build end-to-end metadata driven pipelines and ETL workflows.

• Collaborated with business stakeholders and team members to comprehend functional requirements, extracting data from various and build end to end metadata driven pipelines.
• Prepare and update documentations Data Mapping, Business Requirement Description, Data Validation, TTD) to ensure all the procedures and knowledge are traceable in future cycles.
• Developed Spark frameworks PySpark and Spark-SQL within Databricks to extract, transform, and aggregate data from various file formats for uncovering insights into usage patterns.
• Designed and implemented data ingestion pipelines into Azure services such as ADLS, Azure Storage, Azure SQL, and Azure DWH, processing data efficiently using Azure Databricks.
• Built robust ETL pipelines for extracting data from source systems and loading it into ADLS, leveraging ADF, T-SQL, Spark-SQL, and U-SQL for seamless data transformation and ingestion.
• Optimized data transformation workflows using PySpark to handle large-scale datasets efficiently by using caching and partitions, ensuring faster execution and reduced memory.
• leveraged built-in Spark functions in Databricks to minimize custom code, streamlined joins and aggregations, and implemented checkpointing for fault tolerance in long-running jobs.
• Participated in code reviews, architecture discussions, and performance-tuning initiatives to ensure adherence to best practices and improve overall data processing performance.
• Enhanced data governance by implementing RBAC and auditing features in Azure to monitor and manage data lineage, ensuring compliance with internal and external regulations

Data Analyst - Truefy IT Consulting

(2022-09 - 2023-06)

Worked as Data Analyst for Delhivery Logistics client, managing end-to-end ETL workflows and developing data pipelines with real-time data processing capabilities.

• Managed end-to-end ETL workflows using Databricks, ensuring real-time data processing and transformation, and integrating multiple data sources, Webpages and third-party APIs.
• Utilized Python and PySpark for data transformation and automation, reducing manual intervention in the data preparation process and improving data processing times.
• Integrated Azure services such as Azure Functions, Azure Logic Apps, and ADLS, Delta Lake into data workflows to enhance functionality and address diverse data processing scenarios.
• Supported for all ETL processes, from landing through to populating DWH, ensuring the integrity of data is maintained throughout and that the ETL processes are stable, accurate.
• Used various Azure Data Factory activities like Get Metadata, For Each, IF, Switch, Lookup, Web, set variables, activities to crate control flow pipelines and dataflows.
• Creating end-to-end data pipelines using ADF, managing on-demand clusters, establishing link services between source and destinations, ensuring seamless dataflow and integration.
• Developed OLAP cubes using Azure Analysis Services, enabling multidimensional analysis and driving advanced data insight, provided recommendations using Power BI.
• Enhanced data quality by implementing robust data validation checks within Unity Catalog, improving data accuracy, and ensuring reliable outputs for analytics and decision-making.
• Developed complex Tableau dashboards using advanced features such as Level of Detail (LOD) functions, ensuring that dashboards were both dynamic and user-friendly.

BI Developer - Truefy IT Consulting

(2022-03 - 2022-09)

Worked as BI Developer for Axis Bank client, designing and implementing dimensional models, data marts, and business reporting solutions.

• Gathered data requirement from different teams like marketing, operations, finance, IT and preparing the adequate recommendation using business reporting solution.
• Designed and implemented Dimensional Models, incorporating Slowly Changing Dimensions (SCD), and created Data Marts to support analytical reporting for healthcare operations.
• Involved in complete SSIS life cycle in creating SSIS package, building, deploying and executing the package in both the environments (Development and Production).
• Expertly deployed SSIS packages to production, ensuring environment independence using dynamic package configurations, significantly enhancing the reliability of the data integration
• Created Stored Indexing, Temp Tables, Procedures for performing optimization and other database operations such as complex SQL queries, views, and triggers etc.
• Utilized Power Query for data transformation and DAX for complex calculations and measures. Enhanced report performance and interactivity for better decision-making.
• Utilized DAX Studio for advanced query optimization and performance tuning, enhancing the efficiency and speed of complex DAX calculations and data models in Power BI reports.
• Enforce RLS in Power BI to ensure role-based data access, enhancing security compliance.
• Install and configure Gateways to schedule power bi datasets in power bi services.
• Publish reports into Power BI Services, and assign user permissions, and users to workspace

Masters - Big Data and Data Science Technology - Northumbria University

Bachelor - Electronics and Communications Engineering - Vaggeswari College of Engineering