Rekha N

Proficient in designing and implementing scalable, end-to-end data pipelines using ADF and Databricks, Synapse, and on-premises SSIS to streamline ETL pipelines from diverse data sources. Skilled in optimizing data pipelines and PySpark jobs through advanced techniques like Spark performance tuning, data caching, and partitioning to achieve superior processing efficiency. Adept at employing Databricks Streaming Jobs and Event Hubs for real-time data filtering, aggregation, and enrichment, enabling seamless integration compliance data.

Experienced in deploying ADLS as a scalable and reliable data repository, implementing effective data partitioning and retention strategies for both raw and processed datasets. Commercial hands-on experience in the implementation of OLTP and OLAP, including building Star and Snowflake schema, Data marts, Cubes, and Data Lake solutions.

Experience in designing self-service reports to provide data insights using Power BI, and Tableau.

Technical Data Engineer (Senior Executive Officer – Fixed Term) - Animal and Plant Health Agency (APHA)

(2025-11)

Department: DEFRA – Science Directorate

• Developed and maintained automated data ingestion and processing pipelines for animal health surveillance data using Azure Synapse, Python and SQL.
• Built and orchestrated Azure Synapse pipelines to automate the ingestion of Antimicrobial Resistance (AMR) surveillance data submitted by private veterinary laboratories and external data providers, reducing manual data-processing effort.
• Developed Python-based processing notebooks in Azure Synapse to cleanse, standardise, transform and validate AMR datasets before downstream analysis and reporting.
• Worked with Medallion Architecture (Bronze, Silver and Gold layers) to organise raw, cleansed and curated surveillance datasets, using Delta format for Silver and Gold layers to support reliable and analytics-ready data processing.
• Implemented data transformation and curation processes across the Bronze, Silver and Gold layers, including data cleansing, standardisation, validation and preparation of datasets for downstream analytics and reporting.
• Configured and maintained Synapse pipelines, linked services and datasets to support scheduled and event-driven processing of incoming laboratory data feeds.
• Applied comprehensive data-quality and validation checks to identify inconsistencies in surveillance data and improve the accuracy and reliability of published outputs.
• Performed data transformation, cleansing and validation using Python and SQL, ensuring data accuracy and consistency across source and target systems.
• Supported the design and management of data lake and data warehouse solutions, including effective data organisation, partitioning and retention strategies.
• Integrated internal APHA datasets with third-party and private veterinary laboratory data sources to enhance animal health surveillance reporting.
• Supported the migration of manual data-handling processes to automated data pipelines, improving operational efficiency, consistency and reliability.
• Worked with Azure/AWS cloud services to deploy and support scalable data-processing solutions.
• Implemented basic monitoring, logging and error-handling mechanisms to support stable and reliable pipeline operations.
• Prepared and maintained curated datasets for Tableau dashboards and routine surveillance reports, supporting analysts, scientists and other business users.
• Collaborated with data analysts, scientists and delivery teams to understand data requirements and support analytics and surveillance use cases.
• Ensured compliance with GDPR, data governance and data security standards when handling sensitive surveillance data.
• Used Git for version control and followed CI/CD practices for managing and deploying data pipeline code.
• Provided day-to-day technical support and knowledge sharing within the Surveillance Intelligence Unit (SIU) and participated in team workshops and cross-site collaboration.
• Gained practical knowledge of FME (Feature Manipulation Engine) for data integration, transformation and workflow development.

Data Engineer - Revolut

(2025-02 - 2025-06)

• Designed and implemented end-to-end data pipelines using ADF and Databricks to facilitate efficient data ingestion, transformation, and loading (ETL) from diverse data sources.
• Optimized data pipelines and PySpark jobs in Databricks through advanced techniques like Spark performance tuning, data caching, and partitioning, resulting in superior performance.
• Employed Azure Databricks Streaming Jobs for real-time data filtering, aggregation, and enrichment in integrating financial data with compliance information.
• Leveraged Azure Event Hubs, triggers, and data bricks clusters for high-volume, low-latency ingestion of financial transactions, compliance updates, and client interactions.
• Deployed ADLS as a reliable and scalable data lake solution, implementing efficient data partitioning and retention strategies to store and manage both raw and processed data.
• Developed custom monitoring and alerting solutions using Azure Monitor, providing proactive identification and resolution of performance bottlenecks.
• Implemented robust error, and event handling mechanisms for API, and ETL interactions with Azure.
• Employed medallion architecture to ensure the design and organize data patterns in ADLS.
• Implemented data cataloging and data lineage solutions using Azure Purview and Apache Atlas to provide a comprehensive understanding of data assets and their relationships.
• Integrated ADLS seamlessly with Power BI and Azure Analysis Services to deliver interactive dashboards and reports, empowering business users with self-service analytics capabilities.

Data Engineer - Technoserve It Solutions LTD

(2024-05 - 2025-01)

Client: Generali

• Collaborated closely with cross-functional teams including business stakeholders, ensuring alignment with data requirements and delivering scalable and reliable data solutions.
• Integrated Azure services with APIs to design and develop data pipelines using ADF, Synapses, and Databricks that consume and produce data via APIs.
• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources and write back data into targeted system ADLS, Azure SQL.
• Enforced comprehensive data quality checks using ADF, and Databricks guaranteeing the highest standards of data accuracy and consistency across multiple sources and targeted systems.
• Ensured data quality and integrity through comprehensive data validation, cleansing, and transformation operations performed using Azure Data Factory and Databricks.
• Implemented comprehensive data lineage and metadata management solutions, ensuring end-to-end visibility and governance over data flow and transformations.
• Developed data ingestion pipelines to bring data from various sources into Azure Synapse.
• Developed event-driven data pipelines using Azure Event Hubs to ingest and process large volumes of events and messages in real time, facilitating real-time analytics.
• Developed and deployed Power BI, and SSRS dashboards with custom visuals, slicers, and interactive features like highlight and filter actions to enhance user experience.

BI Developer - Tata Consultancy Services

(2022-12 - 2024-03)

Client: CVS Health

• Collaborated closely with data analysts and business stakeholders to deeply understand their needs and implement well-aligned data models and structures within Snowflake.
• Managed data migration processes using SQL, Azure Data Factory, and SSIS, and developed ETL frameworks to load and process large datasets into Azure Synapse Analytics.
• Developed event-driven analysis reports and dynamic visualizations in Power BI using DAX for tables, aggregation, and iteration functions to deliver valuable business insights.
• Worked with complex database schemas, including normalization, denormalization, and relationship mapping for better-reporting structures in Power BI and Tableau.
• Configured Dataverse environments for secure and scalable data storage, allowing easy integration with Power BI for quick data retrieval and dashboard creation.
• Ensured data accuracy across Power BI and SQL Server through rigorous data validation, data modeling, and regular scheduling of data refreshes using On-premises data gateways.
• Published reports to Power BI Service, configured data connections, scheduled refreshes, and collaborated with end users to deliver tailored solutions.

Data Analyst - Tata Consultancy Services

(2022-01 - 2022-11)

Client: T-Mobile

• Gathered and refined business requirements to design and develop customized reports, delivering actionable insights through interactive reports using Power BI, SSRS, and Tableau.
• Created and published complex Power BI reports with drill-down and drill-through functionalities, leveraging Power BI Service for scheduling and report automation.
• Developed and optimized ETL processes using SSIS for seamless data integration, including transformations like Lookup, and Aggregator, ensuring data accuracy and integrity.
• Engineered SQL Server Reports (SSRS), utilizing global variables, expressions, and functions to create dynamic, parameterized reports for real-time data analysis and decision-making.
• Designed and built data models using SSAS, creating cubes with star and snowflake schemas, enabling efficient multi-dimensional data analysis and reporting.
• Implemented rigorous data validation checks within Power BI and SSIS to ensure data quality, accuracy, and consistency across all reporting environments.
• Implementing Row-Level Security (RLS) in Power BI, ensuring secure, user-specific data access through seamless integration with the Power BI Service portal.

Bachelor's degree - Electronics and Communication Engineering - JNTU