Manoj Kumar Goud Sowdari

Senior Data Engineer with 7 years of extensive experience in designing, developing, and optimizing scalable data pipelines and architectures. Proficient in leveraging cloud technologies such as Azure Data Factory, Databricks, and Azure Synapse Analytics to build robust ETL pipelines, real-time streaming solutions, and data warehouses. Proven expertise in cloud migration projects, including transitioning on-premises systems to Azure Cloud, and delivering business-critical analytics solutions using Snowflake and Power BI.

Skilled in implementing PySpark and Spark Streaming workflows for large-scale data processing and real-time insights, driving operational efficiency and enhancing business decision-making. Adept at ensuring data quality, lineage, and governance through frameworks like Informatica Data Quality and Azure Monitor, with a strong commitment to compliance standards such as HIPAA. Demonstrates expertise in CI/CD practices with tools like Azure DevOps and Git, enabling automated deployments and Agile workflows.

Collaborative and results-driven, with a proven ability to align technical solutions with business goals by working closely with stakeholders and cross-functional teams.

Senior Data Engineer | Tata Consultancy Services | Apr 2021 – Aug 2022

• Designed and implemented end-to-end data pipelines using Azure Data Factory (ADF) for efficient data ingestion, transformation, and loading (ETL) from diverse data sources.
• Developed complex ETL workflows with Informatica PowerCenter to extract, transform, and load data into data lakes and warehouses, tuning mappings and workflows to optimize performance.
• Deployed and managed Azure Data Lake Storage, implementing data partitioning and retention strategies to handle raw and processed data effectively.
• Optimized PySpark jobs in Azure Databricks using Spark performance tuning, caching, and partitioning techniques, achieving a 30% reduction in processing time.
• Leveraged Azure Event Hubs for high-volume, low-latency ingestion of streaming data, including POS transactions and customer interactions, integrating with Azure Databricks Streaming Jobs for real-time filtering, aggregation, and enrichment.
• Designed and implemented medallion architecture in Azure Data Lake to organize and manage data efficiently for analytics and reporting.
• Utilized Azure Synapse Analytics to optimize data transformation pipelines and support scalable analytics solutions, improving query and job performance by 25%.
• Integrated Snowflake with Power BI and Azure Analysis Services, enabling interactive dashboards and self-service analytics for business users.
• Developed data pipelines with Azure Logic Apps for workflow orchestration and triggered operations, improving pipeline efficiency and reliability.
• Implemented and monitored IoT streaming solutions by integrating Azure IoT Hub with Azure Event Hubs, ensuring scalable and secure event streaming.
• Designed and optimized normalized database schemas and transformation models using Dbt for enhanced analytics outcomes.
• Collaborated with cross-functional teams, including data scientists and analysts, to align data solutions with business objectives and deliver high-performance systems.
• Developed and maintained Kafka-based pipelines for ingesting and transforming retail data in real-time using Spark Streaming.
• Implemented robust error-handling mechanisms for APIs and integrated Azure services for seamless data flow and transformation.
• Utilized Azure DevOps for code management, version control, and CI/CD automation, streamlining deployments in a fast-paced Agile environment.

Environment: Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Snowflake, Informatica, Power BI, Kafka, Azure IoT Hub, Dbt, StreamSets, SQL, Python, Scala, Spark, APIs, Azure Logic Apps, Git, Jenkins, JIRA

Data Engineer | Amrutha Infotech Pvt. Ltd | Jun 2017 – Mar 2021

• Designed and implemented ETL pipelines using Azure Data Factory for efficient data ingestion and transformation from diverse sources such as SQL databases, CSV files, and REST APIs into cloud storage solutions like Azure Data Lake Storage.
• Developed distributed data processing workflows with Azure Databricks and PySpark, optimizing performance through partitioning, caching, and Spark tuning, achieving a 30% improvement in processing speed.
• Architected and deployed a cloud-based data warehousing solution using Snowflake, implementing advanced techniques like partitioning, indexing, and caching to enhance query performance and scalability.
• Enabled real-time streaming into Snowflake by integrating Azure Event Hubs, Azure Functions, and Databricks Delta, improving real-time analytics capabilities for business operations.
• Utilized Informatica PowerCenter and Informatica Data Quality (IDQ) to perform data validation, cleansing, and enrichment, ensuring the integrity and accuracy of ingested datasets.
• Leveraged Azure Synapse Analytics for big data processing, integrating analytics pipelines to support efficient query execution and advanced reporting.
• Designed and maintained data models, including logical and dimensional models, to optimize query performance and enhance analytics capabilities.
• Developed and automated workflows using Azure Logic Apps and event-driven triggers, reducing manual intervention in pipelines by 48%.
• Created and optimized custom PySpark UDFs to extend Spark SQL functionality for complex data transformations and aggregations.
• Built real-time streaming pipelines with Kafka and Spark Streaming, enabling continuous data ingestion and transformation, contributing to improved real-time analytics.
• Conducted performance tuning and capacity planning for data infrastructure, reducing latency and improving resource utilization.
• Executed data governance tasks, including implementing data lineage and metadata management, ensuring visibility and compliance across pipelines.
• Proficiently used Hive and Spark SQL for data transformation tasks, ensuring data integrity and scalability in large-scale datasets.
• Collaborated closely with data analysts, business stakeholders, and cross-functional teams to align technical solutions with business requirements, ensuring efficient delivery.
• Worked within Agile methodologies, participating in daily stand-ups and sprint planning to meet project timelines and ensure alignment with stakeholder goals.

Environment: Azure Data Factory, Azure Databricks, Snowflake, Azure Synapse Analytics, AWS Glue, Informatica, Kafka, Hive, PySpark, Python, Scala, Jenkins, Git, JIRA, Azure Functions, REST APIs

University of South Florida – Tampa, FL

Master of Science (MS) in Computer Systems analysis | Aug 2022 – May 2024

Osmania University – Hyderabad, India

Master’s Degree in Mechatronics, Robotics, and Automation Engineering | Aug 2018 – Jan 2021

Osmania University – Hyderabad, India

Bachelor of Engineering (BE) | Sep 2013 – Aug 2017