Ramesh Kota

Results-driven Azure Data Engineer with 7+ years of experience designing, building, and optimizing scalable data pipelines and cloud-based data platforms. Proficient in Azure Data Factory, Azure Synapse Analytics, Azure Databricks, Azure Data Lake Storage, and SQL Server, with hands-on expertise in ETL/ELT development, data warehousing, and big data processing using PySpark and Python. Skilled in designing end-to-end data solutions that support analytics, reporting, and business intelligence needs across large-scale enterprise environments.

Adept at implementing CI/CD pipelines, optimizing data workflows for performance and cost efficiency, and ensuring data quality, security, and governance across cloud environments. Strong collaborator with cross-functional teams including data scientists, analysts, and business stakeholders to translate business requirements into robust technical solutions. Proven track record of migrating on-premises data systems to Azure cloud, reducing processing time and infrastructure costs while improving scalability and reliability.

Azure Data Engineer at CONIN TECHNOLOGIES PRIVATE LIMITED (2024-12 – Present)

Azure Data Engineer at JLL Technologies Pvt Ltd (2024-12 – Present)

• Developed and maintained end-to-end operations of ETL data pipeline and worked with large datasets in Azure data factory (ADF).
• Configured Azure Data Factory (ADF) to efficiently ingest data from diverse sources, including relational and nonrelational databases, meeting critical business functional requirements.
• Orchestrated the seamless transfer of tables from Azure Data Lake to Azure Blob Storage, facilitating the smooth integration of data with ADF for advanced analytics.
• Built ETL Pipelines from Azure data lake to different databases based on the requirements.
• Worked on Azure Data Factory to integrate data of both on-prem and cloud (Blob storage, Azure SQL DB) and applied transformations.
• Deployed Data Factory for creating data pipelines to orchestrate the data into SQL database.
• Analytical approach to problem-solving; ability to use technology to solve business problems using Azure data factory, data lake and azure synapse.
• Demonstrated expertise in creating pipelines for efficient data processing and storage, effectively utilizing Azure Data Factory, Data Lake, and Databricks, while adhering to the Git flow branching strategy for streamlined code management and deployments.
• Developed and optimized code for Azure Functions to extract, transform, and load data from various sources, such as databases, APIs, and file systems.
• Built and maintained a centralized catalog for data assets across Databricks, improving discoverability and consistency of data across departments.
• Leveraged Unity Catalog's metadata management to streamline data classification, helping teams quickly locate and assess data quality and lineage.
• Configured fine-grained access controls with Unity Catalog, managing data permissions and roles to safeguard sensitive data and maintain compliance.
• Developing ETL transformations and validation using Spark-SQL/SparkData Frames with Azure data bricks and Azure Data Factory.
• Increased the efficiency of data fetching by using Queries for optimizing and indexing.
• Utilized Spark SQL API in PySpark to extract and load data and perform SQL queries.
• Worked with Azure Logic Apps administrators to monitor and troubleshoot issues related to process automation and data processing pipelines.
• Implemented robust Azure Active Directory authentication, ensuring secure access to all Azure Resources, and effectively demonstrated this feature to stakeholders.
• Successfully led the end-to-end delivery of PySpark ETL pipelines on Azure-Databricks, expertly orchestrating the transformation of data via Azure Data Factory (ADF) and effectively scheduling and triggering them.

Data Engineer at QATAR ENERGY (2019-03 – 2024-10)

• Designed and maintained ETL/ELT pipelines using Azure Data Factory to ingest data from upstream, midstream, and downstream operations including SCADA systems, IoT sensors, and historian databases.
• Architected Azure Data Lake Storage (ADLS Gen2) solutions for structured and unstructured data including well logs, and production reports.
• Manage data retention policies in compliance with industry regulations.
• Developed large-scale data transformation pipelines using Azure Databricks (PySpark) for production datasets.
• Built real-time streaming pipelines using Azure IoT Hub and Stream Analytics for monitoring drilling operations and enabling predictive maintenance on rig equipment.
• Cleanse and standardize data from disparate field systems (different rig operators, equipment vendors).
• Designed star/snowflake schema data models in Azure Synapse Analytics to support production reporting, reserves estimation, and reservoir engineering analytics.
• Implemented data governance and lineage tracking using Microsoft Purview, ensuring compliance with environmental and safety reporting standards.
• Secured sensitive exploration and reserves data using Azure Key Vault, RBAC, and encryption best practices.
• Collaborated with, petroleum engineers geoscientists, reservoir engineers, and BI teams to translate domain requirements into scalable data solutions.
• Partner with BI teams to enable dashboards in Power BI for production KPIs, HSE (Health, Safety, Environment) metrics, and asset performance tracking.
• Optimized pipeline performance and implemented CI/CD workflows using Azure DevOps, reducing deployment time and improving data reliability.

Security Analyst at DOPET (2016-03 – 2019-02)

Aircraft Maintenance Engineering – AP Aviation Academy

Intermediate (10+2) – Nalanda Junior College

SSC – Pragati School