Harish K Y

’m a Data Engineer with nearly 5 years of professional experience, specializing in big data technologies and cloud-based data solutions. My expertise spans AWS and Azure platforms, where I’ve developed and managed scalable ETL pipelines and implemented real-time and batch analytics using Databricks and Spark. With a strong foundation in both Agile and Waterfall methodologies, I excel in optimizing data workflows, ensuring data quality, and leveraging advanced tools and technologies to deliver actionable insights.

My background in architecting complex data solutions and deploying CI/CD pipelines underscores my commitment to driving efficiency and innovation in data engineering.

Professional Experience Summary

State of Nebraska, Omaha, NE

Data Engineer (May 2022 – Present)

• Led and managed financial data engineering projects, utilizing Agile and Waterfall methodologies.
• Developed and maintained scalable ETL pipelines using Azure Data Factory and Databricks.
• Engineered automated deployment pipelines with Terraform and Azure CLI.
• Enhanced data quality and compliance with HIPAA regulations, resulting in a 20% improvement in financial outcomes.
• Utilized Spark, Python, and Hive for complex data analytics and integrated disparate financial systems through cloud-based API frameworks.

Humana, Bangalore, IN

Data Engineer (Jun 2019 – Dec 2020)

• Architected and developed scalable datasets on AWS, utilizing S3 Data Lake and Snowflake.
• Built data ingestion and processing frameworks with Python and PySpark, orchestrating pipelines with Airflow.
• Implemented data quality frameworks and CI/CD processes using Jenkins and AWS Lambda.

Data Analyst (Jun 2018 – May 2019)

• Managed data ingestion and reconciliation using Azure Data Lake and Databricks.
• Developed ETL applications with Spark and Python, and performed data analysis with R.
• Optimized Spark jobs and performance, and built UDFs for PII data anonymization.

I earned my Master’s in Information Technology from Pittsburg State University, where I developed a robust foundation in advanced data management, cloud computing, and big data technologies. The program provided me with comprehensive knowledge and practical skills in areas such as data architecture, network security, and software development. Through rigorous coursework and hands-on projects, I gained expertise in designing and implementing scalable IT solutions, which has been instrumental in my professional career as a Data Engineer.

This advanced degree equipped me with the critical thinking and technical skills necessary to tackle complex data challenges and drive innovation in the field of information technology.