ETRM Databricks Data Engineer

Job Description:

Position: ETRM Databricks Data Engineer

Location: Houston, Texas (5 days on-site)

Hire Type: Full Time

Experience: 10 Years

ETRM Databricks Data Engineer to design, develop, and maintain scalable data solutions that support business intelligence, analytics, and data-driven decision-making with understanding of Data Domain & Data Assets in ETRM Domain. The ideal candidate will have strong expertise in Databricks, Apache Spark, Azure Cloud platforms, and modern data engineering practices.

Key Responsibilities

• Design, develop, and optimize data pipelines using Databricks & ADF.
• Must possess good understanding of Trading Domain knowledge.
• Build and maintain scalable ETL/ELT processes for structured and unstructured data.
• Integrate data from multiple sources into enterprise data platforms.
• Develop and manage data lakes, lakehouses, and data warehouses.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
• Ensure data quality, integrity, security, and governance across data platforms.
• Monitor and troubleshoot data processing jobs and optimize performance.
• Implement CI/CD practices and automate data engineering workflows.
• Create and maintain technical documentation, data models, and process flows.
• Support cloud-based data solutions and migration initiatives.

Required Qualifications

• Hands-on experience with Databricks and Azure.
• Strong proficiency in Python, SQL, and PySpark.
• Experience with cloud platforms such as Azure
• Knowledge of data warehousing, data modeling, and ETL/ELT concepts.
• Experience with version control tools such as Git.
• Strong analytical, problem-solving, and communication skills.
• Trading Domain experience related for Crude, Refined Products, Natural gas is a must.

Preferred Qualifications

• Experience with Azure Data Factory, Azure Synapse Analytics, Delta Lake, or similar technologies.
• Knowledge of streaming data processing and real-time analytics.
• Familiarity with DevOps practices and CI/CD pipelines.
• Databricks, Azure, or related cloud certifications.