Skip to main content

Data Engineer

Technology
2 days ago
Full time

Job description

JOB RESPONSIBILITIES:

  • Design and develop data pipelines that will ingest the data from the respective source systems into the data warehouse
  • Support and optimize existing data pipelines to ensure that the downstream SLAs are met
  • Responsible in attending agile ceremonies as part of the Agile Tribe set-up and delivering the different demands of the Tribe related to data ingestions to the data warehouse platform
  • Ensures data quality within the data warehouse tables
  • Ensures the data engineering standards and practices are followed like code version history, readability, functionality, extendibility, test runs, deployments, and post deployment support
  • Supports L2 Data Operations team and act as L3 support
  • Manage data storage, computing unit requirements, and data security within the data warehouse
  • Gather data requirements coming from Tribe demands
  • Supports other data engineers on other Tribes or domains if necessary
  • Demonstrate technical excellence when it comes to the data warehouse platform
  • Supports stakeholders of the data warehouse platform by solving data related issues and conducting data investigations
  • Data Delivery Documentations – Data Dictionary, Data Mapping Documents, Data Models
  • Optimize Data Pipelines
  • Project delivery and operations support
JOB QUALIFICATION:
  • At least has 3 to 6 years of experience in IT project delivery or data development delivery particularly in Data Warehousing or Business Intelligence.
  • Graduate of any IT related courses like BS Computer Science, Information Technology or the likes
  • Strong SQL and ETL development experience, preferably related to Data Warehousing, Business Intelligence, or other Data Delivery projects.
  • Proficiency in optimizing data pipelines in order to meet SLA requirements and save cost
  • Experience in Big Data, Hadoop, EMR, Hive, and cloud environments is a plus
  • Proficiency in Python, R, Java, Scala is a plus
  • Proficiency in data modeling, data integrations, data quality, and data architecture is a plus
  • AWS data integrations (Glue, Athena)
  • Spark, Databricks
  • Delta lake, Data Lake
  • Kafka, Confluent, Real-time processing, Data Streaming
  • Azure Fabric (experience in ETL, Azure Data Factory, Azure blob storage)
  • Python (numpy, pandas, sklearn) works with PySpark and Scala technology
  • MustApache Airflow for Orchestration, can work with Azure Fabric pipelines and Azure DevOp
  • Must know these DE frameworks: Data Ingestion and working with API calls (see Azure Key vault), Data Transformation (table joins, aggregations, CTEs)
Work Setup: Hybrid: 1-2x a week onsite
Keywords
Data Engineer

Interested in this role?