Skip to main content

Architect

Technology
Halifax, Canada3 days ago
Full time

Job description

AWS Data Engineer with Databricks and DBT

Responsibilities

  • Design and Development of Data Pipelines:
  • Design, build, and optimize robust ETL/ELT pipelines using AWS services (S3, Glue, Lambda) and the Databricks platform (Spark, Delta Lake, DLT).
  • Ingest and process large volumes of structured and semi-structured data from various sources (APIs, databases, streaming platforms like Kafka or Kinesis) into a centralized data lake or lakehouse.
  • Data Transformation and Modeling:
  • Develop and maintain data models (e.g., star/snowflake schemas, medallion architecture) optimized for analytics and BI tools using dbt (Data Build Tool).
  • Write complex and efficient SQL queries and Python/PySpark code for data manipulation, transformation, and validation within the Databricks environment.
  • Implement data quality checks, tests, and documentation as part of the dbt workflow, enforcing data governance and security standards.
  • Orchestration and Automation:
  • Orchestrate and monitor data workflows using Databricks Jobs or external tools like AWS MWAA (Managed Workflows for Apache Airflow).
  • Implement CI/CD pipelines and version control (Git) for all data engineering artifacts (code, configurations, dbt models) to ensure reliable and consistent deployments.
  • Performance Optimization and Operations:
  • Monitor, troubleshoot, and resolve issues in production data pipelines and environments to ensure high performance, reliability, and cost-efficiency.
  • Tune Spark jobs and optimize Delta Lake features (Z-Order, partitioning) to handle growing data volumes and complexity.
  • Collaboration and Support:
  • Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights.
  • Provide expertise and guidance on data best practices, promoting a culture of data quality and governance.

Must Have skills

  • SQL
  • DBT core and DBT Cloud
  • AWS (redshift)
  • Data bricks with AWS
  • SQL server DB
  • Stone branch scheduling tool
  • Should understand CICD, GIT
  • Work in Agile environment with JIRA

Other Skills Required/ Good To Have

  • Tableau experience
Harness devops

Proficient in Linux / Unix environments

Interested in this role?