Skip to main content

Data Engineer

Technology
1 week ago
Service contract

Job description

Primary Responsibilities

  • Onboard clients via the components of our data engineering pipeline, which consists of UIs, Azure Databricks, Azure ServiceBus, Apache Airflow, and various container-based services configured through IUs, SQL, PL/SQL, Python, YAML, Node, and Shell, with code managed in GitHub, deployed through Jenkins, and monitored through Prometheus, Grafana
  • Work as a part of our client implementation team to ensure the highest standards of product configuration that meet client requirements
  • Test and troubleshoot data pipeline using sample and live client data.

    Utilize

    Jenkins, Python, Groovy Scripts and Java to automate these tests. Must be able to parse logs to determine next actions.

  • Work with product teams to ensure the product is configured appropriately.
  • Utilize dashboards for Kubernetes/OpenShift to diagnose high level issues and ensure services are healthy.
  • Support Implementation immediately after go live, work with O&M team to transition support to that team.
  • Participate in daily AGILE meetings.
  • Estimate project deliverables.
  • Configure and test REST APIs and utilize manual tools to interact with API's.
  • Work with data providers to clarify requirements and remove roadblocks.
  • Drive automation into everyday activities.

Required Qualifications:

  • 3+ years of experience working with SQL (preferably Oracle Pl/SQL and SparkSQL) and data at scale.
  • 3+ years of ETL experience ensuring source to target data integrity. Familiar with various file types (Delimited Text, Fixed Width, XML, JSON, Parque).
  • 1+ years of coding experience with one or more of the follow languages; Java, C#, Python, NodeJS using Git, with practical experience with working collaboratively through Git branching strategies.
  • 1+ years of experience with Microsoft Azure cloud infrastructure, DataBricks, DataFactory, DataLake, Airflow and Cosmos Database.
  • 1+ years of experience in reading and configuring YAML.
  • 1+ years of experience with ServiceBus, setting up ingress and egress within a subscription, or relevant Azure Cloud services administrative experience.
  • 1+ years of experience with Unit Testing, Code Quality tools, CI/CD Technologies, Security and Container Technologies.
  • 1+ years of Agile development experience and knowledge of Agile ceremonies and practices.

What you are like:

  • Have a great, positive attitude
  • Never blocked.

    You find a way!

  • Bring problems with solutions
  • Take extreme ownership and pride in your work
  • Care deeply about team culture and build teams on a foundation of mutual respect
  • Will automate something so you don't have to do it twice
  • Never stop learning (you have a habit of reading about changes in tech and then implementing them)

Required Skills : Strong SQL Experience Experience with PySpark and SparkSQL Experience building Data Pipelines Azure Environment - Experience including: Azure DataFactory Azure Data Lakes Azure Databricks

Strong datapipeline and ETL as a whole

Basic Qualification : SQL Spark SQL OR Pyspark building data pipelines Azure Datafactory, Databricks, Data Lakes

Additional Skills : SQL Spark SQL OR Pyspark building data pipelines Azure Datafactory, Databricks, Data LakesLong Term a lot of growth and promotion opportunities.

100% Remote

Background Check :Yes

Drug Screen :Yes

Notes

Selling points for candidate :Long Term a lot of growth and promotion opportunities.

100% Remote

Project Verification Info

Candidate must be your W2 Employee :Yes

Exclusive to Apex :No

Face to face interview required :No

Candidate must be local :No

Candidate must be authorized to work without sponsorship :Yes

Interview times set :Yes

Type of project :Implementation/Installation

Master Job Title :Big Data: Dev

Branch Code :Nashville

Interested in this role?