Skip to main content

Data Engineer

Technology
1 week ago
Service contract

Job description

  • Onboard clients via the components of our data engineering pipeline, which consists of UIs, Azure Databricks, Azure ServiceBus, Apache Airflow, and various container-based services configured through IUs, SQL, PL/SQL, Python, YAML, Node, and Shell, with code managed in GitHub, deployed through Jenkins, and monitored through Prometheus, Grafana

    Work as a part of our client implementation team to ensure the highest standards of product configuration that meet client requirements

    Test and troubleshoot data pipeline using sample and live client data.

    Utilize

    Jenkins, Python, Groovy Scripts and Java to automate these tests. Must be able to parse logs to determine next actions.

    Work with product teams to ensure the product is configured appropriately.

    Utilize dashboards for Kubernetes/OpenShift to diagnose high level issues and ensure services are healthy.

    Support Implementation immediately after go live, work with O&M team to transition support to that team.

    Participate in daily AGILE meetings.

    Estimate project deliverables.

    Configure and test REST APIs and utilize manual tools to interact with APIs.

    Work with data providers to clarify requirements and remove roadblocks.

    Drive automation into everyday activities.

    Qualifications

    3+ years of experience working with SQL (preferably Oracle Pl/SQL and SparkSQL) and data at scale.

    3+ years of ETL experience ensuring source to target data integrity. Familiar with various file types (Delimited Text, Fixed Width, XML, JSON, Parque).

    1+ years of coding experience with one or more of the follow languages; Java, C#, Python, NodeJS using Git, with practical experience with working collaboratively through Git branching strategies.

    1+ years of experience with Microsoft Azure cloud infrastructure, DataBricks, DataFactory, DataLake, Airflow and Cosmos Database.

    1+ years of experience in reading and configuring YAML.

    1+ years of experience with ServiceBus, setting up ingress and egress within a subscription, or relevant Azure Cloud services administrative experience.

    1+ years of experience with Unit Testing, Code Quality tools, CI/CD Technologies, Security and Container Technologies.

    1+ years of Agile development experience and knowledge of Agile ceremonies and practices.

Required Skills : Strong SQL Experience Experience with PySpark and SparkSQL Experience building Data Pipelines

Azure Environment Experience including: Azure DataFactory Azure Data Lakes

Azure Databricks

Basic Qualification : SQL Spark SQL Pyspark building data pipelines

Azure Datafactory Databricks

Data Lakes

Additional Skills : SQL Spark SQL Pyspark building data pipelines

Azure Datafactory Databricks

Data LakesLong Term a lot of growth and promotion opportunities.

Background Check :Yes

Drug Screen :Yes

Notes :100% Remote

6 month contract to hire- candidates should be willing to convert to FTE no subvend candidates should be willing to work CST schedule

Selling points for candidate :Long Term a lot of growth and promotion opportunities.

Project Verification Info

6 month contract to hire- candidates should be willing to convert to FTE no subvend

Exclusive to Apex :No

Face to face interview required :No

Candidate must be local :No

Candidate must be authorized to work without sponsorship :Yes

Interview times set : :No

Type of project :Development/Engineering

Master Job Title :Big Data: Dev

Branch Code :Nashville

Interested in this role?