Skip to main content

Data Engineer

Technology
Indianapolis, United States2 months ago
Service contract

Job description

Role: Remote Data Engineer

Length: 12 months + extension

Location: 100% Remote

Job Description:

Key Responsibilities

  • Designs and automates deployment of our distributed system for ingesting and transforming data from various types of sources (relational, event-based, unstructured).
  • Designs and implements framework to continuously monitor and troubleshoot data quality and data integrity issues.
  • Implements data governance processes and methods for managing metadata, access, retention to data for internal and external users.
  • Designs and provide guidance on building reliable, efficient, scalable and quality data pipelines with monitoring and alert mechanisms that combine a variety of sources using ETL/ELT tools or scripting languages.
  • Designs and implements physical data models to define the database structure.
  • Optimizing database performance through efficient indexing and table relationships.
  • Participates in optimizing, testing, and troubleshooting of data pipelines.
  • Designs, develops and operates large scale data storage and processing solutions using different distributed and cloud based platforms for storing data (e.g. Data Lakes, Hadoop, Hbase, Cassandra, MongoDB, Accumulo, DynamoDB, others).
  • Uses innovative and modern tools, techniques and architectures to partially or completely automate the most-common, repeatable and tedious data preparation and integration tasks in order to minimize manual and error-prone processes and improve productivity.
  • Assists with renovating the data management infrastructure to drive automation in data integration and management.
  • Ensures the timeliness and success of critical analytics initiatives by using agile development technologies such as DevOps, Scrum, Kanban Coaches and develops less experienced team members
Skillset:
  • Hands-on Azure Data experience
    • Tools to have: Databricks, Synapse, Data Factory, etc.
  • High functional knowledge of Big Data
  • Hands-on experience automating the ETL process
    • No specific tool is required: Jenkins preferred
  • Scala or PySpark knowledge (Scala preferred)

Required Skills : Ranked in Order of Importance: 1. Azure (Databricks, Synapse, Data Factory) 2. Big Data 3. Experience in automation (client wants to automate ETL process) 4. Scala or PySpark (Scala preferred)

Basic Qualification :

Additional Skills : -100% Remote -Long term contract with extension

Background Check :Yes

Drug Screen :Yes

Notes

Selling points for candidate :-100% Remote -Long term contract with extension

Project Verification Info

Exclusive to Apex :Yes

Face to face interview required :No

Candidate must be local :No

Candidate must be authorized to work without sponsorship ::No

Interview times set : :No

Type of project :Assessment/Analysis

Master Job Title :Big Data: Dev

Branch Code :Indianapolis

Interested in this role?