Skip to main content

Data Engineer

Technology
6 days ago
Service contractFully remote

Job description

Position: Sr Spark Snowflake Engineer (Python)

Location: Remote

Contract Role

Job Description:

  • We are seeking a highly experienced Senior Spark Snowflake Data Engineer with 10+ years of expertise in Data Engineering, Enterprise Data Warehousing, Cloud Data Platforms, API Development, and Data Modernization initiatives. The ideal candidate should have extensive experience in Enterprise Data Warehouse (EDW) migration programs, building scalable data pipelines, ingesting and transforming large datasets into Snowflake, and developing enterprise-grade APIs and data services.
  • The candidate should possess strong hands-on expertise in Python, Apache Spark, Scala, Snowflake, REST APIs, and Cloud Data Engineering, along with a solid understanding of the Healthcare Payer domain. The role demands technical leadership, architectural participation, client interaction, mentoring, work allocation, and delivering highly scalable and reliable data solutions.
Key Responsibilities:

Python Development

Develop high-performance data processing applications using Python.

Create reusable libraries, frameworks, and utilities.

Build automation tools supporting migration and operational activities.

Develop data validation and reconciliation utilities.

Implement unit testing frameworks and code quality standards.

Follow secure coding practices and engineering best practices.

Required Experience

10+ years of overall Data Engineering experience.

Strong hands-on experience with Snowflake, Python, Apache Spark, and Scala.

Experience in large-scale EDW migration programs.

Experience building enterprise-grade APIs and microservices.

Experience handling large healthcare datasets.

Proven experience in solution design, stakeholder interaction, and technical leadership.

Spark & Scala Development

Develop distributed processing solutions using Apache Spark.

Build scalable transformations using

Spark SQL

PySpark

Scala

Optimize Spark workloads for large-scale healthcare datasets.

Troubleshoot complex processing and performance bottlenecks.

Implement partitioning, caching, and workload optimization strategies.

Support both batch and streaming data processing frameworks.

Design, develop, and maintain scalable data ingestion, transformation, and processing pipelines.

Build enterprise-grade ETL/ELT frameworks using Python, Spark, and Snowflake.

Develop reusable data engineering components, utilities, and automation frameworks.

Process structured, semi-structured, and unstructured datasets.

Implement data cleansing, enrichment, validation, and standardization frameworks.

Optimize data pipelines for performance, scalability, and reliability.

Support batch and near real-time data processing requirements.

Enterprise Data Warehouse (EDW) Migration & Modernization

Participate in large-scale data migration initiatives from legacy EDW platforms to Snowflake.

Analyze source systems, data models, ETL workflows, and reporting structures.

Support migrations from

Teradata

Netezza

Oracle

SQL Server

Hadoop

Other Enterprise Data Warehouses

Develop source-to-target mappings and transformation logic.

Build migration validation and reconciliation frameworks.

Ensure high-quality migration with minimal business disruption.

Snowflake Design and implement scalable Snowflake-based data solutions.

Develop ingestion frameworks into Snowflake.

Build and optimize

Snowflake Data Pipelines

Enterprise Data Warehouses

Data Marts

Data Sharing Solutions

Secure Data Access Layers

Work with Snowflake capabilities including:

Snow pipe

Streams & Tasks

Time Travel

Clustering

Secure Views

Data Sharing

Zero-Copy Cloning

Optimize Snowflake compute, storage, and query performance.

API Development & Integration

Design, develop, and maintain RESTful APIs and microservices.

Build APIs supporting

Data Access Services

Data Exchange Solutions

Provider Integrations

Claims Data Services

Analytics Services

Implement API security, authentication, and authorization controls.

Manage API lifecycle and versioning strategies.

Collaborate with application and integration teams for enterprise-wide data consumption.

Healthcare Payer Domain Expertise

Develop and support solutions across:

Membership & Enrollment

Claims Processing

Provider Management

Care Management

Regulatory & Quality Reporting

Pharmacy & Clinical Data

Client & Stakeholder Management

Interact directly with client business and technology stakeholders.

Participate in requirement gathering workshops, solution reviews, and design discussions.

Clarify technical requirements and business needs.

Provide project status updates and technical guidance.

Support issue resolution and risk mitigation activities.

Drive successful delivery of business-critical data initiatives.

API & Integration

REST APIs

Microservices

API Security

API Lifecycle Management

Databases

Teradata

Netezza

Oracle

SQL Server

Cloud Technologies

Cloud Data Platforms

Data Modernization Solutions

Cloud Migration Projects

Strong understanding of:

Claims Processing

Membership & Enrollment

Provider Management

Care Management

Regulatory Reporting

Pharmacy Data

Clinical Data Integration

Keywords
sparksnowflakepythonjob-descriptionsscalacloud-datadata-engineeringhealth-carementoringcoaching-mentoringallocationtraining-and-developmentdata-processingdata-validationtesting-and-analysispolicies-and-practicesmicroservicesvisual-art-designplanning-and-designproduct-development-and-design

Interested in this role?