Performance Test Data Engineer

Job Description: Data Platform Engineer (QA Storage Focus)

Role Overview

We are looking for a Data Platform Engineer with strong QA and Data Validation experience to support large-scale data platforms. The ideal candidate will have hands-on experience in testing data pipelines, validating data lakes/storage systems, and ensuring data quality, accuracy, and performance across distributed environments.

Key Responsibilities

• Design, develop, and execute data validation and QA test strategies for ETL/ELT pipelines
• Perform end-to-end data validation between source systems and target data platforms (Data Lake / Data Warehouse)
• Validate large-scale datasets (millions/billions of records) using SQL, Python, and PySpark
• Perform file-level and storage validation across data lakes (S3 / ADLS / HDFS)
• File count validation
• Schema validation
• Partition validation
• Data completeness checks
• Test and validate data ingestion pipelines (batch & streaming)
• Validate data across Bronze / Silver / Gold layers (Medallion architecture)
• Perform data reconciliation and consistency checks across multiple systems
• Develop and maintain automated data validation frameworks using Python (PyTest or similar)
• Implement and monitor data quality checks (nulls, duplicates, referential integrity)
• Validate data formats such as Parquet, ORC, Delta Lake
• Conduct performance testing of data pipelines and queries (Spark / SQL)
• Analyze and validate data processing performance, latency, and throughput
• Collaborate with Data Engineers to identify and fix data issues and optimize pipelines

Required Skills

Data QA / Testing

• Strong experience in ETL/ELT testing and data validation
• Expertise in SQL for data validation and reconciliation
• Experience with test case design, execution, and defect tracking
• Knowledge of data quality frameworks and validation techniques

Data Engineering Knowledge

• Understanding of data pipelines (ADF / Airflow / Glue / Databricks)
• Experience with PySpark / Apache Spark (basic to intermediate)
• Familiarity with data modeling and transformations

Storage / Data Lake Validation (MANDATORY)

• Hands-on experience with Data Lakes (AWS S3 / Azure ADLS / HDFS)
• Strong knowledge of:
• File-based validation
• Partitioning strategies
• Schema evolution
• Experience validating Parquet / ORC / Delta Lake datasets

Programming & Tools

• Python (for automation/testing)
• SQL (strong)
• Experience with PyTest / automation frameworks
• Git / CI-CD basics

Cloud Platforms (Any One)

• AWS (S3, Glue, Athena) OR
• Azure (ADLS, ADF, Databricks)

Nice to Have

• Experience with Great Expectations / Deequ (data quality tools)
• Knowledge of Kafka / streaming validation
• Experience with Delta Lake features (time travel, versioning)
• Exposure to data governance tools (Glue Catalog, Unity Catalog)

Ideal Candidate Profile

• Strong Data Engineer with QA/testing experience
• Hands-on with data validation storage systems
• Comfortable working with large-scale distributed data platforms
• Detail-oriented with a focus on data accuracy, quality, and performance