Lead Data Engineer

Programmers.io is currently looking for

• *a Lead Data Engineer
• *Onsite role in San Diego, California, United States
• *Contract Role - Open for W2 or C2C
• *Role Summary

We are seeking a highly skilled

• *Lead Data Engineer**

with deep, hands-on experience in building large-scale data ingestion pipelines, real-time streaming solutions, and high-performance data processing systems. The ideal candidate excels at writing clean, efficient code, refactoring complex systems, improving scalability and performance, addressing production issues, and delivering reliable data solutions across cloud platforms such as

• *AWS, Azure, or GCP**

.

• *Key Responsibilities
• Design, build, and maintain
• *high-volume data ingestion and processing pipelines

for batch and real-time workloads.

• Implement and optimize
• *real-time streaming pipelines**

using platforms such as

• *Kafka**

.

• Develop scalable data solutions using
• *Databricks, PySpark, Python, and SQL**

.

• Perform
• *core refactoring**

to modernize and optimize existing pipelines and data services.

• Build robust, fault-tolerant pipelines capable of
• *large-scale, high-throughput**

data processing.

• Write
• *unit tests**

, automate validation, and ensure high code quality and reliability.

• Integrate pipelines into
• *CI/CD workflows**

to streamline and automate deployment processes.

• Identify, troubleshoot, and fix
• *production issues**

, ensuring system reliability and stability.

• Address
• *performance bottlenecks**

and implement improvements for scalability, throughput, and efficiency.

• Work extensively across
• *AWS, Azure, or GCP**

cloud environments and cloud-native data services.

• *Required Qualifications
• 8+ years**

of hands-on data engineering experience building and maintaining large-scale data systems.

• Proven experience with
• *high-volume data ingestion**

, ETL/ELT, and real-time data processing.

• Strong expertise with
• *Kafka**

or similar streaming technologies.

• Advanced proficiency in
• *Databricks, PySpark, Python, and SQL**

.

• Experience with
• *core refactoring**

, improving code maintainability and system performance.

• Demonstrated ability to design and build
• *scalable, low-latency data pipelines**

.

If you are interested, please apply or feel free to share your updated resume at

• *anas.khan@programmers.io**