Data Engineer - Apache Spark

Job Title : Data Engineer

Key Responsibilities :

Data Pipeline Development

• Design and develop scalable data pipelines to process and transform large volumes of structured and unstructured data.

Data Processing

• Build and optimize data processing frameworks using Apache Spark.

Data Integration

• Integrate data from multiple sources to create reliable and efficient data workflows.

Data Management

• Ensure efficient storage, retrieval, and transformation of data for analytics and reporting.

System Performance

• Monitor and optimize data pipelines for performance, scalability, and reliability.

Troubleshooting

• Identify and resolve issues in data pipelines and processing systems to ensure data accuracy and availability.

Collaboration

• Work closely with data scientists, backend engineers, and product teams to understand data requirements and deliver solutions.

Documentation

• Maintain clear documentation of data pipelines, data models, and workflows.

Continuous Improvement

• Stay updated with emerging technologies and best practices in big data and data engineering.

Required Skills :

• Strong proficiency with Apache Spark and distributed data processing.
• Experience designing and implementing scalable data pipelines.
• Knowledge of data integration techniques across heterogeneous sources.
• Ability to monitor, troubleshoot, and optimize data workflows for performance and reliability.
• Good communication skills for crossfunctional collaboration and documentation.