Lead Data Engineer - Spark/Python

Primary Skills : Pyspark, python, SQL, scala, should have led or mentored team mates

Core Responsibilities :

• Hands-on delivery oversight : review pipelines, SQL, ETL/ELT logic, Spark jobs, reporting outputs and technical deliverables.
• Mentor junior talent : coach data engineers, data analysts and data quality analysts through daily guidance, reviews and troubleshooting.
• Data quality and controls : define validation, reconciliation, anomaly checks, lineage awareness and issue triage for finance/reporting data.
• Engineering governance : enforce coding standards, documentation, reusable patterns, production-readiness and maintainable data solutions.
• Stakeholder communication : clearly communicate risks, blockers, dependencies and corrective actions to delivery and programme stakeholders.

Tech Stack :

• SQL : advanced joins, window functions, query tuning, data validation and troubleshooting.
• Python : scripting, data processing, API invocation, automation and pipeline support.
• ETL/ELT : ingestion, transformation, scheduling, monitoring, error handling and operational support.
• Apache Spark : Spark SQL, DataFrames/Datasets, batch pipelines, structured-streaming concepts, partitioning and performance tuning basics.
• Scala / PySpark : ability to understand or develop Spark transformations using Scala or PySpark; Scala hands-on preferred where Spark workloads are Scala-based.
• Data modelling and governance : warehousing basics, quality checks, reconciliation, auditability and traceable reporting outputs.

Job Location : Hyderabad (all 5 days Work from TEKsystems Office)