Job Title : Lead Data Engineer
Location : Marathalli, Bangalore
Job Mode : Remote
Key Responsibilities :
- Create and maintain data sources inside our data lake on Azure
- Assemble and analyze large, complex data sets that meet both functional and non-functional business requirements
- Identify, design, and implement internal process improvements : automating manual processes; optimizing data delivery; re-designing infrastructure for greater scalability.
- Work with stakeholders including the Executive, Tech, Product, Data and Design teams to optimize data driven decisions
- Keep our data separated and secure by understanding and applying best practice access policies
- Create tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader
- Ensure data integrity with thorough data quality test coverage and data validation
- Improve, create and implement CI/CD processes and tooling inside the data lake
Required Qualifications :
- Join immediately
- Databricks
- Databricks hands on experience (esp.
Pipelines
PySpark & SQL, Delta Lake, Unity Catalog, DLT-)
- Ingesting & transforming data using Databricks - Min 3 years continuous hands-on
- Advanced SQL and Python knowledge
- Data Lake housing and Modelling experience (e.g. medallion architecture & query performance optimization)
- Experience with backend systems ERP, CRM (e.g. SAP, Oracle, Salesforce)
- Exposure of working with Azure (ADLS, ADF)
- Good Communication skills
Vdart Software Services