Databricks Developer

About Position:

We are looking for an experienced Databricks Developer / Lead with 7+ years of experience in Data Engineering, Analytics, and Cloud Data Platforms. The ideal candidate will possess deep expertise in Databricks, Apache Spark, distributed data processing, and cloud-native data architectures. This role involves designing, developing, and optimizing large-scale data pipelines while ensuring high performance, scalability, reliability, and operational excellence across enterprise data platforms.

• Role: Databricks Developer
• Location: All Persistent Locations
• Experience: 7 to 12 years
• Job Type: Full-Time Employment

What You'll Do:

• Design, build, and optimize distributed data pipelines using Python and Apache Spark.
• Develop, manage, and maintain large-scale data processing workloads on Azure Databricks.
• Build scalable ETL and ELT solutions for data ingestion, transformation, and processing.
• Manage data storage and cloud resources using Azure Storage, Azure Container Registry (ACR), and Azure Kubernetes Service (AKS).
• Develop container-based deployment solutions for data engineering workloads.
• Orchestrate data workflows using Apache Airflow and enterprise scheduling platforms.
• Implement robust data security controls including access management, encryption, and secure credential handling.
• Support vulnerability remediation activities across data platforms using tools such as CodeQL, Cycode, and related security frameworks.
• Monitor data pipelines and troubleshoot performance, reliability, and operational issues.
• Collaborate with data architects, business analysts, and application teams to deliver business-driven data solutions.
• Maintain source code repositories using GitHub and ensure proper documentation in Confluence.
• Participate in code reviews and enforce data engineering best practices and development standards.
• Support CI/CD processes and deployment automation for data platform components.
• Contribute to Agile ceremonies including sprint planning, development activities, testing, and release management.
• Ensure compliance with organizational data governance, security, and regulatory requirements.

Expertise You'll Bring:

• 6 to 9 years of hands-on experience in Data Engineering and large-scale data processing solutions.
• Strong expertise in Python programming for data engineering workloads.
• Extensive experience building distributed data processing pipelines using Apache Spark and PySpark.
• Hands-on experience with Azure Databricks for data transformation and analytics workloads.
• Experience deploying and managing data solutions on Azure Kubernetes Service (AKS).
• Strong knowledge of Azure Storage services, Azure Container Registry (ACR), and cloud-native architectures.
• Experience with Apache Airflow and workflow orchestration frameworks.
• Understanding of ETL/ELT design patterns and data pipeline architecture.
• Experience implementing data security controls, encryption standards, and secure secret management practices.
• Familiarity with vulnerability management tools such as CodeQL, Cycode, or similar security scanning platforms.
• Experience working with GitHub, version control practices, and CI/CD pipelines.
• Strong understanding of containerization technologies and deployment automation.
• Experience troubleshooting and optimizing distributed data processing workloads.
• Strong analytical, debugging, and problem-solving skills.
• Excellent communication and stakeholder collaboration abilities.
• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.

Benefits:

• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Annual health check-ups
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment: Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

Let's unleash your full potential at Persistent - persistent.com/careers "Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind."