Arkhya Tech. Inc.Databricks Data Engineer with DevOps Skills
We are looking for an experienced Databricks Data Engineer with strong DevOps expertise to join our data engineering team. The ideal candidate will design, build, and optimize large-scale data pipelines on the Databricks Lakehouse platform while implementing robust CI/CD and deployment practices.
This role requires strong skills in PySpark, SQL, Azure cloud services, and modern DevOps tooling. You will collaborate with cross-functional teams to deliver scalable, secure, and high‑performance data solutions.
Strong hands-on experience with Databricks, including:
Proficiency in PySpark, Spark, and advanced SQL.
Expertise with Azure cloud services (ADLS, ADF, Key Vault, Functions, etc.).
Experience with relational databases and data warehousing concepts.
Strong understanding of DevOps tools:
Familiarity with infrastructure-as-code (Terraform is a plus).
Develop data processing workflows using PySpark/Spark and SQL for large‑volume datasets.
Integrate data from ADLS, Azure Blob Storage, and relational/non-relational data sources.
Implement Delta Lake best practices including schema evolution, ACID transactions, OPTIMIZE, ZORDER, and performance tuning.
Build and manage automated deployments using Databricks Asset Bundles.
Manage version control for notebooks, workflows, libraries, and configuration artifacts.
Automate cluster configuration, job creation, and environment provisioning.
Collaborate with product owners, business partners, and engineering teams to translate requirements into scalable data solutions.
Document data flows, architecture, and deployment processes.
Monitor workflows, debug failures, and ensure pipeline stability and reliability.
Implement job instrumentation and observability using logging/monitoring tools.
Enforce access controls, data security, and compliance with enterprise policies.
Ensure best practices around data quality, lineage, and auditability.
Knowledge of streaming technologies like Structured Streaming or Spark Streaming.
Experience building real-time or near real-time pipelines.
Exposure to advanced Databricks runtime configurations and tuning.
Databricks Certified Data Engineer Associate / Professional
Mid-Senior level
IT Services and IT Consulting
¿Te interesa este puesto?