Databricks Developer
Technology
2 weeks ago
Service contractFully remote
Job description
Job Title: Databricks Developer (3-5 Years Experience)
Duration: 3 Months contract
Location: Remote
Introduction: We are seeking a skilled Databricks Developer to join our dynamic data engineering team. This role plays a pivotal part in advancing our data analytics and machine learning initiatives by designing, developing, and maintaining scalable data pipelines and solutions on the Azure Databricks platform. The ideal candidate will bring a strong background in big data technologies and a passion for transforming raw data into actionable insights, enabling data-driven decision-making across the organization.Roles and Responsibilities:
- Design, develop, and optimize robust data pipelines and workflows using Azure Databricks and Apache Spark.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver scalable solutions.
- Implement ETL/ELT processes to ingest, process, and transform large datasets from diverse sources using SQL and Python.
- Monitor and troubleshoot pipeline performance, ensuring data quality, accuracy, and timeliness.
- Develop and maintain comprehensive documentation for data workflows, architecture, and processes.
- Integrate Databricks solutions with cloud platforms, primarily Azure, ensuring seamless data operations.
- Apply best practices in data governance, security, and compliance within the Databricks environment.
- Participate in code reviews, testing, and deployment activities to maintain high-quality deliverables.
Qualifications:
- Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field.
- 3-5 years of proven experience as a Databricks Developer or in a similar big data engineering role.
- Strong understanding of data warehousing concepts, ETL processes, and data modeling.
- Hands-on experience with Apache Spark programming using Python and SQL.
- Proficient in working with Azure Databricks and familiar with other cloud platforms such as AWS or Google Cloud.
- Knowledge of data storage solutions like Delta Lake, Parquet, and JSON formats.
- Excellent problem-solving skills and ability to work effectively in a collaborative agile environment.
- Relevant certifications such as Databricks Certified Associate Developer for Apache Spark or Microsoft Azure certifications are a plus.
Tools and Technologies:
- Azure Databricks
- Apache Spark
- Programming languages: Python, SQL (Scala knowledge is a plus)
- Cloud platforms: Microsoft Azure (primary), AWS, Google Cloud
- Data storage and formats: Delta Lake, Parquet, JSON
- Version control tools such as Git
- Workflow orchestration tools like Azure Data Factory or Apache Airflow (preferred)
Keywords
Azure DatabricksApache SparkPythonSQLETL/ELTData Pipeline DevelopmentData WarehousingData ModelingDelta LakeParquetJSONMicrosoft AzureGitAzure Data FactoryApache AirflowData GovernanceScalaData EngineeringData PipelinesAWS
Interested in this role?