Sr. Data Engineer - Databricks

Job Title: Senior Data Engineer

Experience: 7 Years (Preferred)

Employment Type: Permanent Full time

Work Mode: Onsite

Bilingual: Japanese & English

Job Summary

We are looking for an experienced Senior Data Engineer with strong expertise in Databricks, cloud platforms (AWS and/or Azure), and large-scale data engineering solutions. The ideal candidate will have hands-on experience designing, developing, and optimizing scalable data pipelines, implementing modern data architectures, and processing large volumes of structured and unstructured data.

The candidate should be proficient in Apache Spark, Python, SQL, and cloud-based data services, with the ability to deliver reliable, high-performance data solutions.

Key Responsibilities

Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.

Build and optimize ETL/ELT workflows for batch and streaming data processing.

Develop data ingestion, transformation, and integration solutions using PySpark and SQL.

Implement and maintain data lakehouse architectures, including the Medallion Architecture (Bronze, Silver, and Gold layers).

Work with Delta Lake, Delta Tables, and other Databricks capabilities to ensure data reliability and performance.

Develop cloud-based data engineering solutions on AWS and/or Microsoft Azure.

Integrate Databricks with cloud storage, data warehouses, and other enterprise data sources.

Optimize Spark jobs, cluster configurations, and SQL queries to improve performance and reduce costs.

Implement data quality checks, error handling, monitoring, and data validation processes.

Collaborate with data architects, data scientists, analysts, and business stakeholders to deliver data-driven solutions.

Follow best practices for data security, governance, access management, and CI/CD.

Troubleshoot production issues and ensure the reliability and scalability of data pipelines.

Required Technical Skills

Strong hands-on experience with Databricks and Apache Spark.

Proficiency in Python, PySpark, and SQL.

Experience developing complex ETL/ELT pipelines and data integration workflows.

Strong understanding of data lakehouse concepts and Medallion Architecture.

Experience working with Delta Lake, Delta Tables, and data optimization techniques.

Hands-on experience with at least one major cloud platform:

AWS: Amazon S3, AWS Glue, IAM, Lambda, Redshift, and related data services.

Azure: Azure Data Lake Storage (ADLS), Azure Data Factory, Azure Data Lake services, and related cloud data services.

Experience with batch processing and, preferably, streaming data pipelines.

Knowledge of data modeling, schema evolution, partitioning, and performance tuning.

Experience with Git, version control, and CI/CD practices.

Strong problem-solving, communication, and collaboration skills.