Skip to main content

Data Engineering Manager - Data Platforms

Technology
Avisoft
1 month ago

Job description

Job Overview:

We are looking for a Data Engineering Manager with 10 - 15 years of experience to lead data engineering initiatives, build scalable data pipelines, manage data lake architecture, and enable real-time analytics and AI use cases. The role will work closely with application, ML, analytics, and business teams while mentoring data engineers and driving modern engineering practices.

Key Responsibilities:

  • Lead real-time pipelines using Kafka/MSK, Debezium CDC, and Apache Pinot.
  • Design and maintain Java/Python-based frameworks for ingestion, validation, transformation, and replication.
  • Design and manage data lakes using Apache Iceberg, AWS Glue, Athena, and S3.
  • Define data modeling standards across Pinot, Redshift, and warehouse layers.
  • Lead orchestration and workflow platforms using Apache Airflow and Superset.
  • Drive cloud-native deployments using ECS/EKS, Docker, Terraform/CDK, and CI/CD.
  • Support ML, analytics, product, and AI initiatives with reliable data solutions.
  • Ensure data quality, observability, lineage, governance, and security across the data stack.
  • Mentor data engineers and promote modern architecture and engineering best practices.

Required Skills:

  • Real-Time Data: Kafka/MSK, Debezium, Event-Driven Architecture, CDC
  • Data Platforms: Apache Pinot, Redshift, RocksDB/NoSQL, Apache Iceberg, Athena, AWS Glue
  • Programming: Java, Python
  • Orchestration: Apache Airflow, Superset
  • AWS: S3, IAM, Lambda, Glue, ECS/EKS, CloudWatch
  • Data Architecture: Data Lakes, Streaming & Batch Processing, Data Modeling, Parquet, Avro
  • DevOps: Docker, Terraform/CDK, CI/CD, GitHub Actions

Good to Have:

  • dbt, Trino/Presto, ClickHouse, or Druid.
  • Data security, encryption at rest/in transit, GDPR/PCI compliance.
  • CloudFormation and advanced DevOps practices.
  • Experience with scalable cloud-native data architectures.
Keywords
OrchestrationClickHouseOCamlApache KafkaRocksdbCDKRedshiftPrestoApache DruidApache AirflowDevOpsAirflowPythonApache ParquetApache LicenseApache Http ServerApache AvroJavaCI/CDReplication

Interested in this role?