大数据开发工程师（湖仓/ airbus）

Requested Services: ● Design, build, and maintain high-throughput, fault-tolerant local data platform while constructing efficient data processing & analysing jobs with in-depth data monitoring and tuning functionality for online services. ● Build, deploy, and optimize enterprise-grade scalable database and object storage solutions across private cloud or on premise nodes. Conduct indexing optimization, and partition adjustments to handle aerospace workloads. ● Execute the hands-on testing, benchmarking, and implementation required to migrate existing storage layouts to a modern Apache Iceberg lakehouse architecture include but not limited to query efficiency, strict time-travel, ACID and schema evolution controls. ● Build and maintain the foundational platform services that parse, chunk, and extract structured intelligence from complex, unstructured documents into machine-readable structured semantic enabled assets, contribute to highly accurate information retrieval. ● Integrate fragmented, localized datasets storage unit into a unified internal Data Federation, minimizing latency and standardizing cross-platform query access, enable auto database provisioning & resource allocation. ● Implement end-to-end data labeling system, managing annotation lifecycles and ensuring high-quality baseline datasets & ground of truth are generated for predictive AI ready. ● Support on the metrics & data quality foundation implementation for AI trustworthy and engage in traditional data analysis topics.

Deliverables: ● Local Data Platform Construction: Establishing Airbus China local stable, efficient & unified data facade strategy & platform. ● End-to-End Iceberg based lakehouse foundation with high-Performance Data APIs: Delivering stable and efficient data access interfaces for business users via high-performance data API services with strict ACID enhancement. ● Local Data Warehouse Contribution: Serving as a key contributor to the local data warehouse construction, completing the setup of unified crucial storage and computing, and establishing the local metadata management system. ● In-depth data quality gate and monitoring functionality. Establish ground of truth and trustworthy metrics repository based on data labelling framework.

KPIs: ● Local Data Platform Efficiency, Fault Tolerance, and Stability ● Data Analysis foundation with Quality, metrics & labelling framework ● Backend Services &

• Data Access (High-Performance ACID APIs) ● Local Iceberg based Lakehouse &
• E2E data federation centre ● Data base as a service to auto provision & allocation facade ● Structured & unstructured knowledge extraction functions ● Data Catalog & meta data management system Qualifications: ● Education: Bachelor’s or Master’s degree in Computer Science, Software Engineering, Data Science, or a related quantitative field. ● Experience: ○ 5 years of hands-on data platform engineering, or ML infrastructure. At least 2-3 years of hands-on experience in managing large-scale data platforms and integrating them with data insight. ● Technical Skills: ○ Storage &
• Lakehouse: Practical hands-on experience with ETL ecosystem & object storage frameworks and modern Lakehouse structures, specifically Apache Iceberg &
• Parquet. ○ Database Management: Hands on in HIVE and relational databases (PostgreSQL, MySQL), NoSQL, timeseries or vector databases. ○ Data Federation: Hands-on experience working with federated query engines or building automated Database-as-a-Service (DBaaS) control planes. ○ Scripting &
• Parsing: Advanced proficiency in Python and expert-level SQL (advanced profiling and tuning).

Experience writing robust document parsers, structured & unstructured data mapping. One shell scripting is a big plus. ○ Data Quality: Solid hands on experience of Data Quality system establishment, familiar with data analysis and data labelling middleware. Hands on experience on Metrics foundation establishment & data analysis insight in the real production layer. ○ Experience with DataOps or MLOps is a big plus. ○ Experience with Docker and AWS or Huawei cloud stacks is a big plus. ● Basic Skills: ○ Excellent oral and written English communication skills for technical documentation and global collaboration. ○ Exceptional skills in both written and verbal presentations, with the ability to bridge the gap between technical and non-technical stakeholders. ○ Excellent interpersonal skills and team spirit to collaborate effectively within cross-functional teams. ○ A commitment to a Continuous Improvement Mindset, staying updated with evolving technologies in the fast-paced AI/Big Data landscape.