Role Overview :
As a Data Engineer with AI Application expertise, you will serve as a bridge between raw data infrastructure and advanced machine learning deployment. You will work closely with data scientists, software architects, and business stakeholders to design robust pipelines that feed AI models and power real-time analytics. Your daily contributions will directly influence the scalability of our clients' AI initiatives, ensuring that data is not only accessible but also optimized for high-performance model training and inference in production environments.
Key Responsibilities :
- Prepares and pipelines enterprise data required for AI applications.
- Owns ingestion, transformation, metadata, quality controls, retrieval preparation, and data access patterns.
- Ensures source data used for grounding is trustworthy, governed, and fit for operational use.
- Supports structured and unstructured data flows needed for AI applications.
- Also manages data pipelines to purchased AI applications such as Sinequa Enterprise Search and other adjacent AI eco systems like ChatGPT and MS365 CoPilot, manages the knowledge assets used for grounding models, and supports the management of unstructured databases and repositories used for retrieval and grounding.
- Data engineering, ETL/ELT, SQL, Python, data modeling, data quality, metadata/lineage, document processing, vectorization pipeline concepts, AWS data services, secure data access design, enterprise search/data integration, management of unstructured data stores and repositories.
Required Skillset :
- Demonstrated expertise in designing and managing big data ecosystems using tools like Spark, Kafka, and cloud-based data warehouses such as Snowflake or BigQuery.
- Proven ability to implement MLOps practices, including model versioning, deployment, and monitoring, to bridge the gap between data engineering and AI application development.
- Strong proficiency in Python or Scala for data manipulation and building production-grade backend services.
- Excellent communication skills with the ability to articulate technical data strategies to non-technical stakeholders and lead collaborative discussions.
- A Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field, supported by 6 to 11 years of hands-on experience in data engineering.
- Ability to thrive in an on-site work environment in Ahmedabad, demonstrating high adaptability to evolving project requirements and team dynamics.
Pyramid IT Consulting