Skip to main content

Data Engineer

Technology
Codetech
Malvern, United States1 week ago
Service contract

Job description

Required Skills - Top 3-5 Key Words

"Required Technical Skills: Python, SQL, AWS Web services (Glu, S3, Lambda)

Core Programming Skills:

  • Expert proficiency in Python, with experience in building data pipelines and back-end
  • systems.
  • Advanced knowledge of SQL for querying and optimizing large datasets.
  • AWS Cloud Services Expertise:
  • DynamoDB, S3, Athena, GlueETL, Lambda, ECS, Glue Data Quality, EventBridge,
  • Redshift Machine Learning, OpenSearch, and RDS.
API and Resilience Engineering:
  • Proven expertise in designing fault-tolerant APIs using Swagger/OpenAPI, GraphQL,
  • and RESTful standards.
  • Robust understanding of distributed systems, load balancing, and failover strategies.
  • Monitoring and Orchestration:
  • Hands-on experience with Prometheus and Grafana for observability and monitoring.
Job Duties
  • Senior Data Engineer – 7+ Years of Experience
  • We are seeking a highly experienced Senior Data Engineer with 7+ years of expertise in
  • designing, building, and optimizing robust data solutions. The ideal candidate must
  • possess top-tier skills in Python, AWS services, API development, and TypeScript, and
  • have significant hands-on experience with anomaly detection systems.
  • The candidate should have a proven ability to work at both strategic and tactical levels,
  • from designing data architectures to implementing them in the weeds.

Job Requirements

Key Responsibilities:

  • Data Pipeline Development
  • Independently design, build, and maintain complex ETL pipelines, ensuring scalability
  • and efficiency for large-scale data processing needs.
  • Manage pipeline complexity and orchestration, delivering high-performance data
  • products accessible via APIs for business-critical applications.
  • Archive processed data products into data lakes (e.g., AWS S3) for analytics and
  • machine learning use cases.
  • Anomaly Detection and Data Quality
  • Implement advanced anomaly detection systems and data validation techniques, ensuring data integrity and quality.
  • Leverage AI/ML methodologies, including Large Language Models (LLMs), to detect and address data inconsistencies.
  • Develop and automate robust data quality and validation frameworks.
  • Cloud and API Engineering
  • Architect and manage resilient APIs using modern patterns, including microservices,
  • RESTful design, and GraphQL.
  • Configure API gateways, circuit breakers, and fault-tolerant mechanisms for distributed
  • systems.
  • Ensure horizontal and vertical scaling strategies for API-driven data products.
  • Monitoring and Observability
  • Implement comprehensive monitoring and observability solutions using Prometheus and Grafana to optimize system reliability.
  • Establish proactive alerting systems and ensure real-time system health visibility.
  • Cross-functional Collaboration and Innovation
  • Collaborate with stakeholders to understand business needs and translate them into scalable, data-driven solutions.
  • Continuously research and integrate emerging technologies to enhance data engineering practices.

Required Skills

Basic Qualification

Additional Skills

Background Check : No

Drug Screen : No

Interested in this role?