AWS Glue Python Developer

Key Responsibilities :

• Design and develop inbound and outbound file processing using AWS Glue, Python, and PySpark.
• Develop reusable Python modules, utilities, and frameworks for data ingestion, validation, transformation, and error handling.
• Build batch inbound and outbound file processing using Amazon S3 and AWS Glue.
• Configure Amazon EventBridge rules for scheduling and event-based processing.
• Develop AWS Lambda functions for lightweight processing, automation, validation, and integration activities.
• Implement messaging and notification solutions using Amazon SNS and Amazon SQS.
• Configure Amazon CloudWatch logs, metrics, dashboards, and alarms for operational monitoring.
• Develop workflows using AWS Glue Workflows, AWS Step Functions, or equivalent orchestration services.
• Integrate AWS services with relational databases, APIs, file systems, and external applications.
• Implement data ingestion from CSV, XML, JSON, Excel, fixed-length, variable-length, and delimited files.
• Develop data validation, reconciliation, audit, retry, and exception-handling processes.
• Optimize AWS Glue jobs for performance, scalability, and cost efficiency.
• Implement secure access using AWS IAM roles, policies, encryption, and secrets management.
• Support production deployments, incident resolution, root-cause analysis, and performance tuning.
• Collaborate with DevOps teams to implement CI/CD pipelines for AWS Glue jobs and related infrastructure.
• Prepare technical documentation, data flow diagrams, operational procedures, and support documentation.

Qualifications :

Required Skills and Experience :

• 8 - 12 years of experience in Python development and data engineering.
• Strong hands-on experience with AWS Glue and PySpark.
• Excellent knowledge of Python programming, object-oriented programming, and exception handling.
• Experience designing ETL pipelines and data processing frameworks.
• Strong experience with Amazon S3, including bucket structures, lifecycle policies, partitioning, and file formats.
• Experience with Amazon EventBridge for scheduled and event-driven execution.
• Experience developing and deploying AWS Lambda functions using Python.
• Experience with Amazon SNS and SQS for notifications, messaging, and asynchronous processing.
• Experience with Amazon CloudWatch for logging, monitoring, alerting, and troubleshooting.
• Understanding of AWS IAM, security policies, encryption, and least-privilege access.
• Experience working with relational databases such as Oracle, PostgreSQL, MySQL, or Amazon RDS.
• Strong knowledge of SQL and database connectivity from Python or AWS Glue.
• Experience working with REST/SOAP APIs and external system integrations.
• Understanding of JSON, XML, CSV, Excel, and fixed-width file formats.
• Experience with Git and software development best practices.