Python/PySpark Engineer

Job Title

Python/PySpark Engineer

Skill

Digital : Python

Minimum Experience

10 - 15 Years

Qualification

BACHELOR OF COMPUTER SCIENCE

Location

Jersey City, NJ

Job Description

Must Have Technical/Functional Skills

Technical Skills

• Strong proficiency in Python, packaging, dependency management, and virtual environments.
• Hands-on experience with PySpark, including Spark performance tuning (partitioning, caching, broadcast joins, memory optimization).
• Expertise in data ingestion (batch/stream), schema management, and robust error-handling/retry logic.
• Solid unit and integration testing practices, including data quality validations.
• Experience with CI/CD pipelines (Azure DevOps/Jenkins), Git branching strategies, and artifact versioning.
• Working experience with Cloudera/Hadoop (HDFS, Spark, Hive/Impala) and Databricks (Delta Lake, clusters, jobs, notebooks).
• Knowledge of observability techniques: structured logging, metrics, tracing, and debugging in distributed systems.
• Secure coding practices including secrets management, PII protection, and compliance-aware development.

Functional Skills

• Strong documentation discipline for frameworks, reusable components, and best-practice patterns.
• Effective collaboration with Cloud Architects and Data Ops to ensure stable and supportable pipelines.
• Clear communication of technical ideas and solution approaches.
• Comfort working in Agile environments with iterative development and frequent releases.

Roles & Responsibilities

• Design and develop scalable Python/PySpark ingestion and transformation pipelines.
• Implement schema evolution logic, validation frameworks, and resilient error-handling mechanisms.
• Optimize Spark jobs for performance, cost efficiency, and production readiness.
• Integrate all jobs into automated CI/CD pipelines, ensuring versioning and release governance.
• Work closely with Ops teams to ensure proper monitoring, logging, and operational supportability.
• Participate in Agile ceremonies, sprint planning, code reviews, and demo sessions

TCS Employee Benefits Summary:

Discretionary Annual Incentive.

Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.

Family Support: Maternal & Parental Leaves.

Insurance Options: Auto & Home Insurance, Identity Theft Protection.

Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.

Time Off: Vacation, Time Off, Sick Leave & Holidays.

Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

KV#