Fullstack Data Engineer -(Telecom domain)

Job Summary:

• Location &
• Work Mode: Hybrid role; applicants must be local to Texas (TX) or New Jersey (NJ).
• Industry Experience: Must have recent experience in the telecom domain.
• Role Overview:
• Responsible for designing and implementing a real-time data platform using Medallion Architecture.
• Manage end-to-end development of data pipelines from ingestion to consumption-ready output.
• Act as a “Full Stack” data professional, handling infrastructure automation (DataOps) and complex data modeling.

Key Responsibilities:

• Build scalable real-time ingestion pipelines using Spark Structured Streaming and Auto Loader for sources like Kafka/Event Hubs/CDC.
• Develop ELT logic to cleanse, deduplicate, and normalize data into relational formats with ACID compliance.
• Design optimized JSON data structures in Delta tables for low-latency API consumption.
• Implement and manage workflows using Delta Live Tables (DLT), streaming tables, and Databricks Workflows.
• Enforce data governance and security using Unity Catalog for access control and maintain a searchable data catalog.
• Own and automate deployment lifecycle with Databricks Asset Bundles (DABs) and CI/CD tools (GitHub Actions/Azure DevOps).
• Optimize streaming processes for performance and cost efficiency.

Required Skills &

• Qualifications:
• Technical Core:
• Expert in PySpark/Scala, Spark internals, and Delta Lake features (Z-Ordering, Liquid Clustering, CDF, Time Travel).
• Hands-on experience with advanced streaming patterns (watermarking, checkpoints, handling late data).
• Data Modeling &
• Languages:
• Advanced SQL skills for complex transformations.
• Mastery in handling and generating complex nested JSON objects in Spark.
• Experience with Medallion Architecture (Bronze, Silver, Gold layers) and data contracts.
• Full Stack &
• DevOps:
• Proficient in automating deployments (CI/CD, Git-based workflows).
• Capable of setting up observability/monitoring (Databricks SQL Alerts, Grafana).
• Soft Skills:
• Strong architectural thinking to balance cost and latency in streaming solutions.
• Client-focused mindset, understanding API consumption needs for JSON Gold layer.

Other Requirements:

• Must be able to work in a hybrid setup and be located in TX or NJ.
• Recent hands-on experience in the telecom industry is required.