Mayank Raj

Experienced Software Engineer with a strong background in backend systems, distributed data pipelines, cloud infrastructure, and large-scale data processing. Over the past several years, I’ve worked across fast-paced teams building scalable and reliable systems using AWS, Kafka, Spark, Kubernetes, PostgreSQL, Databricks, and Apache Hudi.

Currently at Cardlytics, I work on high-scale transaction and merchant data systems, building end-to-end pipelines, REST APIs, and data reconciliation workflows that process billions of records and improve financial accuracy. Previously, I contributed to real-time streaming infrastructure, observability platforms, ETL systems, and cloud-native services at companies including Viasat and Capgemini.

I enjoy solving complex engineering problems involving scalability, reliability, performance optimization, and distributed systems, while collaborating closely with cross-functional teams to deliver impactful solutions.

Software Engineer II at Cardlytics Inc (2024-05 – Present)

• Implemented scalable data pipeline end to end to onboard American Express merchant and transaction data, leveraging AWS EMR, Apache Hudi, Athena. Resulting in a streamlined data integration process.
• Designed and enhanced RESTful API's across critical services to support dynamic mapping for streamlined Amex merchant onboarding.
• Integrated complex backend logic to handle mapping, validation and relationship management for brands and categories using asynchronous workflows.
• Collaborate cross-functionally with the data engineering and product teams to ensure API compatibility with downstream systems, resulting in seamless ingestion and retrieval of structured metadata.
• Expanded backend responsibilities despite limited exposure to C#, becoming a key contributor to high-impact initiatives by driving production-grade improvements.
• Revamped Spark jobs to optimize resource usage, transitioning from individual to shared cluster on AWS EMR, leading to substantial reduction in operational costs and faster execution times. Implemented comprehensive DAG monitoring using CloudWatch, Datadog.
• Designed and implemented Delta streamer tables to seamlessly ingest data from Kafka into Apache Hudi tables, serving as a trusted data source for downstream analytics and reporting.
• Analyzed and developed algorithms for brand reconciliation leveraging multiple strategies, laying the groundwork for a robust source of truth to drive data accuracy and consistency.
• Created end-to-end pipeline leveraging multiple data sources to come up with a reporting table which serves as source of truth for the reporting/analytics team to forecast and react to spend anomalies faster. The spark job handles more than 30 billion rows of data.
• Led efforts to resolve anomaly detection discrepancies between forecasted and actual spend, significantly mitigating overspend within the redemption network and enhancing financial accuracy.

Software Engineer at Selectiva Systems Inc (2023-07 – 2024-05)

• Analyzed logs from Kibana to identify the major source of errors and bottlenecks across our services.
• Identifying server and user-related issues that can lead to quicker resolution of support-related tickets.
• Utilized AWS Lambda to monitor and raise alerts through CloudWatch and created Grafana dashboards for Prometheus-based monitoring.
• Developed and managed streaming data pipelines using Kafka, and Spark Streaming.
• Enabling real-time analytics, monitoring, and alerting to gain insights and make data-driven decisions.
• Utilizing AWS services like Kinesis for data streaming, S3 for storage, Lambda for serverless computing, and EMR (Elastic MapReduce) for running Spark jobs.
• Applied transformations, aggregations, and enrichments to the streaming data. Used Databricks, PySpark to achieve it.
• Creating and maintaining large ETL pipelines for internal services, using Databricks, PySpark, Spark SQL, Delta Tables.

Software Engineer at Viasat Inc (2020-06 – 2023-06)

• Played a key role in designing and implementing robust infrastructure for data streaming services, ensuring smooth data flow, while prioritizing stability and high-performance standard which included on-call rotation to ensure uninterrupted service availability.
• Designed and implemented RESTful APIs and CLIs that enabled clients to interact with different services effectively and securely.
• Conducted regular performance analysis and optimization of the streaming infrastructure to enhance its efficiency and reliability.
• Utilized Prometheus and Grafana to implement comprehensive monitoring solutions, custom dashboards to provide real-time visibility into system performance, resource utilization, health and proactive measures to prevent downtime.
• Led the end-to-end lifecycle of Spark jobs, including job creation, optimization, testing, and deployment in a distributed environment along with successful migration from Mesos to Kubernetes, ensuring minimal disruption and leveraging Kubernetes orchestration.
• Setup, managed Kubernetes clusters, including creating, configuring namespaces and monitoring resource allocation.
• Developed, maintained scripts for managing Kubernetes and containerized environments, automating deployment, scaling, and monitoring processes.

Application Research and Developer at UIC Innovation Center (2019-05 – 2020-05)

• Led the development of an algorithm, conducted data analysis, and created a proof of concept for a device designed to enable continuous monitoring of Intracranial Pressure (ICP) using Brainstem Auditory Evoked Potential (BAEP).
• Collaborated closely with a multidisciplinary team of researchers, engineers, and neurosurgeons to design and refine the algorithm for accurate and real-time ICP monitoring.

Software Engineer at Aricent Inc (now Capgemini) (2016-01 – 2018-07)

• Designed and Developed Training Management System, used to evaluate incoming employees. TMS automated entire process, reduced human interference and expedited the evaluation process. (JavaFx and C++)
• Conceptualized, designed, and developed a new website for MarginPoint, driving more than a 30% increase in new views and demo requests for clients enhancing their online presence. (WordPress, JavaScript, HTML/CSS, CRM)
• Designed and implemented a new test suite for a web application, ensuring comprehensive coverage of functional features.
• Led offshore QA team, expedited bug discovery using new test suite. Managed coordination with on-site team.
• Received the Best Team Collaborator award for exceptional contributions to a critical feature release at MarginPoint.
• Developed and presented a proof of concept for CI/CD utilizing Jenkins, Ansible, and Docker technologies.

Master (MS) in Computer Science – University of Illinois

Bachelor of Engineering in Electronics and Communications – Visvesvaraya Technological University (VTU)