Suman Majhi

Data Engineer with 3 years of hands-on experience designing and building cloud-based Lakehouse pipelines on Databricks using the Medallion Architecture (Bronze, Silver, Gold). Skilled in ingesting semi-structured JSON data from Azure Data Lake Storage (ADLS) Gen2, implementing data quality and error-handling frameworks, and delivering auditable, production-grade pipelines with SCD Type 2 historization, watermarking, and metadata-driven configuration.

Data Engineer at Intechfy Solutions Pvt Ltd (2023-10 – 2026-08)

• Developed and maintained data pipelines using Azure Databricks, PySpark, SQL, ADLS Gen2, and Delta Lake for processing JSON-based patient data through Bronze, Silver, and Gold layers.
• Worked on the Medallion Architecture to handle raw ingestion, data cleansing, quality validation, and business transformations, reducing unnecessary data processing by approximately 40% through incremental load mechanisms.
• Used Databricks Auto Loader with checkpointing to process newly arriving JSON files from ADLS Gen2, reducing manual file-processing effort by approximately 70%.
• Handled nested JSON structures using the VARIANT datatype and extracted required attributes from complex patient details using JSON path expressions.
• Implemented automated data quality checks for mandatory fields, registration ID length, member/Kaiser ID validation, and null or blank values, reducing invalid records reaching downstream reporting by approximately 30%.
• Developed reprocessing workflows using business keys, reprocess flags, and watermark logic to process corrected source files without reloading historical data, reducing unnecessary processing by approximately 50%.
• Implemented SCD Type 2 logic in the Gold layer using Delta Lake MERGE, effective/expiry dates, and current-record indicators to maintain historical changes.
• Maintained audit and logging for batch IDs, run IDs, execution times, source and target counts, rejected records, pipeline status, and error details, helping reduce troubleshooting effort by approximately 30%.
• Managed pipeline parameters through configuration tables for source paths, target tables, catalog/schema details, load types, and watermark settings, reducing environment-specific configuration effort by approximately 60% across DEV, QA, and PROD.

Bachelor of Science – R.C.M Science College, Khalikote, Odisha (2019-08 – 2023-01)