Suman Majhi
Data Engineer with 3 years of hands-on experience designing and building cloud-based Lakehouse pipelines
Talent
BengaluruMember since September 23, 2026
PySparkSQLSpark SQLAzure DatabricksADLS Gen2Delta LakeUnity CatalogData IngestionData QualityGit
·
Hire this person
Send a job offer directly to this candidate
PySparkSQLSpark SQLAzure DatabricksADLS Gen2
About
Data Engineer with 3 years of hands-on experience designing and building cloud-based Lakehouse pipelines on Databricks using the Medallion Architecture (Bronze, Silver, Gold). Skilled in ingesting semi-structured JSON data from Azure Data Lake Storage (ADLS) Gen2, implementing data quality and error-handling frameworks, and delivering auditable, production-grade pipelines with SCD Type 2 historization, watermarking, and metadata-driven configuration.
Experience
Data Engineer at Intechfy Solutions Pvt Ltd (2023-10 – 2026-08)
- Developed and maintained data pipelines using Azure Databricks, PySpark, SQL, ADLS Gen2, and Delta Lake for processing JSON-based patient data through Bronze, Silver, and Gold layers.
- Worked on the Medallion Architecture to handle raw ingestion, data cleansing, quality validation, and business transformations, reducing unnecessary data processing by approximately 40% through incremental load mechanisms.
- Used Databricks Auto Loader with checkpointing to process newly arriving JSON files from ADLS Gen2, reducing manual file-processing effort by approximately 70%.
- Handled nested JSON structures using the VARIANT datatype and extracted required attributes from complex patient details using JSON path expressions.
- Implemented automated data quality checks for mandatory fields, registration ID length, member/Kaiser ID validation, and null or blank values, reducing invalid records reaching downstream reporting by approximately 30%.
- Developed reprocessing workflows using business keys, reprocess flags, and watermark logic to process corrected source files without reloading historical data, reducing unnecessary processing by approximately 50%.
- Implemented SCD Type 2 logic in the Gold layer using Delta Lake MERGE, effective/expiry dates, and current-record indicators to maintain historical changes.
- Maintained audit and logging for batch IDs, run IDs, execution times, source and target counts, rejected records, pipeline status, and error details, helping reduce troubleshooting effort by approximately 30%.
- Managed pipeline parameters through configuration tables for source paths, target tables, catalog/schema details, load types, and watermark settings, reducing environment-specific configuration effort by approximately 60% across DEV, QA, and PROD.
Education
Bachelor of Science – R.C.M Science College, Khalikote, Odisha (2019-08 – 2023-01)
