Shubham Kashyap

Hi, I'm Shubham Kashyap Data Engineer

Data Engineer with hands-on experience in building end-to-end pipelines using AWS, Apache Kafka, Spark, and Python for scalable analytics and real-time data processing.

Hire me See my Projects

My Project

YouTube Data Analysis

Built an end-to-end data engineering pipeline to ingest YouTube metadata, automate ETL, and deliver actionable analytics dashboards.

  1. Python
  2. AWS S3
  3. AWS Glue
  4. Amazon Athena
  5. AWS Lambda
  6. Power BI
End-to-End Data Engineering Pipeline
YouTube Data Analysis

Stock Market Real-Time Data Streaming

Developed a near real-time stock data streaming pipeline using Apache Kafka on AWS with persistent storage and SQL-based analytics.

  1. Apache Kafka
  2. Python
  3. AWS EC2
  4. Amazon S3
  5. AWS Glue
  6. Amazon Athena
Apache Kafka on AWS
Stock Market Real-Time Data Streaming

IPL Data Analysis

Built a distributed analytics pipeline using Apache Spark to process IPL historical data, generate team and player insights, and optimize batch performance.

  1. Apache Spark
  2. PySpark
  3. Databricks
  4. Spark SQL
Apache Spark Distributed Analytics Pipeline
IPL Data Analysis

Experience

Senior Consultant

JMR Infotech · Full-time May 2025 - Feb 2026 · Bengaluru
  • Led client-facing data engineering workstreams and aligned solution design with business goals.
  • Designed and optimized scalable ETL workflows for high-volume enterprise data pipelines.
  • Improved data quality and reliability using validation checks and operational monitoring.
  • Partnered with cross-functional teams to accelerate delivery and reduce production issues.

Senior Technical Consultant

Northern Trust · Full-time Oct 2024 - May 2025 · India (Hybrid)
  • Delivered technical consulting for enterprise data systems across regulated financial workloads.
  • Supported modernization initiatives by improving integration patterns and data flow performance.
  • Worked with stakeholders to translate reporting and analytics needs into implementable solutions.

Associate

PwC India · Full-time May 2022 - Sep 2024 · Bengaluru, India
  • Built and supported analytics-focused data pipelines for advisory engagements.
  • Developed reusable SQL and Python components to standardize data transformation logic.
  • Collaborated with consulting and engineering teams to deliver data-driven client outcomes.
  • Contributed to testing, documentation, and handover practices for stable production rollout.

OFSAA Technical Consultant

Profinch · Full-time Jun 2021 - Apr 2022 · Bengaluru, India
  • Implemented and customized OFSAA components for financial data processing use cases.
  • Supported end-to-end data mapping, reconciliation, and regulatory reporting requirements.
  • Resolved production issues and optimized batch execution performance in banking workflows.

Associate Software Engineer

JMR Infotech · Full-time Aug 2019 - May 2021 · Bengaluru, India
  • Developed backend and data processing features for enterprise software solutions.
  • Created and maintained scripts for data ingestion, validation, and transformation tasks.
  • Worked closely with senior engineers to debug issues and improve delivery quality.

Certifications

Certification details can be added here

Add your latest data engineering, cloud, and analytics certifications in this section.

My Toolkit

Python Java MySQL AWS GCP Kafka PostgreSQL Docker Git GitHub Oracle PySpark Databricks Looker Studio Databricks

Send Message

Available for hire

As a Data Engineer, I build robust data pipelines and analytics systems with a focus on reliability, scalability, and business impact.

Reach me at sk.skashyap1998@gmail.com or call +91 9008154717.

Shubham Kashyap