shradhanjali pradhan

Architected end-to-end Python and SQL data pipelines to ingest, transform, validate, and curate enterprise datasets, establishing reliable foundations for analytics and downstream applications. Developed scalable PySpark and Databricks pipelines processing 2M+ enterprise documents, standardizing structured/unstructured data into curated datasets for reporting and consumption. Built incremental ETL/ELT workflows with reusable transformation logic, data validation, schema standardization, and error handling to improve consistency across processing jobs. Designed distributed data processing workflows with Databricks and PySpark, optimizing transformations, partitioning, and resource utilization for high-volume datasets. Implemented production data-quality and monitoring workflows using AWS CloudWatch and Prometheus, tracking pipeline health, processing latency, failures, and data issues while reducing detection time from 4 hours to 20 minutes. Built AI-ready pipelines using Python, LangCh

Built scalable ETL pipelines using Python, PySpark, and Databricks to transform raw data into accurate Delta Lake tables for analytics and reporting. Developed dbt models and automated data tests/documentation, standardizing business metrics and reducing reporting inconsistencies by 25%. Built near-real-time pipelines using AWS Kinesis, Lambda, and S3 to process time-sensitive data and reduce operational decision latency by 30%. Improved PostgreSQL query performance through indexing, partitioning, and query optimization. Integrated external systems via REST APIs to automate ingestion and reduce manual data transfers. Used AWS Athena and RDS for querying operational datasets supporting ad-hoc analysis and downstream reporting. Implemented data validation and monitoring workflows with Apache Airflow and Python, achieving 99.5% pipeline reliability and reducing production data incidents by 35%. Partnered with BI/analytics/operations teams to define data requirements and resolve data issue

Built scalable ETL pipelines with Python, PySpark, and Databricks to transform raw data into accurate Delta Lake tables for analytics and reporting. Developed dbt models and automated data tests and documentation, standardizing business metrics and reducing reporting inconsistencies by 25%. Implemented near real-time pipelines using AWS Kinesis, Lambda, and S3 to process time-sensitive data and reduce operational decision latency by 30%. Improved PostgreSQL query performance through indexing, partitioning, and query optimization, reducing processing time for frequently accessed datasets. Integrated external systems via REST APIs, automating data ingestion and reducing manual transfers between platforms. Used AWS Athena and RDS to query and manage operational datasets for ad-hoc analysis and downstream reporting. Built data validation and monitoring workflows with Apache Airflow and Python, achieving 99.5% pipeline reliability and reducing production data incidents by 35%.