Harpreet Singh

Senior Software Engineer at Tripstack

Develop and support the Virtual Interline product using Golang, SQL, Python, and Kafka (Redpanda). Implement alerts and visualizations with Grafana and Prometheus. Design new features and enhancements in collaboration with business stakeholders. Improve performance using graph technology (Memgraph). Manage two offshore resources. Lead sprint planning, daily standups, and code reviews. Migrated all applications from Kafka to Redpanda.

Data Engineer II at Amazon - AWS

Provided code review feedback and technical direction for sprint tasks. Developed ETL solutions using Spark, Python, Redshift, JupyterHub and EMR. Optimized EMR data processing, reducing query runtimes and EMR usage by 80%. Conducted root cause analysis on failed jobs. Moved billing analytics from Redshift to Spark and tuned jobs to finish in ~30 minutes. Tuned Spark jobs that previously ran 6+ hours to complete in under 1 hour. Migrated MP insight reports from Hammerstone to Servimos. Implemented data validation on Redshift using FK/PK and business rules. Implemented production monitoring for data analytic jobs and provided fixes.

Data Engineer II at Paytm

Develop streaming ETL solution using Flink and Kafka. Design and implement reporting for Lending platform. Build ETL pipelines using Spark, Scala, Python, EMR, Elasticsearch, AWS, Kafka, Azkaban, Aurora; added features in Growth Marketing ML modeling project; manage data pipelines for GM ML team. Improve ETL performance, monitor and fix Spark workflows. Integrate Clevertap and export required data attributes to Clevertap as scheduled jobs.

Data Engineer at Tripstack

Design data pipeline for Virtual Interline product. Identify bottlenecks in data pipeline and scale it for better performance. Develop ETL code using Golang, Python, Java, Kafka and PostgreSQL. Develop high performing VI-API code using Golang for hub transfer itineraries. Tune Kafka partitions, PostgreSQL parameters, and JSON marshalling for high throughput and low latency. Deploy application using GitLab, Terraform, Kubernetes and EC2. Monitor data pipeline using Kibana, Kafka Manager, AWS dashboards and Athena. Create dashboards using Druid, Turnillo and Superset. Setup shell scripts and Golang jobs to feed data continuously into Druid from Kibana and PostgreSQL. Administer Druid, Tornillo, Superset and Neo4j. Install and monitor Neo4j cluster in GCP. Perform Neo4j POC for virtual interlining with multiple hubs.

Sr. Hadoop Administrator at Cognizant - HBC

Administration of Cloudera clusters including user management, data management and capacity management. Tune Kafka topics for better performance. Review and tune Spark jobs on Cloudera Cluster. Manage access and users on Cloudera cluster. Resolve user connectivity issues. Upgrade Cloudera cluster. Benchmark Cloudera cluster and Kafka performance. Manage instances of EC2 in AWS.

Spark Developer at Accenture - Metrolinx

Designed and implemented data pipelines using Spark, Scala, Python and Unix shell scripting. Processed data streams according to business logic for rail corridor dashboards. Consulted with business on requirements. Tuned Spark jobs on Cloudera environment. Ingested data from Oracle, Excel and CSV files into Hadoop. Dashboards created using Tableau for required business metrics.

Hadoop Administrator at Infernotions

Developed Ansible scripts using blueprints to deploy Cloudera cluster on Proxmox containers as edge, master and data nodes with specific roles. Administered Cloudera Hadoop cluster. Integrated Livy and Zeppelin with Hadoop cluster and implemented authorization. Resolved Big Data platform issues and performed performance tuning. Resolved data integration issues and access issues with different components using LDAP and Shiro. Integrated external data analysis and visualization components like Zeppelin, Tableau, R and sparklyr securely with Cloudera cluster and automated requests for new clients. Designed hot and cold Hadoop storage for Spark. Developed data ingestion from multiple systems using Hive and Sqoop. Defined data security using Sentry roles and separate work areas in HDFS/Hive. Automated DR for Cassandra, Hive and HDFS using shell scripts on non-enterprise version of Cloudera. Managed user permissions and data backups.

Big Data Architect/ Senior Manager at Bank of America

Led India HaaS architecture team supporting platform management, strategic services and application governance. Reviewed Spark, Kafka and Hive applications for production deployment. Contributed to elastic compute research for large-scale deployments. Developed best practices and provided feedback to developers. Resolved issues related to Impala, Hive, Sqoop and Spark in agile mode. Extracted and logged metrics from Spark and Hadoop job logs. Resolved L2-L3 incidents on Hadoop platform. Collaborated with Cloudera support for enhancements and platform issues.

Software Designer | Hadoop Lead at Royal Bank of Scotland

Designed data lake structure and security/access patterns. Developed Hadoop streaming application for capacity management, compression and data archival. Prepared white papers on query optimization using Hive and Impala. Administered Cloudera cluster for POC environment. Designed and implemented security model for Oozie-Sqoop actions using KeyStore. Implemented Hadoop cluster security (Sentry, AD groups, ACLs, Kerberos). Developed Hadoop usage reports using Cloudera Navigator, Hive and SAS Visual Analytics; defined publication process.

Assistant Consultant at Tata Consultancy Services Ltd

Led team of 6 developers; designed components, created HLD/LLD, presented for approvals. Converted SAS models to Teradata SQL and performed performance tuning. Led ETL development using Informatica and history load using TPT. Managed data migration projects and designed tactical LDM/PDM for multiple applications. Led Universe and Reports migration for Business Objects.

Technical Consultant at Teradata Corporation

Analyzed bad queries and performed tuning opportunities. Maintained statistics, built indexes, and captured TSET. Monitored sessions for skew, CPU, spool, and blocking. Developed ETL scripts using TPT, UNIX shell scripting, Informatica. Created reports and dashboards; followed client SDLC for production deployment. Worked on Teradata, SQL development, and data movement tooling.

Software Engineer at Accenture Services

Supported Barclays Wire Transfer Application. Identified and fixed issues using C++ and UNIX. Enhanced GLASS application. Debugged and resolved issues per change management.