Robin Kuo

Built and scaled DataDog’s data jobs monitoring platform to provide unified observability for Apache Spark, DataBricks, and AWS EMR workloads. Designed telemetry collection to aggregate Spark execution details (jobs, stages, tasks, executors) into a central monitoring platform and created query normalization/fingerprinting to visualize historical performance metrics for thousands of concurrent Spark jobs. Implemented an AI-assisted optimization recommendation system to detect data skew, excessive shuffle, and inefficient partitioning, and led integrations enabling cost correlation between database unit usage and job performance across cluster-based and serverless workloads. Achieved a 40% reduction in DataBricks job runtime for top internal customer workloads through automated recommendations and configuration tuning. Also architected BYOC logs and a federated query layer for compliant, customer-controlled log storage and search at petabyte scale, including streaming ingestion with

Built and scaled Atlassian’s internal Socrates enterprise data lake and self-service ETL platform, consolidating product events, billing, CRM, and microservice data into Amazon S3 and enabling self-service analytics for 1,000+ weekly users. Migrated event-driven ingestion from pull-based ETL to a Stream Hub/Kinesis bus using a data landing framework, reducing daily extraction failures from 20–30% to near-zero and decoupling application sources from analytics infrastructure. Orchestrated EMR clusters per job via Airflow spin-up strategies for cost attribution and easier upgrades. Contributed to TaaS (Transformation as a Service), enabling analysts to define and schedule transformations without blocking data engineering. Led large-scale migration efforts including JIRA and Confluence migration from self-hosted server to Atlassian Cloud, including space-by-space Confluence migration pipelines with binary attachment handling and distributed multi-node migration with retry/re-dispatch