Rui Tong

Led architecture and production deployment of a multi-tier LLM cascading framework routing queries across Gemini Nano, Gemini Pro, and Gemini Ultra to reduce compute cost and improve latency with smart fallback strategies. Built a latency-aware model router (gRPC, TensorFlow Serving) and implemented token-level speculative decoding for streaming responsiveness. Designed a cross-model prompt harmonization and response merging layer for consistent outputs during escalation. Partnered on embeddings-based task routing for on-device vs cloud inference decisions, created offline simulation and A/B testing harnesses, and delivered observability dashboards (BigQuery, OpenTelemetry, Looker Studio). Introduced CI/CD with GitOps, Bazel, and Spinnaker plus canary/rollback validation and shadow testing. Implemented privacy/PII redaction for query logs using Google DLP API and worked with Trust & Safety to add fallback-aware safety interlocks. Mentored engineers and authored technical docs/whitepape

Built a real-time latency profiling framework for Gemini inference paths using OpenTelemetry, Cloud Logging, and Grafana to identify end-to-end bottlenecks. Created a high-throughput evaluation harness running 50,000+ daily test cases with metrics such as BLEURT, WinRate, and Trust & Safety flags. Benchmarked and deployed LoRA fine-tuning pipelines with analysis of quality vs runtime latency trade-offs. Introduced parallel decoding (speculative decoding and early exit optimizations) into the serving stack, reducing average latency by 28% in production. Refactored prompt processing with a domain-specific prompt preprocessor for faster concatenation, context truncation, and memory reuse. Built model canary infrastructure for lightweight evaluations to catch semantic/latency regressions before rollout. Established production SLAs targeting <200ms P90 latency and developed side-by-side evaluation UIs. Collaborated on multilingual evaluation and multi-agent prompting experiments to improve

Led multi-surface deployment of Gemini-powered intelligent multi-agent systems for products including Gmail Smart Reply+, Android assistant, and Workspace AI integrations. Designed a modular multi-agent runtime supporting planning, retrieval, and tool-calling to execute workflows like summarization followed by API calls and synthesis. Implemented RAG using FAISS and ColBERTv2 for grounded enterprise responses. Built secure tool-calling integrations with Google APIs (Gmail, Calendar, Drive, Maps) under controlled permissions. Developed prompt lifecycle management with versioning and automated evaluation (ROUGE, BLEURT, win/loss vs GPT-4) plus human-in-the-loop feedback pipelines. Implemented streaming interfaces (gRPC token streaming) to reduce perceived latency. Optimized agent performance using dynamic batching, quantization, and speculative decoding to reduce inference time and cost by 27%. Launched continuous evaluation integrated into CI, added guardrails for tool safety and latenc