Durable AI Pipeline with Temporal + Next.js

Live Demo: https://www.twine.net/signin

Context

This was built as a take-home project for a senior engineer interview. The prompt: “How would you build a durable pipeline that turns documents into AI-generated guides, handling slow/unreliable dependencies at scale (5-5,000 items)?”

I built a working application instead of a design doc.

What It Does

A document-to-guide generation pipeline with:

• Real-time progress UI showing conversion and generation stages
• Configurable failure rates to demonstrate retry behavior
• Manual retry for failed items with actionable error messages
• Pagination and filtering across thousands of guides

Technical Highlights

Temporal for Durable Execution

• Workflows survive crashes, restarts, and network failures
• Automatic retries with exponential backoff
• Activity heartbeats detect stuck operations
• Full execution history for debugging

Scale Strategy

• Parent/child workflow architecture to stay under Temporal’s 50K event history limit
• Chunked processing: 100 items per child workflow
• Bounded concurrency: 10 children × 5-10 activities
• Verified at 5,000 files processing ~760-800/minute

Idempotent Activities

• Lease-based acquisition prevents duplicate processing
• Token-gated writes ensure exactly-once semantics
• Terminal state short-circuits on retry

Production Patterns

• Postgres as queryable state for UI (Temporal orchestrates, DB serves reads)
• Failure-atomic run creation with compensation
• Graceful degradation (partial success, actionable failures)

Stack

• Next.js 14 - App router, server components, API routes
• Temporal - Workflow orchestration (TypeScript SDK)
• Postgres + Prisma - State persistence
• Railway - Deployment (5 services)

What’s Mocked

Document conversion, search, and LLM generation are simulated with configurable latency/failure rates. The focus was demonstrating durable orchestration patterns, not API integrations.

Repo Structure

web/     → Next.js UI + API (control plane)
worker/  → Temporal workflows + activities

Key files:

• worker/src/workflows/guideGeneration.ts - Parent orchestration
• worker/src/activities/generate.ts - Idempotent activity pattern
• web/app/api/runs/route.ts - Failure-atomic creation

Source

https://www.twine.net/signin