MANDISH NANDA VAIDYA

Designed and developed a containerized web crawling system in Python using Docker, integrating modular Crawl4AI pipelines with FastAPI to enable scalable, high-volume automated extraction of 20,000+ structured web items. Engineered LLM-driven data processing workflows to clean, normalize and structure unstructured web data, leveraging targeted prompt engineering to improve response accuracy and reduce hallucinations. Researched and implemented advanced crawling strategies for dynamic, large-scale web environments, achieving over 90% reduction in LLM token usage and operational cost. Developed and maintained a RAG-based chatbot utilizing Qdrant for semantic search across crawled datasets, and managed Redis and PostgreSQL for structured storage, caching, and user interaction tracking.

Led data crawling research for dynamic, large-scale websites; designed scalable crawling pipelines with Crawl4AI for high-volume extraction; cleaned and normalized unstructured web data using LLM-based workflows; implemented crawling configurations for pagination, sessions, and workflow automation; conducted end-to-end testing and produced API documentation with Sphinx; performed prompt engineering to improve LLM response quality and reduce hallucinations.