webscraping and E-commerce
Tech Stack
Frontend: Angular
Backend: Flask (Python)
Libraries/Tools: Playwright, Selenium, Requests, BeautifulSoup
Database: DynamoDB
Cloud Services: AWS
Task Management: Celery, Redis
Business Problem
The client required a system to scrape product data from Facebook Marketplace and allow users to run scrapers based on specified criteria.
Solution
The solution involved building a fullstack website with a responsive frontend in Angular and a robust backend in Flask. Users could input search criteria on the website, which triggered a background job using a microservices architecture powered by Celery and Redis. The scraper, utilizing Playwright, Selenium, Requests, and BeautifulSoup, fetched product data efficiently based on the user's input.
The scraped data was stored in DynamoDB, ensuring high performance and scalability. Once the scraping job was complete, the system automatically notified the user via email with the results.
This end-to-end solution provided a seamless and user-friendly experience, enabling users to fetch, store, and review Facebook Marketplace data effortlessly. The system's architecture ensured speed, scalability, and reliability for concurrent user requests.…webscraping and E-commerce
Tech Stack
Frontend: Angular
Backend: Flask (Python)
Libraries/Tools: Playwright, Selenium, Requests, BeautifulSoup
Database: DynamoDB
Cloud Services: AWS
Task Management: Celery, Redis
Business Problem
The client required a system to scrape product data from Facebook Marketplace and allow users to run scrapers based on specified criteria.
Solution
The solution involved building a fullstack website with a responsive frontend in Angular and a robust backend in Flask. Users could input search criteria on the website, which triggered a background job using a microservices architecture powered by Celery and Redis. The scraper, utilizing Playwright, Selenium, Requests, and BeautifulSoup, fetched product data efficiently based on the user's input.
The scraped data was stored in DynamoDB, ensuring high performance and scalability. Once the scraping job was complete, the system automatically notified the user via email with the results.
This end-to-end solution provided a seamless and user-friendly experience, enabling users to fetch, store, and review Facebook Marketplace data effortlessly. The system's architecture ensured speed, scalability, and reliability for concurrent user requests.WW…