Skip to content

Web Crawler with Angular 8, Expressjs and Redis

Notifications You must be signed in to change notification settings

Ghokai/SiteAnalyzer

Repository files navigation

SiteAnalyzer

Web Crawler with Angular 8, Expressjs and Redis

You can simple run redis, server and client in docker containers with docker compose

docker-compose up 

docker-compose terminal

image

analyzer server runs in http://localhost:3000

image

analyzer client runs in http://localhost:4200

image

image

redis cache runs on redis:6379 in container

image

User goes to analyzer-client(http://localhost:4200) and enters sitename(url) for getting analyze results of site and click button.

Analyzer-client make post request to analyzer-server(http://localhost:3000/analyze) with sitename parameter

Analyzer-server check redis cache if analyze result calculated before in 24 hours server will be return analyze result immediately from cache.

If analyze result not found in cache. Analyzer server will make request to site and load its content. And will make some web crawling process for extracting information from document content.

After analyze result gathered. result will be storing in the cache server for later usage with 1 day expiration time.

Analyzer server will be return analyze results to analyzer-client and analyzer client shows analyze results to user.