Skip to content

Simple Web UI for Scrapy spider management via Scrapyd

License

Notifications You must be signed in to change notification settings

iamumairayub/scrapymon

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

scrapymon

Simple management UI for scrapyd. The demo is available at http://scrapymon.demo.jxltom.me/ with admin for both username and password. Note that the demo will reset every 40 minutes and it may take some time to spin up if no one has accessed it for a while.

Features

  • Show all projects from a Scrapyd server
  • Show all versions of each project
  • Show all spiders in each project
  • Show all pending, running and finished jobs from a Scrapyd server
  • Show logs of each job
  • Schedule spiders run
  • Cancel pending or running jobs
  • Delete project or a specific version
  • Http basic access authentication supported
  • Served by Gevent for production use

Screenshots

projects_dash jobs_dash logs_dash

Getting Started

  • Install by pip install scrapymon.

  • Run by scrapymon [--host=<host>] [--port=<port>] [--server=<address_with_port>] [--auth=<username:password>].

    • Default --host is 0.0.0.0
    • Default --port is 5000
    • Default --server is http://127.0.0.1:6800
    • Default --auth is admin:admin
  • Or you can run by scrapymon with valid environment variables $HOST, $PORT, $SCRAPYD_SERVER and $BASIC_AUTH.

TODO

  • Support schedule a spider run with arguments.
  • Highlighted and searcharable logs with catagories
  • Logs auto refresh and pagination
  • Create project via addversion.json.
  • Time Localization
  • Add Dockerfile

Contributing

Contributions are welcomed!

About

Simple Web UI for Scrapy spider management via Scrapyd

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 62.0%
  • HTML 25.3%
  • JavaScript 10.2%
  • CSS 2.5%