Skip to content

Latest commit

 

History

History
47 lines (32 loc) · 1.89 KB

README.md

File metadata and controls

47 lines (32 loc) · 1.89 KB
Raw

Site Lab

Site Lab aims to be an open-source replacement for website analysis tools such as BuiltWith, NerdyData, and DataNyze.

Site Lab is a Ruby on Rails application. It uses PostgreSQL as its database and Redis + Sidekiq for background processing.

How Does it Work?

Right now, it's fairly simple:

  • The MetaInspector Gem retrieves some basic info about the site/URL
  • There is a "Technology" model which stores regular expressions
  • Technologies are matched against the source of the sites/URLs
  • Much of the processing now happens in the background (via Sidekiq)

More complex analysis is in the works.

Installation

It's a Rails 4.1 app, so you'll need a dev environment that supports that (prolly RVM). You'll also need Redis installed and running (probably via Homebrew)

  • Clone the repo
  • Edit the database.yml file with your info
  • Run bundle install to install gems
  • Run bundle exec rake db:create to create the DB(s)
  • Run bundle exec rake db:seed to load the seed data
  • Run foreman start -p 3000 to start the rails server & sidekiq locally on port 3000

Importing Data

While you can surely add sites/URLs one-by-one in the app, most use-cases will involve importing large sets of URLs from files or external sites. With that in mind, I've started a set of Rake tasks for importing URLs. Currently, it includes:

  • Importing all startups from AngelList for a given market
  • Importing all startup/product URLs listed on Producthunt
  • Importing URLs from a text file (placed in app/import)
  • Importing all startup URLs from VCDelta

Run a rake -T to see the tasks and required parameters. There is also a sample text file in app/import.

Screenshot

Screenshot