tasks.md

Tasks

Before beta

• ensure text parsing does not include starting newline
• add ability to set project-specific css
• support different fence types for examples (eg ---)
• hello-world project generator
• alpha for images without matching dimensions
• record timestamp when a page started executing, use in the result template
• Node 8
• use pageObj.sourcePath when approving with new result instead of hard-coding the path
• test with large images
• test with complex page and example names
• test with exceptions thrown throughout the test process
• directory summaries for page folders
• skip empty files (instead of blowing up)
• Solve cross-linking (eg replace .md in local links to .html when generating html)
• detect examples with the same name on a page
• support image/* content types for content/contentType fixtures
• Enable people to extend/plug easier
  • add more parsers
• test overlay/positioning in other browsers
  • safari
  • firefox
• handle gracefully cases where the expected result is provided, but the file is not present
• report problems better when the expected result is provided, but there is no example matching that name

before 1.0

• headless firefox
• AWS Lambda chrome puppeteer
• Make it easier to inspect/approve
  • CLI Approval instructions on the results page
  • CLI Approval instructions on the rendered page (for all examples)
  • Overview/Overlay/Inspect on the results page
    • set view/opacity/zoom for actual/expected/diff separately
    • scale down large images so they are easier to see on the overview
      • checkbox for original size/size to fit
• Distinguish between a failure and error
  • record errors from fixtures
  • display error as status when listing results
  • format/test error messages
    • thrown as exceptions
    • thrown as strings
    • rejected from a promise
• Fixture execution
  • inside node
    • detect and parse example format (eg json or yaml)
    • pass fixture parameters in the context
    • pass temporary working dir
    • allow fixtures to return a file
  • URL fixture execution (eg test the same site in various resolutions)
  • in-browser fixture execution
    • record after page load
    • record after network idle
    • record after custom event
  • webpack fixture packaging
  • AWS Lambda fixture engine
• Extend example properties
  • Configurable clip area for screenshots
  • Configurable initial size for html windows
  • Configurable image matching precision and anti-aliasing
• Extract configuration/enable overrides
  • Configurable source and work directories (don't just dump stuff to temp)
  • Configurable clip area for screenshots
  • Configurable initial size for html windows
  • Configurable image matching precision
  • + and anti-aliasing
  • Configurable HTML attributes (data-example)
  • Configurable HTML classes for success/failure
  • Configurable screenshot engine
  • Configurable fixture engines (via NPM modules)
  • Config file to set all options quickly
• Server for interactive work
  • render example folder structure + README.md for folders
  • render markdown from the examples folder
  • offer running the spec if the MD contains any examples
  • approving changes
  • Running tests by directly reading specs from github (eg public site, connect to a gitub repo)
  • Approving by directly committing to github? (eg for specs stored in github repos)
• Result formatters
  • JSON
  • junit
  • tap
• Reporters
  • ASCII console
  • ANSI color console
  • Summary (dot)
• CLI/NPM tasks
  • executing tests
  • run all tests
  • run a specific page
  • run a specific example
  • exit with -1 in case of errors/failures
  • approving changes
  • approve a whole page
  • approve a specific example
  • approve all failed examples

Released in Alpha

• breadcrumbs on rendered page -> links
• Start/stop chrome once per run, not once per page
• navigation on results page (breadcrumbs and page links)
• Retina screen sizing
• change sync FS operations to async
• put example name into the example object/results object for easier templating
• use timestamps for executed and started in result template
• test with multiple examples in a single file
• test with multiple files
• test with files in subfolders
• Choose a markdown rendering system (https://github.com/markdown-it)
• Choose a Node DOM manipulation system (https://github.com/cheeriojs/cheerio)
• Build local screenshots (using Chrome Headless and https://github.com/cyrus-and/chrome-remote-interface)
• Choose an image comparison system (https://github.com/mapbox/pixelmatch)
• generate a combined image if dimensions do not match, showing both images with alpha
• Add a generic layout to HTML when converting from MD
• Add a generic template for result files
• Create an index with all test results
• Clean up templates and provide a nice layout
• documentation on key usage scenarios
• Run examples without expected outcomes (fail + show actual)