Code Operator

Utility to make efficient code operations using the Javascript AST. Toolset:

• estools

• esprima

• escode - code generator

• estraverse

• merge-estraverse-visitors

• esquery

• esdispatch

• escope

• esutils

• estemplate - templating using AST

• esvalid

We will use aster repos from aster@kristianmandrup or when updated dependencies are merged into origins.

Grasp - refactoring code (search/replace AST)

Grasp is usd internally by aster see below.

use as library code replacement squery equery concepts

const grasp = require('grasp')
const replacer = grasp.replace('equery', '__ + __', '{{.l}} - {{.r}}')
const processedCode = replacer(code)

Search

search takes a string choosing a query engine (squery or equery), a string selector, and a string input, and produces an array of nodes. Eg.

const nodes = grasp.search('squery', 'if', code);

Replace

replace takes a string choosing a query engine (squery or equery), a string selector, a string replacement, and a string input, and produces a string of the processed code. Eg.

const processedCode = grasp.replace('squery', 'if.test', '!({{}})', code);

Instead of providing a replacement pattern as a string, you can pass in a function which produces a string, and this string will be used as the replacement.

var processedCode = grasp.replace('squery', 'call[callee=#require]', function(getRaw, node, query) {
    var req = query(".args")[0];
    return "import " + camelize(path.basename(req.value, ".js")) + " from " + getRaw(req);
}, code);

Esquery (Example query)

if(__){ __ } matches any if statement with any test, and one statement in its body. function __(__) { __ } matches a function with any name, one parameter of any identifier, and a body with one statement.

You can also give the wildcard a name which can be used to refer to it during replacement. $name will match any expression, statement, or identifier, and during replacement the matched node can be accessed using its name, eg. {{name}}. If you use a name more than once, then the values for both must match

• eg. $a + $a will match 2 + 2, but not 2 + 1.

You can use _$, which matches zero or more elements. Modifying our previous example, function __(_$) { _$ } matches a function with any name, any amount of parameters, and any amount of statements.

Replace

First, the text {{}} will be replaced with the source of the matched node.

For instance, the replacement text f({{}}) would result in each match being replaced with a call to the function f with the match as its argument.

$ cat file.js
if (y < 2) {
  window.x = y + z;
}
$ grasp '[left=#y]' --replace 'f({{}})' file.js
if (f(y < 2)) {
  window.x = f(y + z);
}

Second, the text {{selector}} will be replaced with the first result of querying the matched node with the specified selector. The query engine used to process the selector will be the same as you used for searching, eg. if you used equery to search for matches (with the -e, --equery flag), then the replacement selector will also use equery.

An example:

$ cat file.js
if (y < 2) {
  window.x = y + z;
}
$ grasp if --replace 'while ({{.test}}) {\n  f(++{{assign bi.left}});\n}' file.js
while (y < 2) {
  f(++y);
}

See Syntax for full overview of Javascript syntax you can use to query AST.

Aster

Perhaps better and easier to use aster Using esprima 3 and above, it has ES6 and ES.next (including async/await) support built in :)

const aster = require('aster');
aster.src.registerParser('.js', require('aster-parse-esnext'));

aster is a reactive builder specialized for code processing and transformations. It's built with debugging in mind and makes building JavaScript code more reliable and faster. Aster uses RxJS Observables for its reactive pipeline infrastructure.

RxJS tutorials

• Observable tutorial video series
• RxJs book

Aster equery example

var aster = require('aster');
var equery = require('aster-squery');

aster.src('src/**/*.js')
.map(squery({
  'if ($cond) return $expr1; else return $expr2;': 'return <%= cond %> ? <%= expr1 %> : <%= expr2 %>'
  // , ...
}))
.map(aster.dest('dist'))
.subscribe(aster.runner);

Alternatively using squery

var aster = require('aster');
var squery = require('aster-squery');

aster.src('src/**/*.js')
.map(squery({
  'if[then=return][else=return]': 'return <%= test %> ? <%= consequent.argument %> : <%= alternate.argument %>'
  // , ...
}))
.map(aster.dest('dist'))
.subscribe(aster.runner);

To remove an AST node such as a function with a specific indentifier, find it via selector and replace with an empty string!!

Customized pipeline

You can also use a custom Observable to feed aster.src

function srcObserver(options) {
  return Rx.Observable.of(options.sources);
}

const sources = ['var a = 1', 'var b = a + 2']

// alternatively:
// const srcObserver = Rx.Observable.of(options.sources);

aster.src({
  srcObserver,
  sources,
})

aster pipeline modules

• aster runner
• aster dest
• aster src
• aster generate
• aster squery
• aster equery
• aster traverse
• aster parse
• aster parse js

Full customized pipeline example

function srcObserver(options) {
  return Rx.Observable.of(options.sources);
}

const sources = ['var a = 1', 'var b = a + 2']

function destinator() {
  return function (sources) {
    sources = options.generate(sources);
    sources.flatMap(function (source) {
      console.log(source)
    }
  }
}

function generator() {
	return function(sources) {
		return sources.flatMap(function (source) {
			var result = escodegen.generate(source.program, options);
			return Rx.Observable.fromArray(result);
    }
  }
}

aster.src({
  srcObserver,
  sources,
})
.map(equery({
  'if[then=return][else=return]': 'return <%= test %> ? <%= consequent.argument %> : <%= alternate.argument %>'
  // , ...
}))
.map(aster.dest({
  generator,
  destinator
}))
.subscribe(aster.runner({
  onSuccess: (item) => {
    console.log('success', item);
  }
}));

Aster sure looks like the best option!

Tutorials

esprima tutorial

Tools of the trade

This library was created using the guides:

• moving-to-webpack-2
• webpack usage
• WebpackTutorial 1 & 2
• how-to-write-a-good-npm-module.html
• code-coverage-with-instanbul-and-coveralls
• babel handbook
• webpack testing
• mocha-webpack
• es7-decorators-babel6

Libs used

• babel-loader
• babel-preset-latest-minimal
• npm-install-webpack-plugin

Testing

• debugging-mocha-unit-tests-in-visual-studio-code

$ mocha --debug-brk
Debugger listening on 127.0.0.1:5858

Configure .launch.json file in root with this host and port.

Code coverage

Use cross-env and nyc interface

npm i nyc --save-dev

• babel-plugin-coverage
• es6-code-coverage-with-babel-plugin
• node-mocha-travis-istanbul-coveralls

"Using a babel plugin for coverage is a no-brainer." - @kentcdodds

Even better:

• babel-plugin-istanbul

npm install --save-dev babel-plugin-istanbul

Decorator

• lodash-decorators
• core-decorators

Documentation

• documentation.js

Karma

• karma-mocha-reporter

Flowtype

• flowtype

Plato reports

npm-run plato -r -d reports ./

Code style

• xo
• standard - javascript standard code style

eslint --init to configure and initialize ESlint

{
    "extends": "standard",
    "installedESLint": true,
    "plugins": [
        "standard",
        "promise"
    ]
}

• eslint-and-mocha

Babel plugins

List current plugins needed according to the version of node:

npm-run babel-node-list-required

[ 'transform-es2015-duplicate-keys',
  'transform-es2015-modules-commonjs',
  'syntax-trailing-function-commas',
  'transform-async-to-generator' ]

npm i babel-plugin-transform-es2015-duplicate-keys babel-plugin-transform-es2015-modules-commonjs babel-plugin-syntax-trailing-function-commas babel-plugin-transform-async-to-generator --save-dev

For Vue projects

npm install --save-dev eslint-config-vue eslint-plugin-vue

• eslint-config-vue
• eslint-plugin-vue

Objectives

Read project directory as stream of files

• pass through filter (add metadata for type of file basd on location/context and file name + extension)
• operate on file
• send to output stream, writing it back or sending file to new location

API

Create new file

operator({
  model: 'person'
})
.extends('base')
.constructor(['name'])
.async.fun('speak', ['text'])
.fun('walk', ['distance'])

Creates file src\models\person.js

import Base from './base'

export default class Person {
  constructor(name) {
    super()
    this.name = name
  }

  async speak(text) {
  }

  walk(distance) {
  }
}

Note: Could also be performed on multiple files!

Modify existing file

Change speak to not be async and remove function walk

operator({
  model: 'person'
})
.fun('speak', ['text'])
.remove('walk')

Note: Could also be performed on multiple files!

Delete existing file

operator({
  model: 'person',
  view: 'person'
})
.delete()

Multiple delete

operator({
  models: ['person', 'account'],
  views: ['person', 'account']
})
.delete()

Delete all domain files of the given names except the test files:

operator({
  domains: ['person', 'account']
  exceptFor: ['test']
})
.delete()