knit.md

Rules

A rule consists of four parts:

• Targets: a list of values that are generated by this rule.
• Prereqs: a list of values that must be generated before this rule can be executed.
• Attributes: a list of flags that customize how this rule is interpreted.
• Recipe: a list of commands that are run to execute the rule.

The syntax for a rule is

targets:attributes: prereqs
    recipe

If the rule has no attributes, it may be omitted, leaving just targets: prereqs on the first line.

When a rule is embedded in Lua, it is prefixed with a $:

$ targets:attributes: prereqs
    recipe

The rule continues until it is de-indented to the original indentation of the $.

Direct rules are rules where the targets and prereqs are explicit names of values. For example,

foo.o: foo.c
    gcc -c foo.c -o foo.o

Specifies that the file foo.o is built from foo.c, using the command gcc -c foo.c -o foo.o.

Within a rule, # denotes the start of a comment. Outside of rules, -- is the start of a comment (the standard Lua syntax).

Meta rules

A meta rule is a special rule that describes a generic way to create direct rules. The meta rule describes how to match targets and prereqs into a direct rule. For example, a % meta rule uses % to match any sequence of characters.

Thus the meta rule

%.o: %.c
    ...

describes that a direct rule

foo.o: foo.c
    ...

could be created. This direct rule would be created if another rule needed foo.o to be generated.

Meta rules may also specify the match by using regular expressions, if the R attribute is provided. For example:

foo-(.*)-(.*).tar.gz:R: $1/$2/foo
    ...

could create a direct rule

foo-linux-amd64.tar.gz: linux/amd64/foo
    ...

Attributes

Knit supports the following attributes:

• Q (quiet): do not print this rule's recipe when executing.
• R (regex): this meta rule uses regular expression matching.
• V (virtual): the value this rule generates is not a file, just a name.
• M (no meta): this rule's targets cannot be matched by meta rules.
• E (non-stop): this rule does not stop if one of its commands fails.
• B (build): this rule must always be built (it is always out-of-date).
• L (linked): this rule always runs if an out-of-date sub-rule requires it as a prereq.
• O (order-only): this rule's prereqs are not considered automatically up-to-date even if this rule is up-to-date.
• D[depfile] (dependency): include depfile as an additional list of dependencies for this rule.

The D attribute takes an argument. It is used for including .d files for C headers. For example, this rule

$ %.o:D[%.d]: %.c
    gcc -MMD -c $input -o $output

specifies that it should read the file %.d as a rule file and include any additional rules (without recipes) as dependencies for the current rule. If the file does not exist it is ignored, and any rules from the file that can't be satisfied are ignored instead of returned as errors.

Attributes can also be applied to particular prerequisites rather than to an entire rule, using the syntax prereq[attributes]. For example:

foo:V:
    echo foo
bar:V: foo[Q]
    echo bar

The foo rule will be quiet only when used as a prerequisite to the bar rule.

Some attributes can only be applied in this way:

• I (implicit): this prereq does not appear in $input.

The [...][attributes] syntax can be used to apply attributes to groups of prerequisites. For example, in the following rule all three prerequisites are implicit.

foo: [a b c][I]
    ...

Recipes

A recipe is a list of commands to execute, each separated by a newline. They are executed within the sh shell. Each command is executed in a separate process, spawned with sh -c <cmd>. One implication of this is that a cd command will not persist to future commands. If you wish to run multiple commands in the same shell, you should use the shell's features (&&, ||, or ;) for this. For example, cd foo; cat bar.txt. Note that you can use \ to escape newlines, so that one command can span multiple lines in the recipe.

Recipes may use variables that will be expanded before the recipe executes. Variables are written with $var, or full Lua expressions can be written with $(expr). Variables/expressions are expanded eagerly when the rule is created. If expansion causes an error, the expansion is delayed until rule evaluation (when special build variables are available). You can use $$ to escape a dollar sign. For example, the recipe echo $$PWD will echo the environment variable PWD, while echo $PWD would attempt to replace $PWD with the Lua variable PWD when the rule is elaborated.

Some special variables are available during recipe expansion:

• input: a string of rule's prereqs.
• inputs: an array of this rule's prereqs.
• output: a string of this rule's targets.
• outputs: an array of this rule's targets.
• match: the value captured with % in a meta rule.
• matches: a list of matches captured by a regular expression meta rule.
• dep: the name of the dependency file (if it exists) for the rule, defined by the D[...] attribute.

When a rule is encountered, $ expressions are immediately expanded. If expansion fails, expansion is re-tried when the rule is evaluated (once the inputs/outputs are known).

Lua expressions should not mix uses of special build variables and Lua local variables. Local variables are only available during immediate expansion, and special build variables are only available during lazy expansion. This constraint may be relaxed in the future if it turns out to be a useful feature.

Out-of-date calculation

To determine if a rule must be re-run, Knit computes whether its output is up-to-date or not. There are two mechanisms for this: a hash-based one and a timestamp-based one. By default, Knit uses the hash-based mechanism: if a file's hash has changed since the previous build, it is considered out-of-date, and all rules that depend on it must be re-run. When depending on a directory, its hash is the recursive hash of all files within it. The hash-based method is good for small builds, but can become too slow for large builds. In those cases you can disable hashing and use the timestamp-based calculation.

When hashing is disabled, Knit uses a similar computation to Make that is based on file modification times. If a rule's prerequisites refer to files that have been modified more recently (according to the system's file modification timestamp) than the output file, then the rule is determined to be out-of-date and must be re-run. Other factors may also cause a rule to be re-run, such as if its recipe has changed, or if it has a rebuild attribute. If the file refers to a directory, the system's timestamp is not used. Instead, the modification time of a directory is the timestamp of the most recently modified file in it (or in any sub-directory). Depending on a very large directory may hinder performance.

Hashing can be disabled on a per-project basis or globally by using the .knit.toml configuration file, described the "Configuration" section of this documentation.

Knitfiles

A Knitfile is a Lua 5.1 program with additional support for rule expressions. The Knitfile ultimately must return a "buildset" -- a list of build rules that are used to construct the build graph.

A rule is defined in Lua with the $ syntax:

$ targets:attributes: prereqs
    ...

A rule expression may be assigned to a variable

local rule = $ foo.o: foo.c
    gcc -c $input -o $output

More often, rule expressions are gathered together in a Lua table:

local rules = {
    $ foo.o: foo.c
        gcc -c $input -o $output
    $ foo: foo.o
        gcc $input -o $output
}

When you run knit, Knit will automatically look for a file called Knitfile or knitfile (or you can specify a custom name with -f). Knit will look in the current directory, and up to ancestor directories if one does not exist in the current directory. This allows you to execute the build from anywhere in your project without fragmenting the build system with multiple Knitfiles. You can still make directory-specific targets that are namespaced relative to that directory with sub-builds.

If you run knit foo.o from the foo directory, and there is a file ../Knitfile that defines a rule for building foo/foo.o, Knit will automatically figure out that you mean to build foo/foo.o (relative to ..), since you specified foo.o (relative to foo). In other words, building sub-files just works.

Likewise, if you run knit all from the foo directory, and the all rule is only defined for the root directory, Knit will automatically use that rule instead of trying to use foo/all.

Deviations from Lua 5.1

There are some differences between Knit Lua and Lua 5.1:

• Knit supports $ for creating rules.
• Knit supports := for creating raw strings.
• Knit will give an error message when attempting to access an undeclared variable, whereas Lua 5.1 will just return nil for the value.

Rulesets

A table of rules can be converted into a "ruleset" by using the special r function, which converts the table into a Lua "userdata" object representing a list of build rules.

local ruleset = r{
    $ foo.o: foo.c
        gcc -c $input -o $output
    $ foo: foo.o
        gcc $input -o $output
}

Note that two rulesets may be combined with the + operator.

ruleset = ruleset + r{
    $ build:V: foo
}

Buildsets

A buildset is a set of rules associated with a particular directory. A buildset may also contain other buildsets (rules from other directories). All rules in the buildset are executed relative to its directory. A buildset can be constructed by using the special b function, which constructs a buildset from a table of rules, rulesets, or other buildsets.

local buildset = b{
    $ foo.o: foo.c
        gcc -c $input -o $output
    $ foo: foo.o
        gcc $input -o $output
}

By default the buildset's directory is the current working directory when it is constructed. A second argument may also be passed to b to directly specify the build directory.

local buildset = b({
    $ foo.o: foo.c
        gcc -c $input -o $output
    $ foo: foo.o
        gcc $input -o $output
}, "directory")

A buildset must be returned by the Knitfile for a build to take place. When a buildset is returned, knit expands it and all the buildsets that it returns into the full set of rules, where each rule is relative to the buildset directory that it came from. Rules may have cross-buildset dependencies.

These facilities for making rules relative to directories are for enabling sub-builds, discussed in the next section.

Sub-builds

A build may use several buildsets.

For example:

-- this buildset is relative to the "libfoo" directory
local foorules = b({
    $ foo.o: foo.c
        gcc -c $input -o $output
}, "libfoo")

return b{
    $ prog.o: prog.c
        gcc -c $input -o $output
    -- libfoo/foo.o is automatically resolved to correspond to the rule in foorules
    $ prog: prog.o libfoo/foo.o
        gcc $input -o $output

    -- include the foorules buildset
    foorules
}

This Knitfile assumes the build consists of prog.c and libfoo/foo.c. It builds libfoo/foo.o using a sub-build and automatically determines that the foorules buildset contains the rule for building libfoo/foo.o. Note that the recipe for foo.o is run in the libfoo directory. Including a buildset inside another will automatically including all of its rules namespaced into the directory that the buildset came from.

It is also useful to combine sub-builds with the include(x) function, which runs the knit program x from the directory where it exists, and returns the value that x produces. This means you can easily use a sub-directory's Knitfile to create a buildset for use in a sub-build.

For example, for the previous build we could use the following file system structure:

libfoo/build.knit contains:

-- this buildset's directory will be the current working directory
return b{
    $ foo.o: foo.c
        gcc -c $input -o $output
}

Knitfile contains:

return b{
    $ prog.o: prog.c
        gcc -c $input -o $output
    -- libfoo/foo.o is automatically resolved to correspond to the rule in foorules
    $ prog: prog.o libfoo/foo.o
        gcc $input -o $output

    -- include the libfoo rules: this will change directory into libfoo, execute
    -- build.knit, and change back to the current directory, thus giving us a buildset
    -- for the libfoo directory automatically
    include("libfoo/build.knit")
}

Note that since knit looks upwards for the nearest Knitfile, you can run knit foo.o from inside libfoo, and knit will correctly build libfoo/foo.o.

Since managing the current working directory is important for easily creating buildsets that automatically reference the correct directory, there are several functions for this:

• include(x): runs a Lua file from the directory where it exists.
• dcall(fn, args): calls a Lua function from the directory where it is defined.
• dcallfrom(dir, fn, args): calls a Lua function from a specified directory.
• rel(files): makes all input files relative to the build's root directory.

Configuration

Knit will search the current directory for a Knitfile called knitfile or Knitfile. If one is not found, it will use the Knitfile in ~/.config/knit/Knitfile.def, or if that does not exist it will throw an error.

Several options are available as command-line flags. They may also be specified in a .knit.toml file. Knit will search upwards from the current directory for .knit.toml files, and use the options set in those files. It will also search ~/.config/knit/.knit.toml.

The default set of flags is:

knitfile = "knitfile"
ncpu = 8 # depends on the number of logical cores on your machine
dryrun = false
directory = ""
always = false
quiet = false
style = "basic"
cache = ""
hash = true
updated = []
root = false
keepgoing = false
shell = "sh"

Sub-tools

Running knit [TARGET] will create a build graph for the target. By default, knit will then execute that build graph. Using the -t TOOL option, you may specify a sub-tool to run instead of building:

• list - list all available tools
• graph - print build graph in specified format: text, tree, dot, pdf
• clean - remove all files produced by the build
• targets - list all targets (pass 'virtual' for just virtual targets)
• compdb - output a compile commands database
• commands - output the build commands (formats: knit, json, make, ninja, shell)
• status - lists dependencies and whether they are up-to-date
• path - shows the path of the current knitfile

The special target :all depends on every target in the build. Thus knit :all -t targets will list all targets.

Some examples are shown below.

Automatic cleaning

knit target -t clean

Output a shell script for the build

knit target -t commands shell

Output a Ninja build file

knit target -t commands ninja

Output a compile commands database

knit target -t compdb

Output a PDF build graph

knit target -t graph pdf > graph.pdf

Special rules

Knit automatically defines two special rules: :all and :build.

The :all rule depends on all possible targets in the build (except those attainable only from meta-rules). For example knit :all -t targets will list all possible targets, and knit :all -t clean will clean all possible outputs.

The :build rule is the root rule of the build and depends on all requested targets. For example knit a b c will generate a :build rule that depends on a, b, and c. In general, you should never refer to the :build rule since doing so will usually create a build cycle.

Default rule

If you run knit without a target, Knit will build the first non-meta rule.

Rule priority

If several rules with recipes that could be used to build a file, Knit uses the last one. In other words, defining a rule later will override previous definitions of a rule. However, if a later rule does not have a recipe, it will not override the rule for the target, but instead just add the new prerequisites for that target to the previous rule.

If several rules from different buildsets could be used to build a target, the rule from the buildset for the target's directory is attempted first. If it does not exist, then rules are attempted from all other buildsets and the first buildset to have a matching rule is used. If a meta-rule is used, it is attmpted in the current buildset before looking in other buildsets.

Built-in Lua syntax

• $ ...: creates a rule. The rule is formatted using string interpolation. The rule continues until indentation returns to the level of the $.

• x := ...: creates a string without quotes. The string value continues until the end of the line, and is automatically formatted using string interpolation.

Both of these expressions are implicitly terminated with a ;, allowing them to be used in tables.

Built-in Lua functions

Note: several functions throw errors. Use the built-in Lua pcall function to perform error handling. local ok, result = pcall(fn, args) returns whether there was an error during the execution of fn(args). The result variable will contain the result, or the error value.

• rule(rule): define a rule. The $ syntax is shorthand for this function.

• rulefile(file): define a rule by reading it from file. Throws an error if the file does not exist.

• include(file): run a Knitfile from its directory (changes the current working directory while the file is being executed) and return the generated ruleset. Throws an error if file does not exist.

• dcall(fn, args): call fn(args) from the directory where fn is defined.

• dcallfrom(dir, fn, args): call fn(args) from dir.

• rel(files): make all paths in the table files relative to the build root (the location of the Knitfile).

• r{$ ...}, r({...}): turn a table of rules into a ruleset.

• b{...}, b({...}, dir): turn a table of rules, rulesets, or buildsets into a buildset associated with directory dir. dir is optional, and if not specified will be the current working directory.

• tobool(value) bool: convert an arbitrary value to a boolean. A nil value will return nil, the strings false, off, or 0 will become false. A boolean will not be converted. Anything else will be true.

• eval(code): evaluates a Lua expression in the global scope and returns the result. Throws an error if the code has an error.

• f"...", f(s): formats a string using $var or $(expr) to expand variables/expressions. Throws an error if the variable does not exist, or the expression has an error.

• expand(s): formats a string in the same way as f, but if there is an error, it does not expand that particular $... expression.

• use(pkg): imports all fields of pkg into the global namespace. Meant to be used with require: use(require("knit")).

• sel(cond, a, b): if cond is true return a, otherwise return b.

• choose(a, b, c...): return the first value in the list of arguments that is not nil.

• r{} + r{}: you may use the + operator to combine rulesets together.

• b{} + val: you may use the + operator to combine buildsets with rules/rulesets/buildsets.

• {s} + {s}: string tables returned by knit functions can be added together.

The knit Lua package

The knit package can be imported with require("knit"), and provides the following functions:

• repl(in, patstr, repl): replace all occurrences of the Go regular expression patstr with repl within the array in. Throws an error if there is an error with patstr.

• extrepl(in, ext, repl): replace all occurrences of the literal string ext as a suffix with repl within the array in.

• glob(pat): return all files in the current working directory that match the glob pat.

• suffix(in, suffix): add the string suffix to the end of every string in the array in, and return the new array.

• prefix(in, prefix): add the string prefix to the beginning of every string in the array in, and return the new array.

• filterout(in, exclude): returns a new table containing all the elements of in, except those in exclude.

• shell(cmd) string: execute a command with the shell and return its output. Throws an error if the command exits with an error.

• trim(s): trim leading and trailing whitespace from a string.

• abs(path): return the absolute path of a path.

• dir(path): return the directory part of a path.

• base(path): return the basename of a path.

• os: a string containing the operating system name.

• arch: a string containing the machine architecture name.

• flags: a struct containing the values of the flags when Knit was invoked. See https://pkg.go.dev/github.com/zyedidia/knit#Flags.

• addpath(p): adds the path p to the global require path. Files with ending with .lua or .knit are added.

• knit(flags): executes the shell command knit flags (where flags is a string of CLI arguments) using the current instance of Knit.

CLI and environment variables

Variables may be set at the command-line when invoking Knit with the syntax var=value. These variables will be available in the Knitfile in the cli table. Environment variables are similarly available in the env table.