Better automation for /sample-policies

[christophwille]

Currently

• has a Windows batch file
• using the Windows OPA binary
• that is manually placed in that folder
• with a manually modified capabilities file

Ideally, I want this to be

• cross-platform (.sh like Implement builtin0-4 #3 (comment) is not where I want to be)
• ".NETy" - yes, I could use pscore, or use npm, but a native .NET build automation solution would be nice
• definitely should work in GH actions

with the following properties:

• be able to work with a specific version of OPA (latest is not a version)
• create the wasms so they no longer are checked in

[christophwille]

Options for "getting OPA onto the machine": download & make executable (per platform) or maybe use the docker image?

[christophwille]

I am all for using jq https://stedolan.github.io/jq/download/ to do the capabilities files modifications (requiring it to be installed beforehand).

Because we are going to use it with Unit Tests, the generated capabilities file must have a defined name. And the capabilities files from OPA change the file names per version. How about a command line along those lines (pun intended):

concatcap.ps1 capabilities.json -merge v0.34.json -merge unittest1.json -merge sdkfuncs.json

@iinuwa your take?

[iinuwa]

Because we are going to use it with Unit Tests, the generated capabilities file must have a defined name.

Unless we use a temp file, right?

concatcap.ps1 capabilities.json -merge v0.34.json -merge unittest1.json -merge sdkfuncs.json

In this case, is capabilities.json the generated file name? I'm good with that as long as we use named parameters for everything.

(Can you specify an arg multiple times in PS? I thought it would have to be a comma-separated array. Away from computer now, so I can't test.)

Also, since this is theoretically a tool that users could also use for generating their own capabilities files, it might be good to decide whether we are going to allow builtins to be defined multiple times? If so, we need to set some sort of precedence. I think "last-specified wins" is a natural rule.

[christophwille]

My command line sample might not be proper PS syntax; also, I am mostly only concerned to solve the unit test use case here, not creating a "general purpose one-size fits all tool".

[iinuwa]

Oh, ok. Here's what I plan to use for myself; feel free to adapt it as you need: https://gist.github.com/iinuwa/af7f2f038ae817e640e2569e256c3268#file-mergecapabilities-ps1

(Examples are included in the script and can be viewed with Get-Help ./MergeCapabillities.ps1)

[christophwille]

I tried to make it work slightly dumbed-down on Windows:

param(
  [Parameter(Mandatory=$true)]
  [string[]] $Files=@(),
  [Parameter(Mandatory=$false)]
  [string] $Destination
)

[string[]]$duplicatedBuiltins =
jq -s -r "[ .[].builtins | .[] ] | group_by(.name) | map(select(length > 1)) | map((.[0].name))" $Files | ConvertFrom-Json
if ($LASTEXITCODE -ne 0) { throw }

if ($duplicatedBuiltins.Count -gt 0) {
  throw "Duplicated builtins defined: $($duplicatedBuiltins -join ' ')`nExiting."
}

$output = jq -s "{ allow_net: [ .[] | .allow_net[] ] | unique, builtins: [ .[].builtins | .[]] | group_by(.name) | map(.[-1]) }" $Files
if ($LASTEXITCODE -ne 0) { throw }

if ($Destination) {
  $output | Out-File -FilePath $Destination
} else {
  $output
}

I had to put everything on one line otherwise it complained. However

.\concat-capabilities.ps1 -Files v0.34.0.json, simple-custom-builtincall.capabilities.json
jq: error (at simple-custom-builtincall.capabilities.json:17): Cannot iterate over null (null)

This is the $output line. For testing, my simple-custom-builtincall.capabilities.json looked like

{
    "builtins": [
      {
        "name": "custom.func",
        "decl": {
          "args": [
            {
              "type": "any"
            }
          ],
          "result": {
            "type": "any"
          },
          "type": "function"
        }
      }
    ]
}

'cause I know way too little about jq, could it be the allow_net thingy?

[christophwille]

Seems so because $output = jq -s "{ builtins: [ .[].builtins | .[]] | group_by(.name) | map(.[-1]) }" $Files works

[christophwille]

ok, at least I can diff it now - one thing does change, it doesn't merge the following:

  "future_keywords": [
    "in"
  ],
  "wasm_abi_versions": [
    {
      "version": 1,
      "minor_version": 1
    },
    {
      "version": 1,
      "minor_version": 2
    }

[iinuwa]

'cause I know way too little about jq, could it be the allow_net thingy?

Yeah, you got it; in order for it to merge, the keys must be present on all merging documents.

ok, at least I can diff it now - one thing does change, it doesn't merge the following:

Oh, I hadn't noticed those keys before, but they can be merged using the same process as the builtins key. Looks like you figured it out in that last commit?

[christophwille]

Yes, I played around with diffs to figure out what I need and then it merged nicely.

[christophwille]

* create the wasms so they no longer are checked in

Maybe https://www.thorsten-hans.com/distribute-webassembly-modules-as-oci-artifacts/ is a viable option (store to ACR)

[iinuwa]

Are you aware that GitHub Actions can cache the output of a job? I've never used it, but if the wasm files were all stored in a separate directory (e.g. build/sample-policies/), something like this seems like it would work:

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v1

    - name: Cache OPA policy builds
      id: cache-policy-builds
      uses: actions/cache@v1
      with:
        path: build/sample-policies/
        key: opa-policy-cache-${{ hashFiles('sample-policies/*.rego') }}

    - name: Build OPA policies
      run: pwsh ./sample-policies/build.ps1
      if: steps.cache-packages.outputs.cache-hit != 'true'

This should only run the the OPA build if one of the rego files changes.