Integrate MacOS DMG and Mac App Store builds into Nix Build Pipeline

[CMCDragonkai]

These can only be done on MacOS.

Can it work if Nix is running on MacOS? Need to experiment with this on Mac system. We have the M1 mac system to test on and also Mac systems that @gideonairex is running on.

If it works, we can at least have a consistent platform to work on, using Nix on NixOS and using Nix on MacOS.

However I have heard from @rbellec that Nix has some problems with the latest OS BigSur. So we should probably avoid that.

[CMCDragonkai]

Also we have manual code signing that we have to do. How to integrate this? I'm not sure of the workflow. @gideonairex can you describe it?

[DrFacepalm]

After some experimenting, this is what we've found. the @electron-forge/maker-dmg package creates issues when trying to run nix-build as there exists an optional dependency that cannot be built on the nix-build environment. This is because node2nix does not differentiate between optional and regular dependencies, and therefore will not skip the dependency that can normally be skipped in a userland npm install.

I have tried adding --strip-optional-dependencies to node2nix however, this strips app optional dependencies and therefore cause issues with other builds, e.g. the deb builds do not work anymore.

Attempting to substituteInPlace the package.json also does not give any results, as:

• If we don't include package-lock.json, node2nix attempts to generate it, which requires fetching from online registries, which cannot be done during a nix build as it must remain offline.
• Attempting to recreate it using npm install leads to the same problem
• Replacing all affected lines in package-lock.json is too messy a process to consider as we need to consider blocks of code and several subdependencies.

As a result, for now in the source, we will not include @electron-forge/maker-dmg. It can STILL BE INSTALLED and used in userland, in the nix-shell.

[CMCDragonkai]

This requires some upstream fix then: svanderburg/node2nix#229.

For now, anytime we have this situation, either we repackage that as Nix dependency if it provides an executable, or we just have to interactively do it with npm install ... and then not commit that to the master. Or maintain it as a separate branch.

[joshuakarp]

This is also related to the integration of macOS CI/CD MatrixAI/Polykey#4

[CMCDragonkai]

This might be worth exploring: #16 (comment)

Also we did do some overrides of Nix and NPM packages here: https://gitlab.com/MatrixAI/Engineering/nixpkgs-overlay/-/merge_requests/15. This shows you how to override the node packages structure in Nixpkgs upstream. And therefore how to apply an override at the end.

There's definitely a way to ignore certain dependencies once we have generated them. But we will wait until we have MacOS CI/CD available before experimenting on this. We'll need to consider the usage of our application key and a manual Q&A process for our Mac applications.