Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Simple Nix Flake #7656

Merged
merged 6 commits into from
Oct 21, 2024
Merged

Conversation

JosephGoulden
Copy link
Contributor

@JosephGoulden JosephGoulden commented Sep 10, 2024

This is new PR to add a simple flake.nix to this project. It will allow downstream projects to easily import CLN into their system. They will be able to get new releases as soon as they are available instead of having to wait for nixpkgs to be updated. And they will be able to more easily test any changes being made in CLN.

There is an existing PR #6650 but progress has stalled with getting the Python dependencies working. I think it is best to leave that for a later date and focus on a minimal working and useful flake which I have created in this PR.

The default package is called cln, it has been copied from the one currently in nixpkgs. The derivation calls the Makefile and builds the binaries (lightningd, lightning-cli, lightning-hsmtool, reckless).

There is also a package called rust which builds the Cargo workspace at the root of the project. Binaries are then copied from here into the build directory of the cln package.

The use of git submodules makes the UX a little awkward but still works as expected with the right commands. e.g;

  • nix develop will create the default shell env with the build and runtime dependencies of the cln package. Then you can call make manually and the project will compile successfully.

  • nix develop .#rust will create a shell env for developing rust.

  • nix build .?submodules=1 will build the default package (cln).

  • nix flake check .?submodules=1 will build the cln and rust packages. Rust tests are run during the build. There are also checks to run cargo audit and nixfmt.

  • If you have nix installed you can run use nix run "git+https://github.com/hashrelay/lightning?ref=flake&submodules=1#lightningd" to run lightningd without having to manually clone the repo. This make use of the flake output apps.

I've added a new .version file to source control which is required in order to make a pure nix flake (fully deterministic, reproducible builds without dependencies on external environment). It will be updated during the release by the update-versions command.

I've added a new target to the CI workflow that checks the nix package builds successfully.

@JosephGoulden JosephGoulden mentioned this pull request Sep 10, 2024
13 tasks
@chrisguida
Copy link
Contributor

Hmm, for some reason it's not working for me on Ubuntu

$ nix develop
error:
       … while updating the lock file of flake 'git+file:///home/cguida/work/lightning/elementsproject?ref=refs/heads/pr7656&rev=0fc6bdf6d11a8a2a33567e480f0f7207a0172edc'

       error: unsupported tarball input attribute 'lastModified'
(cln-meta-project-py3.12) 2024-09-16 12:45:22 cguida@cg-acer:~/work/lightning/elementsproject$ nix --version
nix (Nix) 2.17.0

got any ideas?

@chrisguida
Copy link
Contributor

ok i ran sudo -i nix upgrade-nix which upgraded my nix


$ sudo -i nix upgrade-nix
[sudo] password for cguida: 
replacing old 'nix-2.17.0'
installing 'nix-2.18.5'
building '/nix/store/yrgxb0lilsz84d1m8sllqq7pf6mxn5vr-user-environment.drv'...

Now it works!!

New problem though, the Rust build is failing with a giant pile of errors:

image

Any ideas on that?

@JosephGoulden
Copy link
Contributor Author

All those errors stem from the "corrupt metadata" of lazy_static. I've not seen that before, can you try deleting your .cargo/registry, or at least that crate. And run cargo clean as well.

@chrisguida
Copy link
Contributor

I ran rm -rf ~/.cargo/registry and now it builds!

$ lightningd/lightningd --version
v24.08-11-g0fc6bdf-modded

Seems like maybe the dev shell should handle this though, don't you think?

@chrisguida
Copy link
Contributor

chrisguida commented Sep 18, 2024

nix build .?submodules=1 works too!! not sure if this would have worked if I hadn't deleted the cargo registry though.

@vincenzopalazzo
Copy link
Collaborator

I ran rm -rf ~/.cargo/registry and now it builds!

This does not sounds right? the all point of nix is to have a sandbox, so ~/ should not be your home

@JosephGoulden
Copy link
Contributor Author

I ran rm -rf ~/.cargo/registry and now it builds!

This does not sounds right? the all point of nix is to have a sandbox, so ~/ should not be your home

This is because I have not written derivations specifically for the cargo builds yet. I can do that later using crane if desired. At the moment the cargo builds are done by the Makefile with the cargo nativeBuildInput. It's still an improvement on the current situation as everyone using the flake will have the same version of cargo.

@JosephGoulden
Copy link
Contributor Author

Hi @ShahanaFarooqui I'm not sure how to progress this PR. I can see you have done the past few releases. Would you be able to comment please? In particular about the .version file I proposed.

@ShahanaFarooqui
Copy link
Collaborator

@JosephGoulden Sorry, I am not familiar with Nix currently, so I might not be helpful here.

But as for this PR, with @chrisguida and @vincenzopalazzo reviewing, it is already in good hands.

Regarding the .version file, my suggestion might not apply in this situation, but just wanted to propose using the existing ./version_gen.h instead of adding another file for the same purpose (if that makes sense).

@chrisguida
Copy link
Contributor

@JosephGoulden I think you should see if you can incorporate crane or some other framework to make sure the cargo deps are reproducible.

Lmk if you want to pair on it, I've never built anything on crane but I have some examples that other people have written that could maybe help.

Another option would be to simply disable rust since it's only needed for grpc atm, though there is also a new rust rest plugin that will hopefully be included as a built-in soon.

If you think it will take too long, we can still merge this and maybe we can get more eyes on it :)

@chrisguida
Copy link
Contributor

For instance this project uses naersk and it works really well

https://github.com/chrisguida/smaug/blob/update-upstream/flake.nix

@chrisguida
Copy link
Contributor

Here's another example from @RCasatta in the sling plugin daywalker90/sling@11abe1b

@JosephGoulden
Copy link
Contributor Author

@chrisguida I added a crane derivation to build the cargo workspace. The cln-grpc binary gets copied into the plugins directory by the postInstall phase of the default package. It will be easy to add the new rest plugin when that's ready.

As you know there is lots more we could do with nix but I think this is a good start.

@JosephGoulden
Copy link
Contributor Author

@JosephGoulden Sorry, I am not familiar with Nix currently, so I might not be helpful here.

But as for this PR, with @chrisguida and @vincenzopalazzo reviewing, it is already in good hands.

Regarding the .version file, my suggestion might not apply in this situation, but just wanted to propose using the existing ./version_gen.h instead of adding another file for the same purpose (if that makes sense).

Thanks @ShahanaFarooqui. I did consider trying to reuse version_gen.h but really I think its better to have a .version file with only the version in it. Then other tools can use that to generate their required format.

@chrisguida
Copy link
Contributor

Hmm build still failing. Maybe i need to update the submodules?
image

@chrisguida
Copy link
Contributor

Ok that seems to have worked...

Now i get
image

It actually does work, just prints out a weird warning about clnrest.

Anyway cool, I'm good with this as long as there's documentation somewhere to tell the user to manually update the submodules by running git submodule update (maybe --init --recursive too?)

@chrisguida
Copy link
Contributor

The Rust build does seem to take very long and makes my computer really hot! But at least it works 🚀

@JosephGoulden
Copy link
Contributor Author

The Rust build does seem to take very long and makes my computer really hot! But at least it works 🚀

It's a release build, that's normal.

@JosephGoulden
Copy link
Contributor Author

Ok that seems to have worked...

Now i get image

It actually does work, just prints out a weird warning about clnrest.

Anyway cool, I'm good with this as long as there's documentation somewhere to tell the user to manually update the submodules by running git submodule update (maybe --init --recursive too?)

Yeah git submodules are a pain. There is a reference to it in the getting-started guide. It's not specific to nix, you need to do that however you build the project.

I could add a new section to the developer guide about nix and mention it there?

Also I added a new variable to the plugins Makefile so python plugins can easily be excluded, which will fix that warning.

@chrisguida
Copy link
Contributor

It's a release build, that's normal.

I've done cln-grpc build a lot, and it's never this intense. But again, I'm happy with it for now, let's ship it!

It's not specific to nix, you need to do that however you build the project.

No, the Makefile automatically updates the submodules. I've never needed to update them manually.

But again, as long as it's documented I think we're good :)

@JosephGoulden
Copy link
Contributor Author

hi @chrisguida will you be able to approve this then?

@chrisguida
Copy link
Contributor

I'll ACK if you document the git submodule thing somewhere.

But someone else will have to merge :)

@JosephGoulden
Copy link
Contributor Author

I'll ACK if you document the git submodule thing somewhere.

But someone else will have to merge :)

I've added some Nix documentation to the developers guide. Who can we ask to merge it then?

@chrisguida
Copy link
Contributor

Looks great!

Thanks so much for putting this together @JosephGoulden !

ACK 93103ed

@ShahanaFarooqui @vincenzopalazzo I think this is good to go, what do you think?

@chrisguida
Copy link
Contributor

Let's close #6650 when this is merged too

Copy link
Collaborator

@vincenzopalazzo vincenzopalazzo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 93103ed

Thanks for working on this

@ShahanaFarooqui
Copy link
Collaborator

@JosephGoulden Please re-base it on master to trigger the CI again. I do not have permission to push changes to your fork.

@ShahanaFarooqui
Copy link
Collaborator

ShahanaFarooqui commented Oct 18, 2024

@JosephGoulden The CI is failing because we need changelog entry in at least one of the commits.

@JosephGoulden
Copy link
Contributor Author

Added a change log entry.

Thanks for helping me to get this in. Feel free to tag me in any related issues or pull requests.

@vincenzopalazzo vincenzopalazzo merged commit 34187bb into ElementsProject:master Oct 21, 2024
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants