Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DevX boostrap] cut down closure size of iohk/devx #21

Open
yvan-sraka opened this issue Feb 22, 2023 · 7 comments
Open

[DevX boostrap] cut down closure size of iohk/devx #21

yvan-sraka opened this issue Feb 22, 2023 · 7 comments

Comments

@yvan-sraka
Copy link
Contributor

yvan-sraka commented Feb 22, 2023

We should analyse the size of the closure needed for:

nix develop "github:input-output-hk/devx#ghc8107"

… and similar (also --minimal).

And then figure out why they are too large and what could (or should) be dropped.

@yvan-sraka
Copy link
Contributor Author

I was planning to start by giving a try to @lf- experiment: https://jade.fyi/nixcon2022/

The idea would be to copy-paste content of JSON output generated inside https://mercurytechnologies.github.io/looking-glass-viewer/ visualizer! And to obtain them with the command:

% ./nix-closure-graph --lg "github:input-output-hk/devx#devShells.aarch64-darwin.ghc8107" > aarch64-darwin.ghc8107.json

… where nix-closure-graph was pulled from https://github.com/lf-/dotfiles/tree/main/programs/nix-closure-graph, but I got this error:

this path will be fetched (0.00 MiB download, 0.01 MiB unpacked):
  /nix/store/0fd2carac1bh89jlj7r47kgscw1iiic1-nix-shell
error: path '/nix/store/0fd2carac1bh89jlj7r47kgscw1iiic1-nix-shell' is not valid

That I could reproduce using only nix path-info (on which nix-closure-graph rely):

% nix path-info -rsSh "github:input-output-hk/devx#ghc8107"
error: flake 'github:input-output-hk/devx' does not provide attribute 'packages.aarch64-darwin.ghc8107', 'legacyPackages.aarch64-darwin.ghc8107' or 'ghc8107'
% nix path-info -rsSh "github:input-output-hk/devx#devShells.aarch64-darwin.ghc8107"
this path will be fetched (0.00 MiB download, 0.01 MiB unpacked):
  /nix/store/0fd2carac1bh89jlj7r47kgscw1iiic1-nix-shell
error: path '/nix/store/0fd2carac1bh89jlj7r47kgscw1iiic1-nix-shell' is not valid
% nix path-info -rsSh "github:input-output-hk/devx#devShells.aarch64-darwin.ghc8107-minimal"
this path will be fetched (0.00 MiB download, 0.01 MiB unpacked):
  /nix/store/162pdvqwjf504ch4yvpgw1gyz5b5bpjl-nix-shell
error: path '/nix/store/162pdvqwjf504ch4yvpgw1gyz5b5bpjl-nix-shell' is not valid

@yvan-sraka
Copy link
Contributor Author

This works better:

nix path-info -rsSh $(nix print-dev-env --json "github:input-output-hk/devx#ghc8107" | jq -r ".variables.out.value")

… and attached the different outputs:

@yvan-sraka
Copy link
Contributor Author

Add so on the JSON output that could be visualized through https://mercurytechnologies.github.io/looking-glass-viewer/:

… obtained with, e.g.:

./nix-closure-graph --lg $(nix print-dev-env --json "github:input-output-hk/devx#ghc8107-static-minimal" | jq -r ".variables.out.value") > ghc8107-static-minimal.json

@yvan-sraka
Copy link
Contributor Author

Heavy closure (potentially removable?) found while playing with the tool (using ghc8107-minimal.json):

  • /nix/store/lj2f7dis2wyfa1kimxczhkar71lizlm9-Cabal-syntax-lib-Cabal-syntax-3.8.1.0: 3.1 GiB
  • /nix/store/dhf7q353g7bhjin80nyaql6gai1rpc47-R-4.2.1: 1.4 GiB
  • /nix/store/9ck48bfwbwca06zbkg5xks7givmp3k54-ghc-8.10.7-doc: 527.3 MiB
  • /nix/store/q4g579axnjlni3ys2w2iv2879dbdg88c-zulu17.34.19-ca-jdk-17.0.3: 311.2 MiB
  • /nix/store/fcp5vy436gmqcyzlqc2bfjhiyrjskjgz-gfortran-11.2.0: 261.8 MiB
  • /nix/store/6n36dpm9625l4s1s3sqnhwzg8lcy9sxa-ruby-2.7.6: 128.5 MiB

@lf-
Copy link

lf- commented Feb 22, 2023

Hi! The issue you're having with the invalid paths is that Nix is issuing the worst possible error for not having built it yet. That's most likely what's up :)

@yvan-sraka
Copy link
Contributor Author

Hi! The issue you're having with the invalid paths is that Nix is issuing the worst possible error for not having built it yet. That's most likely what's up :)

Oh! Thanks for the explanation, that's somehow what I understood but with the still unlogical fact that running nix develop (and so building the nix-shell derivation, right?) first and then the command still show the error … Still have to investigate a bit to understand why.

And also thanks again for your so handy tool! Having an interactive thing is really useful when we have to inspect the huge number of derivations that contain a closure. Maybe having some extra overkill features like the possibility to mark with colours the graph while playing with it, or to filter out nodes under a given size in the interface would be great. Happy to find extra time in the future to contribute a bit to it. 🙂

@lf-
Copy link

lf- commented Feb 23, 2023

Yeah, for sure, one feature I was wanting to implement but didn't get the chance is search (which actually exists in one of the examples for sigma.js); IMO the prototype there should be rewritten in react or similar; it was already running into "writing plain js is unpleasant" by the time i got the data table built.

Since I am done my internship at Mercury, it's probably best to build a new looking glass viewer forked off the last one, since I am not sure what the reviews situation is given it's not really an official project that is intended to be maintained into the future.

It's possible that it's building a different derivation or something similarly insane; maybe --debug helps? i know that i would generally nix build dev shells for instance.

@yvan-sraka yvan-sraka self-assigned this Mar 7, 2023
@yvan-sraka yvan-sraka removed their assignment Apr 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants