feat(install): add support for `--unsafe` install flag #1008

rustdevbtw · 2024-05-11T17:52:46Z

closes #997
closes #991

This PR adds support for the --unsafe install flag to PKGX, which, as proposed by @jhheider and @felipecrs (thanks!), creates additional env stubs for a package on the first run, making subsequent runs make use of the already installed binaries, thus making it faster[1].

The stub

As for the stub, it makes it a bit different, for instance, here's a "safe" node wrapper (installed normally):

#!/bin/sh
if [ "$PKGX_UNINSTALL" != 1 ]; then
  exec pkgx +nodejs.org -- node "$@"
else
  cd "$(dirname "$0")"
  rm node && echo "uninstalled: nodejs.org" >&2
fi

And here's the "unsafe" (as in experimental) node wrapper:

 #!/bin/sh
+#MANAGED BY PKGX
 if [ "$PKGX_UNINSTALL" != 1 ]; then
-  exec pkgx +nodejs.org -- node "$@"
+  ARGS="$*"
+  ENV_FILE="${XDG_CACHE_DIR:-$HOME/.cache}/pkgx/envs/nodejs.org.env"
+  PKGX_DIR="${PKGX_DIR:-$HOME/.pkgx}"
+
+  pkgx_resolve() {
+    mkdir -p "$(dirname "$ENV_FILE")"
+    pkgx +nodejs.org 1>"$ENV_FILE"
+    run
+  }
+  run() {
+    if test -e "$ENV_FILE" && test -e "$PKGX_DIR/nodejs.org/v*/bin/node"; then
+      set -a
+      # shellcheck source=$ENV_FILE
+      . "$ENV_FILE"
+      set +a
+      exec "$PKGX_DIR/nodejs.org/v*/bin/node" "$ARGS"
+    else
+      pkgx_resolve
+    fi
+  }
+  run
 else
-  cd "$(dirname "$0")"
+  cd "$(dirname "$0")" || exit
   rm node && echo "uninstalled: nodejs.org" >&2
 fi

Edit: this version of the stub is outdated, and doesn't match the actual one implemented. Please refer to the code instead.

Additional changes

Additionally, this PR also makes the following behavioural changes to PKGX:

pkgx uninstall now also removes the env stub of the package
pkgx install now checks for both exec pkgx (old behaviour) and #MANAGED BY PKGX (comment) to determine whether that file was created by PKGX.

Usage

To do an unsafe installation, there are two options:

Use the --unsafe flag (e.g pkgx install --unsafe node)
Use the PKGX_UNSAFE_INSTALL environmental variable.

In the presence of both, the --unsafe flag takes precedence.

Caveats

This approach doesn't come without any caveat. It's necessary to document them upon having such a functionality.
As per my knowledge, these are the known issues:

If the cache dir doesn't exist (after installing the package), it fails.

Edit: it has been fixed

If the envs need updating at a later point, that has to be done separately.
If a certain dependency is removed, it doesn't properly check for that, and would error.
These stubs aren't 100% managed by PKGX (unless you remove the env, when they'll be updated by PKGX)

Most (if not all) of them error correctly and offer (potentially) helpful error messages.

[1]

Signed-off-by: Rajdeep Malakar <rajdeepm.dev@gmail.com>

rustdevbtw · 2024-05-11T18:56:16Z

Here is another benchmark, comparing node.unsafe (installed through --unsafe), node.safe (normally installed), pkgx node, and the system-installed (in this case, installed from Arch repos) node:

I believe (though I could be wrong) that the 1.1x diff is because of the shell overhead, and overhead of checking files.

…ts nested package caches from failing)

rustdevbtw · 2024-05-12T16:01:43Z

Changed the mkdir -p (on the first run, or when the cache doesn't exist) to do that on $(dirname ~/.cache/pkgx/envs/gnu.org/emacs.env) so it creates the gnu.org dir also (the install also takes care of this, but if the cache is missing, it'll cause error, as mentioned in 1st caveat). It handles the first caveat.

src/modes/install.ts

src/modes/uninstall.ts

src/parse-args.ts

rustdevbtw · 2024-05-14T19:32:54Z

@jhheider It seems to be okay, but I'm not in PC rn (it's 1:02AM here lol). Can you try it now please, if possible?

jhheider · 2024-05-14T19:44:14Z

@jhheider It seems to be okay, but I'm not in PC rn (it's 1:02AM here lol). Can you try it now please, if possible?

bugs in the script, but i can add a commit to fix them.

jhheider · 2024-05-14T19:45:10Z

but the speed looks good:

pkgx hyperfine "node --version"
Benchmark 1: node --version
  Time (mean ± σ):      26.4 ms ±   2.1 ms    [User: 18.8 ms, System: 3.5 ms]
  Range (min … max):    25.0 ms …  46.3 ms    103 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

rustdevbtw · 2024-05-14T19:46:40Z

@jhheider It seems to be okay, but I'm not in PC rn (it's 1:02AM here lol). Can you try it now please, if possible?

bugs in the script, but i can add a commit to fix them.

Thanks. Sucks to be without laptop/PC, lol. Anyways, thanks again.

rustdevbtw · 2024-05-14T19:47:48Z

but the speed looks good:

pkgx hyperfine "node --version"
Benchmark 1: node --version
  Time (mean ± σ):      26.4 ms ±   2.1 ms    [User: 18.8 ms, System: 3.5 ms]
  Range (min … max):    25.0 ms …  46.3 ms    103 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Is that right after doing the install, or after doing an initial run? It caches only at the first run, so subsequent runs should be faster than the first one. And the first one might interfere with the rest.

src/modes/install.ts

jhheider · 2024-05-15T14:38:06Z

You mean, the env should take priority?

No, I mean if either is set, it should be on. Which is the way you have it, I believe. Setting the envvar to zero shouldn't do anything.

rustdevbtw · 2024-05-15T14:39:52Z

You mean, the env should take priority?

No, I mean if either is set, it should be on. Which is the way you have it, I believe. Setting the envvar to zero shouldn't do anything.

By shouldn't do anything, do you mean it should ignore that? So, if that's set, regardless of its value, it should do an unsafe install?

rustdevbtw · 2024-05-15T14:42:36Z

You mean, the env should take priority?

No, I mean if either is set, it should be on. Which is the way you have it, I believe. Setting the envvar to zero shouldn't do anything.

By shouldn't do anything, do you mean it should ignore that? So, if that's set, regardless of its value, it should do an unsafe install?

If yes, it gets simpler:

function is_unsafe(unsafe: boolean): boolean {
    return Deno.env.get("PKGX_UNSAFE_INSTALL") || unsafe;
}

jhheider · 2024-05-15T14:43:58Z

So, if that's set, regardless of its value, it should do an unsafe install?

No. If it's non-zero, or the --unsafe flag is passed, it should do an unsafe install. If the envvar is zero, it has no effect one way or the other.

rustdevbtw · 2024-05-15T14:45:46Z

The default behaviour (without any env or flag) is to install normally. So, if the env is set to 0, it should do that? Sorry for sounding like an idiot.

…

On Wed, 15 May, 2024, 8:14 pm Jacob Heider, ***@***.***> wrote: So, if that's set, regardless of its value, it should do an unsafe install? No. If it's non-zero, or the --unsafe flag is passed, it should do an unsafe install. If the envvar is zero, it has no effect one way or the other. — Reply to this email directly, view it on GitHub <#1008 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATXDOIZKEBPBKQ6ZCQFUVFDZCNYEJAVCNFSM6AAAAABHSDC4DWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJSG42DCNRYHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

jhheider · 2024-05-15T14:48:11Z

The default behaviour (without any env or flag) is to install normally. So, if the env is set to 0, it should do that?

Exactly. Setting it 0 is the same as not setting it. So, it doesn't affect behavior.

If it's set to 0 and the flag is passed it's unsafe. If it's 0 and no flag, then safe. Set to 0 is a no-op.

rustdevbtw · 2024-05-15T16:52:02Z

The default behaviour (without any env or flag) is to install normally. So, if the env is set to 0, it should do that?

Exactly. Setting it 0 is the same as not setting it. So, it doesn't affect behavior.

If it's set to 0 and the flag is passed it's unsafe. If it's 0 and no flag, then safe. Set to 0 is a no-op.

So, this:

function is_unsafe(unsafe: boolean): boolean {
    const env = parseInt(Deno.env.get("PKGX_UNSAFE_INSTALL") || "0") ? true : false;
    return env || unsafe;
}

With this, there are six possible combinations:

ENV doesn't exist, but Flag is used == unsafe.
ENV doesn't exist, and Flag isn't used == normal.
ENV exists and is 1, but Flag isn't used == unsafe.
ENV exists and is 0, but Flag isn't used == normal.
ENV exists and is 1, and Flag is used == unsafe.
ENV exists and is 0, and Flag is used == unsafe.

rustdevbtw · 2024-05-15T19:32:42Z

@jhheider implemented it, is it okay? If so, can you please test that the env works as expected? (It probably does, but still a test is a good idea)

jhheider · 2024-05-16T16:39:41Z

right, i don't think that changed anything. your original implementation checked the truth value of the env || the flag. so, it should still be working correctly.

rustdevbtw · 2024-05-17T19:44:30Z

So @jhheider anything else that's left? (Maybe check if it's by PKGX part? We can improve that)

Also @mxcl can you please share thoughts?

jhheider · 2024-05-17T21:52:18Z

looks like good work

felipecrs · 2024-06-07T16:36:28Z

pkgx install now checks for pkgx (instead of exec pkgx) in a script to consider it as installed by PKGX (because unsafe installations don't use exec pkgx)

I've got to say this approach is just as not future-proof as the previous. Perhaps we should fix it for good by adding some known comment to the stubs? Like:

#!/bin/sh
# managed by pkgx

felipecrs · 2024-06-07T16:41:50Z

as proposed by @jhheider (thanks!)

Guys, I don't want to be silly, but I had proposed this on Apr 11. @jhheider is there a chance you missed my comment and later came up with the same proposal?

Don't get me wrong, all I want is pkgx to be improved. I'm asking for pure curiosity.

rustdevbtw · 2024-06-07T16:46:09Z

pkgx install now checks for pkgx (instead of exec pkgx) in a script to consider it as installed by PKGX (because unsafe installations don't use exec pkgx)

I've got to say this approach is just as not future-proof as the previous. Perhaps we should fix it for good by adding some known comment to the stubs? Like:
#!/bin/sh
# managed by pkgx

That is possible, but it'll break compatibility with the stubs installed by older versions. PKGX wouldn't detect them as installed by PKGX if we completely move to this approach. As for unsafe install, I've had the idea to see and check for .env files (generated for unsafe installations only). That's not as reliable, though, because it would fail if the cache was somehow deleted, or not yet generated.

felipecrs · 2024-06-07T17:08:40Z

That is possible, but it'll break compatibility with the stubs installed by older versions. PKGX wouldn't detect them as installed by PKGX if we completely move to this approach.

Perhaps we can keep searching for pkgx while still in pkgx 1.x, and maybe drop support for it in pkgx 2.x. Anyway, this can be done in another PR, no need to have it done here.

jhheider · 2024-06-07T17:16:47Z

as proposed by @jhheider (thanks!)

Guys, I don't want to be silly, but I had proposed this on Apr 11. @jhheider is there a chance you missed my comment and later came up with the same proposal?

Don't get me wrong, all I want is pkgx to be improved. I'm asking for pure curiosity.

Ha. I don't specifically recall your comment, but, yes, we were talking in discord about what and how to cache. I certainly don't deserve the credit for the idea, but we came to the same conclusion as you.

rustdevbtw · 2024-06-07T17:21:14Z

maybe i read that proposal before coming up with this PR, and forgot about it before doing it. it is certainly inspired by your proposal. so, i think you deserve the credit!

…

On Sat, Jun 8, 2024 at 1:42 AM Felipe Santos ***@***.***> wrote: as proposed by @jhheider (thanks!) Guys, I don't want to be silly, but I had proposed this on Apr 11. @jhheider is there a chance you missed my comment and later came up with the same proposal? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

rustdevbtw · 2024-06-07T17:21:50Z

That is possible, but it'll break compatibility with the stubs installed by older versions. PKGX wouldn't detect them as installed by PKGX if we completely move to this approach.

Perhaps we can keep searching for pkgx while still in pkgx 1.x, and maybe drop support for it in pkgx 2.x. Anyway, this can be done in another PR, no need to have it done here.

That sounds like a good option. What do you think @jhheider?

We can also add that to unsafe install now (which is in scope of this PR), because backwards compatibility doesn't exist for it.

jhheider · 2024-06-07T17:22:51Z

sure, sounds like a good idea.

rustdevbtw · 2024-06-07T17:26:52Z

does `#MANAGED BY PKGX` sound good for that comment? (it's unique enough to not cause conflicts)

jhheider · 2024-06-07T17:33:16Z

go for it!

Signed-off-by: Rajdeep Malakar <rajdeepm.dev@gmail.com>

rustdevbtw · 2024-06-07T18:01:06Z

go for it!

done!

rustdevbtw · 2024-06-07T18:11:09Z

Also, updated the example to reflect the current diff (as of now)

coveralls · 2024-09-06T14:38:33Z

Pull Request Test Coverage Report for Build 10740267243

Details

2 of 5 (40.0%) changed or added relevant lines in 1 file are covered.
10 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-1.5%) to 91.389%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/parse-args.ts	2	5	40.0%

Files with Coverage Reduction	New Missed Lines	%
src/utils/execve.ts	10	81.72%

Totals
Change from base Build 10740131780:	-1.5%
Covered Lines:	1475
Relevant Lines:	1603

💛 - Coveralls

mxcl · 2024-09-06T14:40:13Z

Nice feature. Additive so easy merge.

rustdevbtw added 3 commits May 11, 2024 23:03

feat(install): add support for --unsafe install

7235159

Signed-off-by: Rajdeep Malakar <rajdeepm.dev@gmail.com>

forgot to remove debug

ae5fa3a

--unsafe takes precedence, not the env

132605a

Signed-off-by: Rajdeep Malakar <rajdeepm.dev@gmail.com>

dosubot bot added the size:M This PR changes 30-99 lines, ignoring generated files. label May 11, 2024

rustdevbtw added 2 commits May 11, 2024 23:24

let's just warn if deleting cache fails

31d63d6

Signed-off-by: Rajdeep Malakar <rajdeepm.dev@gmail.com>

use pre-parsed unsafe flag

cae08c4

Signed-off-by: Rajdeep Malakar <rajdeepm.dev@gmail.com>

do mkdir -p on the expected dir instead of just the cache one (preven…

09b1e76

…ts nested package caches from failing)

jhheider reviewed May 14, 2024

View reviewed changes

rustdevbtw added 2 commits May 15, 2024 00:56

improve it

b8dd655

use Config and Path hooks

b2c483c

fix script invocations

8dc97eb

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels May 14, 2024

jhheider added 2 commits May 14, 2024 16:09

linter making more work

963bce6

a few more

a5ab5db

dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels May 14, 2024

jhheider reviewed May 14, 2024

View reviewed changes

src/modes/install.ts Outdated Show resolved Hide resolved

jhheider and others added 3 commits May 14, 2024 16:13

last one

b39b3c3

Use .slice() instead of .pop()

d3f2779

Use .slice() instead of .pop(), in uninstall too

5366c08

rustdevbtw added 2 commits May 15, 2024 23:45

handle PKGX_UNSAFE_INSTALL=0 env

322d14d

fix it

22f7a97

rustdevbtw requested a review from jhheider June 1, 2024 14:47

figure out what's managed by pkgx, but with comments.

f33e732

Signed-off-by: Rajdeep Malakar <rajdeepm.dev@gmail.com>

Merge branch 'main' into main

5b69c95

mxcl merged commit 4c0a903 into pkgxdev:main Sep 6, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(install): add support for `--unsafe` install flag #1008

feat(install): add support for `--unsafe` install flag #1008

rustdevbtw commented May 11, 2024 •

edited

Loading

rustdevbtw commented May 11, 2024

rustdevbtw commented May 12, 2024 •

edited

Loading

rustdevbtw commented May 14, 2024

jhheider commented May 14, 2024

jhheider commented May 14, 2024

rustdevbtw commented May 14, 2024

rustdevbtw commented May 14, 2024

jhheider commented May 15, 2024

rustdevbtw commented May 15, 2024

rustdevbtw commented May 15, 2024

jhheider commented May 15, 2024

rustdevbtw commented May 15, 2024 via email

jhheider commented May 15, 2024 •

edited

Loading

rustdevbtw commented May 15, 2024 •

edited

Loading

rustdevbtw commented May 15, 2024

jhheider commented May 16, 2024

rustdevbtw commented May 17, 2024

jhheider commented May 17, 2024

felipecrs commented Jun 7, 2024

felipecrs commented Jun 7, 2024 •

edited

Loading

rustdevbtw commented Jun 7, 2024 •

edited

Loading

felipecrs commented Jun 7, 2024

jhheider commented Jun 7, 2024

rustdevbtw commented Jun 7, 2024 via email

rustdevbtw commented Jun 7, 2024 •

edited

Loading

jhheider commented Jun 7, 2024

rustdevbtw commented Jun 7, 2024 via email •

edited

Loading

jhheider commented Jun 7, 2024

rustdevbtw commented Jun 7, 2024

rustdevbtw commented Jun 7, 2024

coveralls commented Sep 6, 2024

mxcl commented Sep 6, 2024

feat(install): add support for --unsafe install flag #1008

feat(install): add support for --unsafe install flag #1008

Conversation

rustdevbtw commented May 11, 2024 • edited Loading

The stub

Additional changes

Usage

Caveats

rustdevbtw commented May 11, 2024

rustdevbtw commented May 12, 2024 • edited Loading

rustdevbtw commented May 14, 2024

jhheider commented May 14, 2024

jhheider commented May 14, 2024

rustdevbtw commented May 14, 2024

rustdevbtw commented May 14, 2024

jhheider commented May 15, 2024

rustdevbtw commented May 15, 2024

rustdevbtw commented May 15, 2024

jhheider commented May 15, 2024

rustdevbtw commented May 15, 2024 via email

jhheider commented May 15, 2024 • edited Loading

rustdevbtw commented May 15, 2024 • edited Loading

rustdevbtw commented May 15, 2024

jhheider commented May 16, 2024

rustdevbtw commented May 17, 2024

jhheider commented May 17, 2024

felipecrs commented Jun 7, 2024

felipecrs commented Jun 7, 2024 • edited Loading

rustdevbtw commented Jun 7, 2024 • edited Loading

felipecrs commented Jun 7, 2024

jhheider commented Jun 7, 2024

rustdevbtw commented Jun 7, 2024 via email

rustdevbtw commented Jun 7, 2024 • edited Loading

jhheider commented Jun 7, 2024

rustdevbtw commented Jun 7, 2024 via email • edited Loading

jhheider commented Jun 7, 2024

rustdevbtw commented Jun 7, 2024

rustdevbtw commented Jun 7, 2024

coveralls commented Sep 6, 2024

Pull Request Test Coverage Report for Build 10740267243

Details

💛 - Coveralls

mxcl commented Sep 6, 2024

feat(install): add support for `--unsafe` install flag #1008

feat(install): add support for `--unsafe` install flag #1008

rustdevbtw commented May 11, 2024 •

edited

Loading

rustdevbtw commented May 12, 2024 •

edited

Loading

jhheider commented May 15, 2024 •

edited

Loading

rustdevbtw commented May 15, 2024 •

edited

Loading

felipecrs commented Jun 7, 2024 •

edited

Loading

rustdevbtw commented Jun 7, 2024 •

edited

Loading

rustdevbtw commented Jun 7, 2024 •

edited

Loading

rustdevbtw commented Jun 7, 2024 via email •

edited

Loading