Improve versioning of TypeScript to allow for transitive dependencies on pre-releases

[TimvdLippe]

Suggestion

Publish pre-releases with semver compatible version ranges

🔍 Search Terms

semver version

✅ Viability Checklist

My suggestion meets these guidelines:

• This wouldn't be a breaking change in existing TypeScript/JavaScript code
• This wouldn't change the runtime behavior of existing JavaScript code
• This could be implemented without emitting different JS based on the types of the expressions
• This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, new syntax sugar for JS, etc.)
• This feature would agree with the rest of TypeScript's Design Goals.

⭐ Suggestion

Figure out a way on how a project can depend on a pre-release of TypeScript (to verify an upcoming release is working as intended) while preventing package duplication of (transitive) dependencies.

📃 Motivating Example

Unfortunately, this is looking like an NPM "bug"/"feature". Currently, Chrome DevTools eagerly upgrades to pre-releases of upcoming TypeScript versions to test for compatibility with our codebase. Historically, this led us to finding bugs in the JSDoc handling and also uncovered problems in the build system. By reporting these issues early, we are seeing a higher chance of getting them fixed prior to the full release (thank you so much for that!!).

However, we are now trying to add another dependency to our node_modules which transitively depends on parse-literals. parse-literals is a package that directly depends on typescript as well: https://github.com/asyncLiz/parse-literals/blob/1022b479d2cf23f5a16e742e67e3c11f209b51c8/package.json#L59

parse-literals basically says: I don't really care which version of TypeScript you are using, as long as it is newer than TypeScript 2.9.2. Since TypeScript doesn't follow semver (but NPM does), you need to explicitly list all "major" breaking versions. Again, TypeScript doesn't have major breaking releases like this, but NPM doesn't know that.

What this leads to is if you have the following:

{
  "dependencies": {
    "typescript": "4.3.1-rc",
    "parse-literals": "1.2.1"
  },
}

The end result is that typescript is both in node_modules and in node_modules/parse-literals/node_modules. Sadly, this can lead to functional differences between what parse-literals sees and which version of TypeScript we are actually running. Not just that, we also duplicate the whole typescript package (which is big) and thus increase our node_modules folder by a lot (https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2917088 has 1 million lines added, primarily because of TypeScript).

I originally filed asyncLiz/parse-literals#18 and thought "we can probably somehow update the version range of parse-literals to allow for pre-releases", but oh boy I got myself into a rabbit hole of NPM version range shenanigans. (See https://twitter.com/TimvdLippe/status/1397509326554738690)

Things I tried (and sadly didn't work):

1. Use * in parse-literals. Doesn't work, as NPM declares that pre-releases don't match *. This was "broken" once and then fixed: "*" is satisfied by versions with a pre-release npm/node-semver#123
2. Use ^2.9.2 || ^3.0.0 || ^4.0.0" || ^4.0.0-0
3. Use ^2.9.2 || ^3.0.0 || ^4.0.0" || >4.0.0-rc
4. Use ^2.9.2 || ^3.0.0 || ^4.0.0" || @beta || @rc. Based on the versions tab of https://www.npmjs.com/package/typescript I thought "maybe I can use the tags", but that doesn't appear to be working in ranges. That appears to be only working if you do npm i typescript@beta and then it resolves the version range, prior to writing to package-lock.json
5. Use ^2.9.2 || ^3.0.0 || ^4.0.0" || 4.3.1-rc. Yes, even listing the exact release candidate version I wanted to use didn't work either, asreleases have higher precendence than pre-releases. That includes pre-releases of future stable releases
6. Use ^2.9.2 || ^3.0.0 || ^4.0.0" || ^4.0.0-alpha Surprisingly, this worked better! The -alpha suffix is special apparently and will match pre-releases. However, it only matches pre-releases of 4.0.0.... So if you want to say "I just want any pre-release of any 4.X TypeScript version", you would need to do the following: ^2.9.2 || ^3.0.0 || ^4.0.0" || ^4.0.0-alpha || ^4.1.0-alpha || ^4.2.0-alpha || ^4.3.0-alpha || ^4.4.0-alpha || ^4.5.0-alpha || ^4.6.0-alpha || ^4.7.0-alpha || ^4.8.0-alpha || ^4.9.0-alpha
   1. Well, it gets a bit worse. In the example, you can see that the specific pre-release was 4.3.1-rc. Unfortunately, this means that ^4.3.0-alpha doesn't match. You would have to specify ^4.3.1-alpha specifically to make it match
   2. ^2.9.2 || ^3.0.0 || ^4.0.0-alpha does not appear to improve the situation either
7. I even tried *-alpha based on the previous match, but that is simply illegal syntax in semver
8. I also tried specifying "typescript": "https://registry.npmjs.org/typescript/-/typescript-4.3.1-rc.tgz" in our dependencies, but that didn't work either

(Btw, I wasn't aware of https://semver.npmjs.com/ until today linked to me in https://twitter.com/jeronevw/status/1397780765396684801, but it is a great tool to test out all the possible ranges and see what NPM will do to the version resolution)

All of that is to say: if you want to test out a pre-release of TypeScript to potentially discover and report bugs, but you have another transitive dependency on TypeScript, you run into issues. We would like to continue testing pre-release of TypeScript to provide actionable feedback for the team, but we are now considering dropping that in favor of adopting the other package. (The package we want to add allows for further minification of our source code and is a net positive for our end users)

Some options that come to mind, but you might have more ideas:

1. Release pre-releases of TypeScript with a "special" version that can be matched against. E.g. something like 0.0.4-beta.3.1 where this would be the pre-release version of 4.3.1. It seems like this would reliably work, as one could match onto >0.0.4-alpha, which should match 0.0.4-beta.3.1. However, I do admit that this is kind of a wild idea and I don't think anybody has really tried this out
   1. The reason for this weird notation is that 0.4.3-beta.1 wouldn't work, as that wouldn't match ^0.4.0-alpha per the findings above
2. Start following semver. This has been debated many times over in other issues and I don't think this will happen based on these discusssions, but I wanted to list it nonetheless. Especially since I think it will not work if you would start publishing more minor versions, which still wouldn't match versions per the "we only match pre-releases for this exact combination of Major.Minor.Patch range". Therefore, I don't see this as a solution, but I might be wrong on that
3. Don't pre-release versions and don't collect feedback up front. I don't think this is a desirable way forward either, but it might become necessary once typescript becomes a transitive dependency of more projects

💻 Use Cases

To make sure that we only have one version of TypeScript. If we have multiple versions of TypeScript in our node_modules, it can lead to behavioral differences that could potentially make pre-release testing result in false positive. E.g. it all works and then it turns out it didn't, because a transitive dependency was using an older version of TypeScript.

[JoostK]

This sounds like what Yarn solves using resolutions and here's some notes on support for something like that in NPM.

[orta]

Yeah, resolutions is a structural answer to this and the RFC for overrides npm/rfcs#129 in npm which is basically the same was accepted, and already implemented in pnpm - another potential fix is npm/rfcs#23 which allows a module to declare 'only one of these in the module tree'