Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: high level repodata access #560

Merged
merged 63 commits into from
May 10, 2024
Merged

feat: high level repodata access #560

merged 63 commits into from
May 10, 2024

Conversation

baszalmstra
Copy link
Collaborator

@baszalmstra baszalmstra commented Mar 8, 2024 •
edited
Loading

This PR introduces the concept of a Gateway into rattler_repodata_gateway.

Fixes #520

baszalmstra and others added 25 commits March 6, 2024 23:56
@baszalmstra
wip
15df8a8
@baszalmstra
docs: more docs
7b2c25f
@baszalmstra
fix: refactor into files
3842349
@baszalmstra
Merge remote-tracking branch 'upstream/main' into feat/gateway_v3
9b8c2e3
@baszalmstra
fix: works with missing subdirs
f8f2de2
@tdejager
feat: remove unused file
cd041f3
@tdejager
Revert "feat: remove unused file"
b2fcdc8 
This reverts commit cd041f3.
@tdejager
test: test for barrier cell
f1981c4
@tdejager
fix: allow unused code
56f9952
@baszalmstra
Merge remote-tracking branch 'upstream/main' into feat/gateway_v3
1aff06b
@baszalmstra
feat: sharded repodata
e67c8c5
@baszalmstra
feat: msgpack
ec64f2f
@baszalmstra
chore: format
52eff36
@baszalmstra
feat: update to latest
4960b2e
@wolfv
fix computing the URL for sharded repodata
ecd9099
@aochagavia @baszalmstra
feat: create SparseRepoData from byte slices (conda#624)
5d66e1f 
Fixes conda#619

---------

Co-authored-by: Bas Zalmstra <zalmstra.bas@gmail.com>
@baszalmstra
fix: expose RepoData
d7cf370
@baszalmstra
fix: base_url has a trailing slash
8672cfd
@baszalmstra
feat: also adds non-recursive option
b68965e
@wolfv
cache the shard index file
69e714f
@wolfv
remove cache file if parsing fails
60ecf29
@baszalmstra
feat: http cache semantics
c942166
@baszalmstra
feat: add token request
4dc3c96
@wolfv
fix create command
d4bf52c
@baszalmstra
fix: enable multiple matchspecs for the same package
9b16da8
@baszalmstra baszalmstra mentioned this pull request Apr 30, 2024
Merged
wolfv
wolfv reviewed May 8, 2024
View reviewed changes
crates/rattler_repodata_gateway/src/gateway/builder.rs
}

/// Sets the channel configuration to use for fetching repodata.
pub fn set_channel_config(&mut self, channel_config: ChannelConfig) -> &mut Self {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need both versions (with and set)?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It enables two APIs that I frequently use:

Gateway::builder().with_channel_config().build()

// and

let mut builder= Gateway::builder();
builder.set_channel_config();
let gateway = builder.finish()

I havent figured out a better way to do this.

@baszalmstra baszalmstra marked this pull request as ready for review May 10, 2024 14:48
@wolfv wolfv merged commit 50dae59 into conda:main May 10, 2024
14 checks passed
This was referenced May 10, 2024
Closed
Merged
baszalmstra added a commit that referenced this pull request May 13, 2024
@baszalmstra
chore: release (#652)
e669af0 
## 🤖 New release
* `rattler_conda_types`: 0.22.1 -> 0.23.0 (⚠️ API breaking changes)
* `rattler_digest`: 0.19.3 -> 0.19.4 (✓ API compatible changes)
* `rattler_package_streaming`: 0.20.8 -> 0.20.9 (✓ API compatible
changes)
* `rattler_networking`: 0.20.5 -> 0.20.6 (✓ API compatible changes)
* `rattler_lock`: 0.22.5 -> 0.22.6 (✓ API compatible changes)
* `rattler_repodata_gateway`: 0.19.11 -> 0.20.0 (⚠️ API breaking
changes)
* `rattler_solve`: 0.21.1 -> 0.21.2 (✓ API compatible changes)
* `rattler`: 0.24.0 -> 0.24.1
* `rattler_shell`: 0.20.2 -> 0.20.3
* `rattler_virtual_packages`: 0.19.9 -> 0.19.10
* `rattler_index`: 0.19.10 -> 0.19.11

### ⚠️ `rattler_conda_types` breaking changes

```
--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.31.0/src/lints/enum_variant_added.ron

Failed in:
  variant ParseChannelError:NonAbsoluteRootDir in /tmp/.tmp0TYRfv/rattler/crates/rattler_conda_types/src/channel/mod.rs:334
  variant ParseChannelError:NotUtf8RootDir in /tmp/.tmp0TYRfv/rattler/crates/rattler_conda_types/src/channel/mod.rs:338

--- failure method_parameter_count_changed: pub method parameter count changed ---

Description:
A publicly-visible method now takes a different number of parameters.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.31.0/src/lints/method_parameter_count_changed.ron

Failed in:
  rattler_conda_types::Channel::from_url now takes 1 parameters instead of 3, in /tmp/.tmp0TYRfv/rattler/crates/rattler_conda_types/src/channel/mod.rs:183
  rattler_conda_types::Channel::from_name now takes 2 parameters instead of 3, in /tmp/.tmp0TYRfv/rattler/crates/rattler_conda_types/src/channel/mod.rs:225
```

### ⚠️ `rattler_repodata_gateway` breaking changes

```
--- failure function_parameter_count_changed: pub fn parameter count changed ---

Description:
A publicly-visible function now takes a different number of parameters.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.31.0/src/lints/function_parameter_count_changed.ron

Failed in:
  rattler_repodata_gateway::fetch::jlap::patch_repo_data now takes 5 parameters instead of 4, in /tmp/.tmp0TYRfv/rattler/crates/rattler_repodata_gateway/src/fetch/jlap/mod.rs:412

--- failure method_parameter_count_changed: pub method parameter count changed ---

Description:
A publicly-visible method now takes a different number of parameters.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.31.0/src/lints/method_parameter_count_changed.ron

Failed in:
  rattler_repodata_gateway::fetch::jlap::JLAPResponse::apply now takes 4 parameters instead of 3, in /tmp/.tmp0TYRfv/rattler/crates/rattler_repodata_gateway/src/fetch/jlap/mod.rs:312

--- failure struct_missing: pub struct removed or renamed ---

Description:
A publicly-visible struct cannot be imported by its prior path. A `pub use` may have been removed, or the struct itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.31.0/src/lints/struct_missing.ron

Failed in:
  struct rattler_repodata_gateway::fetch::DownloadProgress, previously in file /tmp/.tmpRam5A0/rattler_repodata_gateway/src/fetch/mod.rs:202
```

<details><summary><i><b>Changelog</b></i></summary><p>

## `rattler_conda_types`
<blockquote>

##
[0.23.0](rattler_conda_types-v0.22.1...rattler_conda_types-v0.23.0)
- 2024-05-13

### Added
- high level repodata access
([#560](#560))

### Other
- update README.md
</blockquote>

## `rattler_digest`
<blockquote>

##
[0.19.4](rattler_digest-v0.19.3...rattler_digest-v0.19.4)
- 2024-05-13

### Added
- high level repodata access
([#560](#560))

### Other
- update README.md
</blockquote>

## `rattler_package_streaming`
<blockquote>

##
[0.20.9](rattler_package_streaming-v0.20.8...rattler_package_streaming-v0.20.9)
- 2024-05-13

### Added
- high level repodata access
([#560](#560))

### Fixed
- set last modified for zip archive
([#649](#649))

### Other
- update README.md
</blockquote>

## `rattler_networking`
<blockquote>

##
[0.20.6](rattler_networking-v0.20.5...rattler_networking-v0.20.6)
- 2024-05-13

### Added
- high level repodata access
([#560](#560))
- add AuthenticationStorage::from_file()
([#645](#645))

### Other
- update README.md
</blockquote>

## `rattler_lock`
<blockquote>

##
[0.22.6](rattler_lock-v0.22.5...rattler_lock-v0.22.6)
- 2024-05-13

### Added
- high level repodata access
([#560](#560))

### Other
- update README.md
</blockquote>

## `rattler_repodata_gateway`
<blockquote>

##
[0.20.0](rattler_repodata_gateway-v0.19.11...rattler_repodata_gateway-v0.20.0)
- 2024-05-13

### Added
- add clear subdir cache function to repodata gateway
([#650](#650))
- high level repodata access
([#560](#560))

### Other
- update README.md
</blockquote>

## `rattler_solve`
<blockquote>

##
[0.21.2](rattler_solve-v0.21.1...rattler_solve-v0.21.2)
- 2024-05-13

### Added
- high level repodata access
([#560](#560))

### Other
- update README.md
</blockquote>

## `rattler`
<blockquote>

##
[0.24.1](rattler-v0.24.0...rattler-v0.24.1)
- 2024-05-13

### Other
- updated the following local packages: rattler_conda_types,
rattler_digest, rattler_package_streaming, rattler_networking
</blockquote>

## `rattler_shell`
<blockquote>

##
[0.20.3](rattler_shell-v0.20.2...rattler_shell-v0.20.3)
- 2024-05-13

### Other
- updated the following local packages: rattler_conda_types
</blockquote>

## `rattler_virtual_packages`
<blockquote>

##
[0.19.10](rattler_virtual_packages-v0.19.9...rattler_virtual_packages-v0.19.10)
- 2024-05-13

### Other
- updated the following local packages: rattler_conda_types
</blockquote>

## `rattler_index`
<blockquote>

##
[0.19.11](rattler_index-v0.19.10...rattler_index-v0.19.11)
- 2024-05-13

### Other
- updated the following local packages: rattler_conda_types,
rattler_digest, rattler_package_streaming
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/MarcoIeni/release-plz/).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tdejager tdejager tdejager left review comments

@wolfv wolfv wolfv approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Noisy error from rattler_repodata_gateway::fetch
4 participants
@baszalmstra @tdejager @wolfv @aochagavia