Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Skuber should make it easier to stream large lists of resources #258

Open
doriordan opened this issue Feb 23, 2019 · 0 comments
Open

Comments

@doriordan
Copy link
Owner

doriordan commented Feb 23, 2019

Skuber now supports using the limit and continue flags in ListOptions, which makes it possible for clients to retrieve large lists of resources in chunks, thus improving efficiency and reducing memory consumption, and mitigating related issues.
However it currently requires the client application to itself implement the paging/continuation processing described at https://kubernetes.io/docs/reference/using-api/api-concepts/#retrieving-large-results-sets-in-chunks , even though that algorithm is general rather than client-specific.
This work could be abstracted by skuber into a new variant of the list API method (e.g. listChunked) that returns an Akka stream of chunk elements, where a chunk is basically a ListResource[O <: ObjectResource] type. Each chunk would be capped in size by a limit parameter`.
Example potential usage - something like:

 val podListChunkedSource: Source[PodList] = k8s.listChunked[PodList](limit=10)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant