Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Storage engine improvements #44

Open
3 tasks
MatteoJoliveau opened this issue Oct 27, 2020 · 0 comments
Open
3 tasks

Storage engine improvements #44

MatteoJoliveau opened this issue Oct 27, 2020 · 0 comments
Assignees
Labels
enhancement New feature or request epic For long tales and legends

Comments

@MatteoJoliveau
Copy link
Member

The current storage engine is a simple abstraction over cloud object storage services such as S3. A file is received by the server and stored as-is in the backing service. This can be inefficient in case of large files.

The following activities can be carried out to improve the storage engine:

  • Build an abstraction around the hold crate that can implement custom logic around files and stores metadata in CouchDB
  • Split files into equally sized chunks and store those as blobs
  • Compress chunks before writing them to reduce storage space
@MatteoJoliveau MatteoJoliveau added enhancement New feature or request epic For long tales and legends labels Oct 27, 2020
@MatteoJoliveau MatteoJoliveau self-assigned this Oct 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request epic For long tales and legends
Projects
None yet
Development

No branches or pull requests

1 participant