Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Improvements on MemmapTensor #194

Open
2 of 6 tasks
vmoens opened this issue Jan 27, 2023 · 1 comment
Open
2 of 6 tasks

[BUG] Improvements on MemmapTensor #194

vmoens opened this issue Jan 27, 2023 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@vmoens
Copy link
Contributor

vmoens commented Jan 27, 2023

Describe the bug

We need to do some woodwork on MemmapTensor:

  • Remove the ownership and replace it by a check over the number of times the array is being opened (for temporary files only)
  • Solve a bug where calling MemmapTensor(memmap_tensor[idx]) returns the wrong view of the MemmapTensor [Feature] apply across multiple TDs #193
  • Improve MemmapTensor creation args, including: creating a Memmap on a specific location ([Feature] More arguments for MemmapTensor construction #189)
  • Improve the way we copy a Memmap from one place to another, if we want and if we don't want to change the filename
  • simplify the logic of MemmapTensor so that extracting a tensor is less confusing (i.e. check as_tensor, _tensor, _load_item and how they interact).
  • the device attribute is confusing, as the data is never really on another device but gets loaded there if needed. The usage is that one can put the device attr as a destination, then move the Memmap from process to process and when accessing the data it'll be on the right device. However, this may give the false illusion that the content of the tensor is stored on the device...

cc @tcbegley

@vmoens vmoens added the bug Something isn't working label Jan 27, 2023
@vmoens vmoens self-assigned this Jan 27, 2023
@tcbegley
Copy link
Contributor

tcbegley commented Jan 27, 2023

I'll also update #189, though the desired no-op behaviour there is blocked by #192.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants