-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf: desperate attempts to unlink output arrays #1305
base: main
Are you sure you want to change the base?
perf: desperate attempts to unlink output arrays #1305
Conversation
--- a/src/uproot/behaviors/TBranch.py
+++ b/src/uproot/behaviors/TBranch.py
@@ -212,10 +212,12 @@ def iterate(
report = report.to_global(global_offset)
popper = [arrays]
del arrays
+ del item
yield popper.pop(), report
else:
popper = [library.global_index(item, global_offset)]
+ del item
yield popper.pop() I'm happy to run your memory-over-time graph with this change, if you have a script. The inner code from the slack thread was this - you did import gc
import uproot
filenames = [{"/tmp/zlib9-jagged0.root": "tree"}] * 5
for batch in uproot.iterate(filenames, step_size="1 GB"):
# gc.collect()
print(repr(batch)) |
I can't believe I missed that! If I'm reading this right, I don't have scripts to produce the memray plots from Slack—I'd lose them anyway, if I kept them—so the messages in Slack include the code. |
I made a to-do item to finish this off. I'll include your diff. |
My loop for lowest memory usage was like this: for i, batch in enumerate(uproot.iterate(filenames, step_size="1 GB")):
# gc.collect()
if i:
objgraph.show_growth()
else:
objgraph.growth()
print(repr(batch))
del batch
(so not very useful; I don't know what ReferenceType is) I am thinking there might be many of these non-functional-programming invocations down the graph (with or without generators) that is making refcounting hard for python. |
I discovered the same thing with Pympler: it was clearly calling |
Well, I added that commit (7cfc360), but it didn't fix the original memory issue. Here's memray of import gc
import uproot
filenames = [{"/tmp/zlib9-jagged0.root": "tree"}] * 5
for batch in uproot.iterate(filenames, step_size="1 GB"):
# gc.collect()
print(repr(batch)) in this PR, before the commit: Here's after the commit: And, just for reference, here it is with the Even if objgraph isn't showing it, the big arrays are still held until garbage collection. I'm setting up this PR to be merged, anyway, since it's a step forward, at least. |
Yeah; I spent some time trying to find out where the cycle(s) is, but no luck. The code goes deep, and I couldn't figure out which tbranch/basket/etc refer to one-enother. Doing more |
No description provided.