Error in untested AwkwardForth case

[renyhp]

Hi, I have just installed uproot, but I can't get it to read a tree.

My code is very simple:

import uproot
f = uproot.reading.open("myfile.root")
t = f.get("aTree")
t['TheOnlyBranch'].array()

This just crashes with this stacktrace:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/venv/lib/python3.10/site-packages/uproot/behaviors/TBranch.py", line 834, in arrays
    _ranges_or_baskets_to_arrays(
  File "/tmp/venv/lib/python3.10/site-packages/uproot/behaviors/TBranch.py", line 3124, in _ranges_or_baskets_to_arrays
    uproot.source.futures.delayed_raise(*obj)
  File "/tmp/venv/lib/python3.10/site-packages/uproot/source/futures.py", line 36, in delayed_raise
    raise exception_value.with_traceback(traceback)
  File "/tmp/venv/lib/python3.10/site-packages/uproot/behaviors/TBranch.py", line 3066, in basket_to_array
    basket_arrays[basket.basket_num] = interpretation.basket_array(
  File "/tmp/venv/lib/python3.10/site-packages/uproot/interpretation/objects.py", line 139, in basket_array
    output = self.basket_array_forth(
  File "/tmp/venv/lib/python3.10/site-packages/uproot/interpretation/objects.py", line 206, in basket_array_forth
    output = self._discover_forth(
  File "/tmp/venv/lib/python3.10/site-packages/uproot/interpretation/objects.py", line 286, in _discover_forth
    output[i] = self._model.read(
  File "/tmp/venv/lib/python3.10/site-packages/uproot/containers.py", line 609, in read
    return uproot.deserialization.read_object_any(
  File "/tmp/venv/lib/python3.10/site-packages/uproot/deserialization.py", line 274, in read_object_any
    obj = cls.read(chunk, cursor, context, file, selffile, parent)
  File "/tmp/venv/lib/python3.10/site-packages/uproot/model.py", line 1372, in read
    versioned_cls.read(
  File "/tmp/venv/lib/python3.10/site-packages/uproot/model.py", line 867, in read
    self.read_members(chunk, cursor, context, file)
  File "<dynamic>", line 567, in read_members
KeyError: dtype('int8')

The branch does contain subbranches, some of them are of type int8 and uint8.

Am I doing anything wrong or is this a bug to be reported?

[Moelf]

do you have a sample file?

[renyhp]

I tried patching _awkward_forth.convert_dtype() by adding

    elif format == "b":
        return "int8"

but now I get:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/uproot5/src/uproot/behaviors/TBranch.py", line 1818, in array
    _ranges_or_baskets_to_arrays(
  File "/tmp/uproot5/src/uproot/behaviors/TBranch.py", line 3124, in _ranges_or_baskets_to_arrays
    uproot.source.futures.delayed_raise(*obj)
  File "/tmp/uproot5/src/uproot/source/futures.py", line 36, in delayed_raise
    raise exception_value.with_traceback(traceback)
  File "/tmp/uproot5/src/uproot/behaviors/TBranch.py", line 3066, in basket_to_array
    basket_arrays[basket.basket_num] = interpretation.basket_array(
  File "/tmp/uproot5/src/uproot/interpretation/objects.py", line 139, in basket_array
    output = self.basket_array_forth(
  File "/tmp/uproot5/src/uproot/interpretation/objects.py", line 254, in basket_array_forth
    context["forth"].vm.resume()
ValueError: 'read beyond' in AwkwardForth runtime: tried to read beyond the end of an input

Using library="np" or library="pd" works, though!

[jpivarski]

Thanks for finding this other missing piece; I've added it to the PR.

library="np" and library="pd" both bypass the AwkwardForth reading, switching to a much slower (like, 400× slower) Python reader. It's possible for you to do that with library="ak" by setting

t["your/branch"].interpretation._forth = False

before reading the array. This is not a public interface (see the underscore), so be aware that it's not officially supported—we want to get the AwkwardForth right.

You've encountered a case that we couldn't test for because of the range of ROOT formats we have available. It would be great to add your sample to our testing suite, but I know that's not something we can do.

How about this, can you print out

t["your/branch"].debug_array(0).tobytes()

and

t["your/branch"].typename

so I can try to step through it? I have the class_code and the Forth code. I'm a little suspicious of

    0 do
    byteoffsets I-> stack
    stream seek
    0 stream skip 
6 stream skip
10 stream skip 
0 stream skip 
6 stream skip
10 stream skip 

because you have only one TVector3, which has a 16 byte header to skip. Maybe the fact that your SubBranch is a TObject means there's another 4 or 6 bytes to skip, but not 16.

Another possibility: could you send me your file by email? I would keep it private.

[renyhp]

It might be relevant at this point to mention that this object is a custom one, so t['TheOnlyBranch/SubBranch'].typename is just something like CustomObject*.

I think I can provide you with a sample file by email, but it will take a while (maybe next week).

[jpivarski]

I know; I expected the typename to be SubBranch (because that's in the name of the class provided by class_code). I just wanted to make sure that it's not inside of an STL container, because that would change my interpretation of the bytes.

I can wait until next week.

I've half-convinced myself that the first 16 bytes is the SubBranch's size header + TObject header and the next 16 bytes is the TVector3's size header + TObject header, so it wouldn't explain why this is running off the end of the data stream.

[renyhp]

Writing here too for documentation: I have sent @jpivarski a confidential test file via email.

An identical tree with 0 entries fails with a different, more meaningful, error:

ValueError: cannot produce Awkward Arrays for interpretation AsObjects(AsPointer(Model_SubBranch)) because

    arbitrary pointer

instead, try library="np" instead of library="ak" or globally set uproot.default_library

[jpivarski]

Thanks for (privately) sending the file! As you noticed when you sent it to me, empty TTrees fail with

ValueError: cannot produce Awkward Arrays for interpretation
AsObjects(AsPointer(Model_SubBranch)) because

     arbitrary pointer

instead, try library="np" instead of library="ak" or globally set
uproot.default_library

and that is the fundamental error. When executed in library="np" (arbitrary Python objects in NumPy arrays), it goes through read_object_any, which deserializes a pointer type in C++, which could be a polymorphic class (which could be a union type in Awkward Array if we knew the set of possible types, which we don't) and could have reference cycles (Awkward Arrays are strictly tree-like). The Python code in library="np" can handle those dynamic situations, but Awkward Array and AwkwardForth can't.

The AwkwardForth code that was generated is not correct because the read_object_any function doesn't generate any AwkwardForth code. But then again, there is no AwkwardForth code that would be correct, anyway.

So the issue here is just that the above error message didn't happen sooner. The zero-entry TTrees got to the point where this error is shown because they had nothing to deserialize and didn't error-out there. Sorry, but library="np" is your only option for this type of data. With PR #838, though, you'd get an error message saying that.

I'm deleting your file now, so that I don't mistakenly post it publicly in the future.