Problem accessing subbranches and/or leaves

[ivoschulthess]

Good afternoon everyone,

I need help accessing the subbranches or leaves of a ROOT file. It is a problem that I have never encountered before. I uploaded the file (one of many) here: https://github.com/ivoschulthess/uprootProblem/blob/main/datafile.root.

I am not very used to ROOT itself, but to look and access the data with ROOT, I can do the following:

root [0] TFile *_file0 = TFile::Open("datafile.root")
root [1] TTree *tree = (TTree*)_file0->Get("Neutron Detector")
root [2] tree->Print()
******************************************************************************
*Tree    :Neutron Detector: Event "Neutron Detector", ID 4                         *
*Entries :        1 : Total =           10078 bytes  File  Size =      10094 *
*        :          : Tree compression factor =   1.00                       *
******************************************************************************
*Br    0 :CDT0      : nCDT0/I:CDT0[nCDT0]/i                                  *
*Entries :        1 : Total  Size=       8868 bytes  File Size  =       8291 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    1 :CDT1      : nCDT1/I:CDT1[nCDT1]/i                                  *
*Entries :        1 : Total  Size=        784 bytes  File Size  =        207 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
root [3] tree->Draw("CDT0.CDT0")

This gives me the tree information and draws a histogram of the branch "CDT0.CDT0".

If I try to do the same thing with uproot (5.0.7) using Python 3.11.5, I encounter the following problem:

f = uproot.open('datafile.root')
tree = f['Neutron Detector;1']

# this gives: ['CDT0', 'CDT1']
tree.keys()

# this gives: []
tree['CDT0'].keys()

# this gives "NameError: name 'nCDT0' is not defined"
tree.show()

# this gives me a hint that also uproot sees the subbranches / leaves, e.g.
# 'fName': 'CDT0',
# 'fTitle': 'nCDT0/I:CDT0[nCDT0]/i',
# 'fLeaves': <TObjArray of 2 items at 0x7f5be9ad08d0>,
tree['CDT0'].all_members

Same thing happens when trying to access the other branch CDT1. I tried various things to access the content but it always gives me nothing or an error. It seems uproot does not recognize the leaves properly.

Do you have an idea what the problem is or do you know other ways to access the data?

Thanks in advance for your help and advice.

Cheers,
Ivo

[jpivarski]

There does seem to be something wrong: just trying to look at the interpretation of tree["CDT0"] causes an exception inside of Uproot.

>>> tree["CDT0"].interpretation
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jpivarski/irishep/uproot5/src/uproot/behaviors/TBranch.py", line 1935, in interpretation
    self._interpretation = uproot.interpretation.identify.interpretation_of(
  File "/home/jpivarski/irishep/uproot5/src/uproot/interpretation/identify.py", line 407, in interpretation_of
    _leaf_to_dtype(leaf, getdims=True).newbyteorder(">"),
  File "/home/jpivarski/irishep/uproot5/src/uproot/interpretation/identify.py", line 69, in _leaf_to_dtype
    dims = tuple(eval(m.group(2).replace("][", ", ")))
  File "<string>", line 1, in <module>
NameError: name 'nCDT0' is not defined

(Which is true: there is no TBranch named "nCDT0".) But I can't say for sure that the file is correctly formatted. At least, I don't know how to interpret it.

From ROOT's own tree->Print(), it doesn't have sub-branches within branches tree["CDT0"] and tree["CDT1"]; they each have two leaves (class TLeaf). It's unusual for a ROOT TBranch to have multiple TLeaves (these days), but not problematic. Uproot interprets TBranches, including nested sub-TBranches, as separate arrays and a TBranch with multiple TLeafs as a NumPy structured array. The choice of NumPy structure is mirroring the internal layout of the data: TBranches are stored in separate buffers, whereas a TBranch with multiple TLeafs are interleaved, like a structured array ("array of structs").

The title of the TBranches are both "nCDT1/I:CDT1[nCDT1]/i", which is what's giving Uproot a hard time. Interpreted literally, it would mean that the first leaf has type np.uint32 (unsigned int) and the second leaf is a variable number of np.int32 (int) values—specifically, the number given by the first leaf. We can also see it this way:

>>> tree["CDT0"].member("fLeaves")[0].member("fTitle")
'nCDT0'
>>> tree["CDT0"].member("fLeaves")[1].member("fTitle")
'CDT0[nCDT0]'

That would be possible for branches, which are not interleaved, because the counter can have a different number of values than the counted. Specifically, there's one integer for the counter and who-knows-how-many integers for the counted. Data specified by multiple TLeafs, however, are interleaved—there has to be exactly as many of each.

And yet, that seems to be exactly what this is saying. If I avoid Uproot's own interpretation and look at it through each TBasket's raw bytes,

>>> tree["CDT0"].num_baskets
1
>>> tree["CDT0"].basket(0).data.view(">i4")
array([2048,    0,    0, ...,    0,    0,    0], dtype='>i4')

>>> tree["CDT1"].num_baskets
1
>>> tree["CDT1"].basket(0).data.view(">i4")
array([       27,  53753945, 100000000, 100000000,         0,         0,
               0,         5,         1,        -1,        -1,        -1,
              -1,        -1,        -1,        -1,        -1,  51338476,
         2431479,      2756,      5420,       332,         0,  44542698,
         7819608,   1200644,    215180,      6575], dtype='>i4')

I see that the number of np.int32 values (">i4" is just asking for np.int32 with big-endian byte order) is equal to the first value minus one. The tree["CDT0"] array has 2049 integers, and the first one, 2048, is specifying the size. The tree["CDT1"] array has 28 integers, and the first one, 27, is specifying the size.

So this kind of multi-TLeaf TBranch is interleaved but not equal-sized, so it can't be interpreted as a NumPy structured array. I don't know what to make of it.

Hopefully, though, this gives you a way to work with your data. I can't add the above to Uproot because I don't know how it generalizes, but it should work for your specific problem.

[ivoschulthess]

Thanks a lot.
This is perfect and allows me to work with the data.

To give some context and maybe help others who are searching for a solution with the same question: The ROOT files are produced by the data acquisition system "MIDAS" which is an acronym for Maximum Integrated Data Acquisition System. It was developed by PSI and Triumf.

[agoose77]

If you still have access to the raw MIDAS data, I wrote a MIDAS decoder that leverages Awkward Forth. This does the interpretation and row→columnar conversion in one step, which is convenient. I am not sure what state it's in; I fernet-encrypted my test data and lost the key (I'm judging past @agoose77 too!), but very little has changed in the parts of Awkward Array that it uses since then. It might not be relevant: the word MIDAS seems to mean different things to different people, but for my group it was the MIDAS tape-server data.

https://github.com/agoose77/midas-tape