Errors in loading model checkpoint

[icecap360]

Using the checkpoint for visual genome, I get the following errors when I try run evaluate_egtr.py. The error relates to missing weights and size mismatches.

[icecap360]

@jinbae

[jinbae]

It looks like DETR weights were used rather than EGTR weights.
Would you like to try again using this link?

[geoleva]

There are several problems with this.

First, for Vis Genome model, the labels (150 total) are abstract (LABEL_0 to LABEL_149). I checked, and OI model's config labels are also abstract. So, how do I know what are the objects detected?
The labels are defined in "config.json" file - can we assume to use the first 150 COCO labels? From here: https://github.com/nightrome/cocostuff/blob/master/labels.md

[geoleva]

when executing _load_pretrained_model, here is what I see for missing_keys (which are the modules in initialized Deformable DETR - modules loaded from checkpoint dict states). Essentially, these are modules that are initialized in DETR (based on config) but are not loaded from state dict, so they are un-initialized (total of 22), not "pre-trained":

['proj_q.1.weight', 'connectivity_layer.layers.0.bias', 'rel_predictor.layers.2.bias', 'rel_predictor.layers.1.bias', 'proj_q.3.bias', 'proj_k.3.weight', 'proj_q.3.weight', 'proj_k.4.bias', 'connectivity_layer.layers.1.weight', 'rel_predictor.layers.2.weight', 'proj_k.5.weight', 'proj_k.1.weight', 'proj_k.1.bias', 'connectivity_layer.layers.0.weight', 'proj_q.5.weight', 'rel_predictor.layers.1.weight', 'rel_predictor.layers.0.weight', 'proj_q.1.bias', 'connectivity_layer.layers.2.bias', 'final_obj_proj.bias', 'proj_q.0.bias', 'proj_k.2.bias', 'proj_q.2.weight', 'proj_q.4.bias', 'final_obj_proj.weight', 'triplet_dist', 'connectivity_layer.layers.1.bias', 'proj_k.0.weight', 'rel_predictor_gate.bias', 'proj_k.2.weight', 'final_sub_proj.weight', 'connectivity_layer.layers.2.weight', 'proj_q.4.weight', 'proj_k.4.weight', 'proj_q.2.bias', 'proj_q.5.bias', 'proj_k.3.bias', 'proj_k.0.bias', 'rel_predictor_gate.weight', 'rel_dist', 'proj_q.0.weight', 'rel_predictor.layers.0.bias', 'proj_k.5.bias', 'final_sub_proj.bias']

[geoleva]

It looks like DETR weights were used rather than EGTR weights. Would you like to try again using this link?

@jinbae - this is the same file, it does not contain the modules for relation prediction (see above), so I think this is a checkpoint with DETR's state dict and not a full EGTR model's

[geoleva]

@icecap360 and @jinbae: I think I somewhat figured out the steps:

(1) The Deformable DETR model is initialized via call:

model = DetrForSceneGraphGeneration.from_pretrained(
args.architecture, config=config, ignore_mismatched_sizes=True
)

At this point, there are warnings, because the "architecture" by default is "SenseTime/deformable-detr", which does not match the EGTR config with 150 class labels and 300 queries. But this warning can be ignored.

(2) After, we load the state dict from the specific checkpoint.

Here are two issues:

• The labels in the config.json file are abstract (LABEL_0,...,LABEL_149), so it is not clear what things the DETR is detecting.
• The checkpoint file is for Deformable DETR, not the full EGTR model, so the relation predictors (all proj, rel predictor, connectivity layer) parameters are not loaded.

Two questions for @jinbae :

1. Could you please provide a list of 150 labels that your Deformable DETR was trained on?
2. Can the new file be posted that contains state dict for the full EGTR model (DetrForSceneGraphGeneration), and not just Detr Object detector (DeformableDetrForObjectDetection) as is currently?

[geoleva]

@jinbae - found 150 labels used in VG-150 dataset. Still, would be nice to have the config.json fixed with these. I am attaching the updated config I did.

So, only one thing remaining - would you be able to post a checkpoint with full EGTR model (and not DETR detector, as is currently)?

Thanks!
config.json

[jinbae]

Checkpoints

I downloaded and checked the files again and there were no problems.

• pretrained_detr: https://drive.google.com/file/d/1vEN-Nat8CtrqP9LoKj0llgUJjGoI7CX7/view
• egtr: https://drive.google.com/file/d/18phcRxbrEI7HqIuM2OLAPuwAF5k3pUC2/view

Please check that you have downloaded and loaded the correct file.

Object Categories

• vg: dataset/visual_genome/test.json
• oi: dataset/open-imagev6/annotations/categories_dict.json

You can find object categories in the paths above.
Please note that object categories in the config are not important in the process of training or evaluation.

[geoleva]

@jinbae - :

1. There is no folder "dataset" in the EGTR commit, and hence those JSON files aren't accessible. Is it possible to commit them? I think I figured out the list, but wanted to double-check.
2. I think the "warning messages" were due to the init of the Detr for Scene generation from the Detr for object detection... So, it all should be good now. Sorry!

[jinbae]

The readme explains how to download datasets.
https://github.com/naver-ai/egtr?tab=readme-ov-file#download-dataset

However, since you don't seem to be using those datasets, I've uploaded the object categories directly here.

• vg_objects.json
• oi_objects.json

[geoleva]

The Visual Genome project distribution does not seem to have this file - at least I could not find it. It has the object.json file, but it's the annotations of all objects in all images. It has a synsets file, but it's not the same thing, and has no IDs.

Checked with the VG-150 labels that I found elsewhere, and these seem to perfectly align. Many thanks!