Failed to reproduce the results in the paper on KITTI

[Tord-Zhang]

Hi, I train the PackNet with train_kitti.yaml and the dataset split you provided, But the results is far more worse than the numbers in the paper. I get abs_rel 0.121 while the results in the paper was about 0.07.

This is the config I used for training:
model: name: 'SelfSupModel' optimizer: name: 'Adam' depth: lr: 0.0002 pose: lr: 0.0002 scheduler: name: 'StepLR' step_size: 30 gamma: 0.5 depth_net: name: 'PackNet01' version: '1A' pose_net: name: 'PoseNet' version: '' params: crop: 'garg' min_depth: 0.0 max_depth: 80.0 datasets: augmentation: image_shape: (192, 640) train: batch_size: 4 dataset: ['KITTI'] path: ['datasets/KITTI_raw'] split: ['data_splits/eigen_zhou_files.txt'] depth_type: ['velodyne'] repeat: [2] validation: dataset: ['KITTI'] path: ['datasets/KITTI_raw'] split: ['data_splits/eigen_val_files.txt', 'data_splits/eigen_test_files.txt'] depth_type: ['velodyne'] test: dataset: ['KITTI'] path: ['datasets/KITTI_raw'] split: ['data_splits/eigen_test_files.txt'] depth_type: ['velodyne'] checkpoint: filepath: kitti_ckpt monitor: 'rmse_pp_gt' monitor_index: 0 mode: 'min'

and this is the result I get:

[VitorGuizilini-TRI]

Thank you for that, I will take a look to see if there is something wrong from our end. What is your hardware configuration for training?

[Tord-Zhang]

Thank you for that, I will take a look to see if there is something wrong from our end. What is your hardware configuration for training?

@VitorGuizilini The number of GPUs are 6. All V100 GPUs. Below is the command:

#!/bin/bash
NGPUS=$1
LOG_FILE=$2
echo $NGPUS
MPI_CMD="mpirun -allow-run-as-root
-np ${NGPUS}
-H localhost:${NGPUS}
-x MASTER_ADDR=127.0.0.1
-x MASTER_PORT=23457
-x HOROVOD_TIMELINE
-x OMP_NUM_THREADS=1
-x KMP_AFFINITY='granularity=fine,compact,1,0'
-bind-to none -map-by slot -x NCCL_DEBUG=INFO -x NCCL_MIN_NRINGS=4
--report-bindings"
COMMAND="python3 scripts/train.py configs/train_kitti.yaml"
bash -c "${MPI_CMD} ${COMMAND}" 2>&1 | tee >(sed -r 's/\x1b[[0-9;]*m//g' > ${LOG_FILE})

[liortalker]

The problem is probably that you are training with a resized image (datasets: augmentation: image_shape: (192, 640)).
Try using the crop in the default YAML:
datasets:
augmentation:
crop_train_borders: (-352, 0, 0.5, 1216)
crop_eval_borders: (-352, 0, 0.5, 1216)