KITTI odometry training
Training took 7 hours 26 minutes for 10 epochs. Least validation loss is after 8th epoch. Total translation error: 7.832 cm. Total rotation error: 0.678 degrees.
Training loss per iteration (651 iterations per epoch):
Validation loss per iteration (76 iterations per epoch):
Validation example:
Logs
python train_with_sacred.py
WARNING - LCCNet - No observers have been added to this run
INFO - LCCNet - Running command 'main'
INFO - LCCNet - Started
Loss Function Choice: combined
Val Sequence: 00
Ground truth poses are not avaialble for sequence 11.
Ground truth poses are not avaialble for sequence 12.
Ground truth poses are not avaialble for sequence 13.
Ground truth poses are not avaialble for sequence 14.
Ground truth poses are not avaialble for sequence 15.
Ground truth poses are not avaialble for sequence 16.
Ground truth poses are not avaialble for sequence 17.
Ground truth poses are not avaialble for sequence 18.
Ground truth poses are not avaialble for sequence 19.
Ground truth poses are not avaialble for sequence 20.
Ground truth poses are not avaialble for sequence 21.
Ground truth poses are not avaialble for sequence 11.
Ground truth poses are not avaialble for sequence 12.
Ground truth poses are not avaialble for sequence 13.
Ground truth poses are not avaialble for sequence 14.
Ground truth poses are not avaialble for sequence 15.
Ground truth poses are not avaialble for sequence 16.
Ground truth poses are not avaialble for sequence 17.
Ground truth poses are not avaialble for sequence 18.
Ground truth poses are not avaialble for sequence 19.
Ground truth poses are not avaialble for sequence 20.
Ground truth poses are not avaialble for sequence 21.
VAL SET - Not found: /home/jay/datasets/KITTI/odometry_color/sequences/val_RT_left_seq00_5.00_0.50.csv
Generating a new one
Number of the train dataset: 39011
Number of the val dataset: 4541
651
76
Number of model parameters: 69987887
This is -1-th epoch
Init worker 0 with seed 968601691
Init worker 1 with seed 968601692
Init worker 2 with seed 968601693
Init worker 3 with seed 968601694
Init worker 4 with seed 968601695
Init worker 5 with seed 968601696
/home/jay/mambaforge/envs/lcc/lib/python3.6/site-packages/torch/nn/functional.py:2423: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
"See the documentation of nn.Upsample for details.".format(mode))
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument
Iter 50/651 training loss = 1.216, time = 0.0801, time for 50 iter: 182.1077
Iter 100/651 training loss = 0.560, time = 0.0709, time for 50 iter: 155.9080
Iter 150/651 training loss = 0.496, time = 0.0795, time for 50 iter: 156.2185
Iter 200/651 training loss = 0.452, time = 0.0761, time for 50 iter: 155.7937
Iter 250/651 training loss = 0.450, time = 0.0800, time for 50 iter: 158.2878
Iter 300/651 training loss = 0.431, time = 0.0637, time for 50 iter: 158.7071
Iter 350/651 training loss = 0.429, time = 0.0802, time for 50 iter: 156.3049
Iter 400/651 training loss = 0.362, time = 0.0723, time for 50 iter: 156.2439
Iter 450/651 training loss = 0.346, time = 0.0650, time for 50 iter: 153.2599
Iter 500/651 training loss = 0.334, time = 0.0632, time for 50 iter: 154.7639
Iter 550/651 training loss = 0.321, time = 0.0630, time for 50 iter: 158.1921
Iter 600/651 training loss = 0.327, time = 0.0795, time for 50 iter: 159.3211
Iter 650/651 training loss = 0.311, time = 0.1012, time for 50 iter: 137.9026
------------------------------------
epoch -1 total training loss = 0.464
Total epoch time = 2043.13
------------------------------------
Init worker 0 with seed 968601691
Init worker 1 with seed 968601692
Init worker 2 with seed 968601693
Init worker 3 with seed 968601694
Init worker 4 with seed 968601695
Init worker 5 with seed 968601696
Iter 50 val loss = 0.365 , time = 0.07
------------------------------------
total val loss = 0.358
total traslation error: 30.39106699076814 cm
total rotation error: 2.2700494187856344 °
------------------------------------
Model saved as ./checkpoints/kitti/odom/val_seq_00/models/checkpoint_r5.00_t0.50_e-1_0.358.tar
This is 0-th epoch
Init worker 0 with seed 968601791
Init worker 1 with seed 968601792
Init worker 2 with seed 968601793
Init worker 3 with seed 968601794
Init worker 4 with seed 968601795
Init worker 5 with seed 968601796
Iter 50/651 training loss = 0.305, time = 0.0695, time for 50 iter: 192.2153
Iter 100/651 training loss = 0.294, time = 0.0740, time for 50 iter: 155.2757
Iter 150/651 training loss = 0.295, time = 0.0700, time for 50 iter: 152.9292
Iter 200/651 training loss = 0.286, time = 0.0832, time for 50 iter: 156.3506
Iter 250/651 training loss = 0.275, time = 0.0823, time for 50 iter: 157.0801
Iter 300/651 training loss = 0.279, time = 0.0801, time for 50 iter: 154.6650
Iter 350/651 training loss = 0.274, time = 0.0635, time for 50 iter: 154.8831
Iter 400/651 training loss = 0.268, time = 0.0747, time for 50 iter: 156.5389
Iter 450/651 training loss = 0.268, time = 0.0724, time for 50 iter: 157.5728
Iter 500/651 training loss = 0.253, time = 0.0746, time for 50 iter: 158.9319
Iter 550/651 training loss = 0.246, time = 0.0750, time for 50 iter: 160.0717
Iter 600/651 training loss = 0.235, time = 0.0791, time for 50 iter: 157.9720
Iter 650/651 training loss = 0.236, time = 0.0889, time for 50 iter: 138.0961
------------------------------------
epoch 0 total training loss = 0.270
Total epoch time = 2052.70
------------------------------------
Init worker 0 with seed 968601791
Init worker 1 with seed 968601792
Init worker 2 with seed 968601793
Init worker 3 with seed 968601794
Init worker 4 with seed 968601795
Init worker 5 with seed 968601796
Iter 50 val loss = 0.232 , time = 0.06
------------------------------------
total val loss = 0.231
total traslation error: 18.55699067996793 cm
total rotation error: 1.7315288065708936 °
------------------------------------
Model saved as ./checkpoints/kitti/odom/val_seq_00/models/checkpoint_r5.00_t0.50_e0_0.231.tar
This is 1-th epoch
Init worker 0 with seed 968601891
Init worker 1 with seed 968601892
Init worker 2 with seed 968601893
Init worker 3 with seed 968601894
Init worker 4 with seed 968601895
Init worker 5 with seed 968601896
Iter 50/651 training loss = 0.233, time = 0.0712, time for 50 iter: 187.4334
Iter 100/651 training loss = 0.211, time = 0.0712, time for 50 iter: 158.7539
Iter 150/651 training loss = 0.209, time = 0.0723, time for 50 iter: 158.7271
Iter 200/651 training loss = 0.206, time = 0.0650, time for 50 iter: 157.8971
Iter 250/651 training loss = 0.203, time = 0.0640, time for 50 iter: 158.0176
Iter 300/651 training loss = 0.187, time = 0.0763, time for 50 iter: 157.1829
Iter 350/651 training loss = 0.178, time = 0.0682, time for 50 iter: 155.6360
Iter 400/651 training loss = 0.179, time = 0.0698, time for 50 iter: 158.3499
Iter 450/651 training loss = 0.172, time = 0.0802, time for 50 iter: 158.4298
Iter 500/651 training loss = 0.170, time = 0.0714, time for 50 iter: 157.2625
Iter 550/651 training loss = 0.164, time = 0.0784, time for 50 iter: 157.0725
Iter 600/651 training loss = 0.156, time = 0.0764, time for 50 iter: 158.9253
Iter 650/651 training loss = 0.164, time = 0.0992, time for 50 iter: 137.2699
------------------------------------
epoch 1 total training loss = 0.187
Total epoch time = 2061.11
------------------------------------
Init worker 0 with seed 968601891
Init worker 1 with seed 968601892
Init worker 2 with seed 968601893
Init worker 3 with seed 968601894
Init worker 4 with seed 968601895
Init worker 5 with seed 968601896
Iter 50 val loss = 0.180 , time = 0.06
------------------------------------
total val loss = 0.181
total traslation error: 16.394560181551164 cm
total rotation error: 1.391414075295743 °
------------------------------------
Model saved as ./checkpoints/kitti/odom/val_seq_00/models/checkpoint_r5.00_t0.50_e1_0.181.tar
This is 2-th epoch
Init worker 0 with seed 968601991
Init worker 1 with seed 968601992
Init worker 2 with seed 968601993
Init worker 3 with seed 968601994
Init worker 4 with seed 968601995
Init worker 5 with seed 968601996
Iter 50/651 training loss = 0.164, time = 0.0703, time for 50 iter: 182.6435
Iter 100/651 training loss = 0.154, time = 0.0757, time for 50 iter: 159.0791
Iter 150/651 training loss = 0.145, time = 0.0753, time for 50 iter: 156.9057
Iter 200/651 training loss = 0.149, time = 0.0722, time for 50 iter: 158.3375
Iter 250/651 training loss = 0.147, time = 0.0768, time for 50 iter: 154.1539
Iter 300/651 training loss = 0.144, time = 0.0756, time for 50 iter: 155.4711
Iter 350/651 training loss = 0.138, time = 0.0721, time for 50 iter: 154.0126
Iter 400/651 training loss = 0.141, time = 0.0696, time for 50 iter: 156.9396
Iter 450/651 training loss = 0.130, time = 0.0617, time for 50 iter: 156.4081
Iter 500/651 training loss = 0.134, time = 0.0787, time for 50 iter: 156.6252
Iter 550/651 training loss = 0.123, time = 0.0641, time for 50 iter: 155.7265
Iter 600/651 training loss = 0.126, time = 0.0700, time for 50 iter: 158.3003
Iter 650/651 training loss = 0.125, time = 0.0901, time for 50 iter: 136.9406
------------------------------------
epoch 2 total training loss = 0.140
Total epoch time = 2041.69
------------------------------------
Init worker 0 with seed 968601991
Init worker 1 with seed 968601992
Init worker 2 with seed 968601993
Init worker 3 with seed 968601994
Init worker 4 with seed 968601995
Init worker 5 with seed 968601996
Iter 50 val loss = 0.158 , time = 0.07
------------------------------------
total val loss = 0.157
total traslation error: 12.43331704205864 cm
total rotation error: 1.2662466175622862 °
------------------------------------
Model saved as ./checkpoints/kitti/odom/val_seq_00/models/checkpoint_r5.00_t0.50_e2_0.157.tar
This is 3-th epoch
Init worker 0 with seed 968602091
Init worker 1 with seed 968602092
Init worker 2 with seed 968602093
Init worker 3 with seed 968602094
Init worker 4 with seed 968602095
Init worker 5 with seed 968602096
Iter 50/651 training loss = 0.129, time = 0.0645, time for 50 iter: 181.8550
Iter 100/651 training loss = 0.124, time = 0.0693, time for 50 iter: 155.3118
Iter 150/651 training loss = 0.120, time = 0.0778, time for 50 iter: 157.2444
Iter 200/651 training loss = 0.116, time = 0.0640, time for 50 iter: 155.4236
Iter 250/651 training loss = 0.117, time = 0.0807, time for 50 iter: 159.9052
Iter 300/651 training loss = 0.116, time = 0.0899, time for 50 iter: 157.5048
Iter 350/651 training loss = 0.115, time = 0.0684, time for 50 iter: 157.1619
Iter 400/651 training loss = 0.111, time = 0.0795, time for 50 iter: 158.5380
Iter 450/651 training loss = 0.112, time = 0.0790, time for 50 iter: 157.7939
Iter 500/651 training loss = 0.109, time = 0.0754, time for 50 iter: 156.1770
Iter 550/651 training loss = 0.115, time = 0.0735, time for 50 iter: 154.9603
Iter 600/651 training loss = 0.112, time = 0.0790, time for 50 iter: 156.6746
Iter 650/651 training loss = 0.104, time = 0.1032, time for 50 iter: 135.4875
------------------------------------
epoch 3 total training loss = 0.115
Total epoch time = 2044.19
------------------------------------
Init worker 0 with seed 968602091
Init worker 1 with seed 968602092
Init worker 2 with seed 968602093
Init worker 3 with seed 968602094
Init worker 4 with seed 968602095
Init worker 5 with seed 968602096
Iter 50 val loss = 0.113 , time = 0.05
------------------------------------
total val loss = 0.115
total traslation error: 11.61447027962892 cm
total rotation error: 0.831139387052192 °
------------------------------------
Model saved as ./checkpoints/kitti/odom/val_seq_00/models/checkpoint_r5.00_t0.50_e3_0.115.tar
This is 4-th epoch
Init worker 0 with seed 968602191
Init worker 1 with seed 968602192
Init worker 2 with seed 968602193
Init worker 3 with seed 968602194
Init worker 4 with seed 968602195
Init worker 5 with seed 968602196
Iter 50/651 training loss = 0.108, time = 0.0704, time for 50 iter: 182.0418
Iter 100/651 training loss = 0.107, time = 0.0716, time for 50 iter: 154.1219
Iter 150/651 training loss = 0.106, time = 0.0701, time for 50 iter: 157.0738
Iter 200/651 training loss = 0.104, time = 0.0918, time for 50 iter: 156.2897
Iter 250/651 training loss = 0.102, time = 0.0731, time for 50 iter: 155.1306
Iter 300/651 training loss = 0.104, time = 0.0728, time for 50 iter: 157.7138
Iter 350/651 training loss = 0.098, time = 0.0790, time for 50 iter: 156.0219
Iter 400/651 training loss = 0.097, time = 0.0704, time for 50 iter: 157.3418
Iter 450/651 training loss = 0.098, time = 0.0811, time for 50 iter: 158.2240
Iter 500/651 training loss = 0.094, time = 0.0698, time for 50 iter: 153.9692
Iter 550/651 training loss = 0.096, time = 0.0719, time for 50 iter: 157.8142
Iter 600/651 training loss = 0.096, time = 0.0633, time for 50 iter: 155.0785
Iter 650/651 training loss = 0.098, time = 0.0998, time for 50 iter: 138.2774
------------------------------------
epoch 4 total training loss = 0.100
Total epoch time = 2039.22
------------------------------------
Init worker 0 with seed 968602191
Init worker 1 with seed 968602192
Init worker 2 with seed 968602193
Init worker 3 with seed 968602194
Init worker 4 with seed 968602195
Init worker 5 with seed 968602196
Iter 50 val loss = 0.134 , time = 0.05
------------------------------------
total val loss = 0.130
total traslation error: 12.657383619492023 cm
total rotation error: 0.9130034800590174 °
------------------------------------
This is 5-th epoch
Init worker 0 with seed 968602291
Init worker 1 with seed 968602292
Init worker 2 with seed 968602293
Init worker 3 with seed 968602294
Init worker 4 with seed 968602295
Init worker 5 with seed 968602296
Iter 50/651 training loss = 0.102, time = 0.0736, time for 50 iter: 182.5113
Iter 100/651 training loss = 0.091, time = 0.0788, time for 50 iter: 156.7559
Iter 150/651 training loss = 0.093, time = 0.0698, time for 50 iter: 157.4367
Iter 200/651 training loss = 0.093, time = 0.0651, time for 50 iter: 157.6360
Iter 250/651 training loss = 0.098, time = 0.0791, time for 50 iter: 158.0680
Iter 300/651 training loss = 0.096, time = 0.0744, time for 50 iter: 155.9608
Iter 350/651 training loss = 0.094, time = 0.0730, time for 50 iter: 154.8634
Iter 400/651 training loss = 0.097, time = 0.0753, time for 50 iter: 157.2517
Iter 450/651 training loss = 0.092, time = 0.0873, time for 50 iter: 156.7742
Iter 500/651 training loss = 0.090, time = 0.0756, time for 50 iter: 154.4779
Iter 550/651 training loss = 0.088, time = 0.0772, time for 50 iter: 156.5272
Iter 600/651 training loss = 0.085, time = 0.0679, time for 50 iter: 157.5687
Iter 650/651 training loss = 0.087, time = 0.0974, time for 50 iter: 137.2146
------------------------------------
epoch 5 total training loss = 0.093
Total epoch time = 2043.17
------------------------------------
Init worker 0 with seed 968602291
Init worker 1 with seed 968602292
Init worker 2 with seed 968602293
Init worker 3 with seed 968602294
Init worker 4 with seed 968602295
Init worker 5 with seed 968602296
Iter 50 val loss = 0.113 , time = 0.05
------------------------------------
total val loss = 0.112
total traslation error: 11.094853962131912 cm
total rotation error: 0.8165065170620749 °
------------------------------------
Model saved as ./checkpoints/kitti/odom/val_seq_00/models/checkpoint_r5.00_t0.50_e5_0.112.tar
This is 6-th epoch
Init worker 0 with seed 968602391
Init worker 1 with seed 968602392
Init worker 2 with seed 968602393
Init worker 3 with seed 968602394
Init worker 4 with seed 968602395
Init worker 5 with seed 968602396
Iter 50/651 training loss = 0.093, time = 0.0650, time for 50 iter: 182.7805
Iter 100/651 training loss = 0.091, time = 0.0642, time for 50 iter: 155.4410
Iter 150/651 training loss = 0.085, time = 0.0732, time for 50 iter: 156.7134
Iter 200/651 training loss = 0.090, time = 0.0661, time for 50 iter: 157.5453
Iter 250/651 training loss = 0.089, time = 0.0680, time for 50 iter: 157.6137
Iter 300/651 training loss = 0.086, time = 0.0743, time for 50 iter: 157.7967
Iter 350/651 training loss = 0.086, time = 0.0701, time for 50 iter: 157.0972
Iter 400/651 training loss = 0.081, time = 0.0688, time for 50 iter: 157.3120
Iter 450/651 training loss = 0.082, time = 0.0751, time for 50 iter: 156.6239
Iter 500/651 training loss = 0.087, time = 0.0702, time for 50 iter: 155.0438
Iter 550/651 training loss = 0.083, time = 0.0704, time for 50 iter: 157.1769
Iter 600/651 training loss = 0.082, time = 0.0748, time for 50 iter: 158.8847
Iter 650/651 training loss = 0.082, time = 0.0934, time for 50 iter: 136.2693
------------------------------------
epoch 6 total training loss = 0.086
Total epoch time = 2046.45
------------------------------------
Init worker 0 with seed 968602391
Init worker 1 with seed 968602392
Init worker 2 with seed 968602393
Init worker 3 with seed 968602394
Init worker 4 with seed 968602395
Init worker 5 with seed 968602396
Iter 50 val loss = 0.100 , time = 0.06
------------------------------------
total val loss = 0.102
total traslation error: 9.397754708672116 cm
total rotation error: 0.7484173964782056 °
------------------------------------
Model saved as ./checkpoints/kitti/odom/val_seq_00/models/checkpoint_r5.00_t0.50_e6_0.102.tar
This is 7-th epoch
Init worker 0 with seed 968602491
Init worker 1 with seed 968602492
Init worker 2 with seed 968602493
Init worker 3 with seed 968602494
Init worker 4 with seed 968602495
Init worker 5 with seed 968602496
Iter 50/651 training loss = 0.085, time = 0.0752, time for 50 iter: 184.1278
Iter 100/651 training loss = 0.084, time = 0.0667, time for 50 iter: 158.0808
Iter 150/651 training loss = 0.087, time = 0.0644, time for 50 iter: 157.1080
Iter 200/651 training loss = 0.081, time = 0.0707, time for 50 iter: 153.8705
Iter 250/651 training loss = 0.081, time = 0.0696, time for 50 iter: 155.2375
Iter 300/651 training loss = 0.080, time = 0.0700, time for 50 iter: 155.4881
Iter 350/651 training loss = 0.081, time = 0.0705, time for 50 iter: 157.4313
Iter 400/651 training loss = 0.080, time = 0.0721, time for 50 iter: 157.8142
Iter 450/651 training loss = 0.083, time = 0.0671, time for 50 iter: 156.9585
Iter 500/651 training loss = 0.078, time = 0.0832, time for 50 iter: 157.6023
Iter 550/651 training loss = 0.081, time = 0.0730, time for 50 iter: 156.9374
Iter 600/651 training loss = 0.079, time = 0.0676, time for 50 iter: 156.6520
Iter 650/651 training loss = 0.080, time = 0.0972, time for 50 iter: 136.7829
------------------------------------
epoch 7 total training loss = 0.082
Total epoch time = 2044.24
------------------------------------
Init worker 0 with seed 968602491
Init worker 1 with seed 968602492
Init worker 2 with seed 968602493
Init worker 3 with seed 968602494
Init worker 4 with seed 968602495
Init worker 5 with seed 968602496
Iter 50 val loss = 0.099 , time = 0.06
------------------------------------
total val loss = 0.098
total traslation error: 8.685474166416487 cm
total rotation error: 0.7859130368781914 °
------------------------------------
Model saved as ./checkpoints/kitti/odom/val_seq_00/models/checkpoint_r5.00_t0.50_e7_0.098.tar
This is 8-th epoch
Init worker 0 with seed 968602591
Init worker 1 with seed 968602592
Init worker 2 with seed 968602593
Init worker 3 with seed 968602594
Init worker 4 with seed 968602595
Init worker 5 with seed 968602596
Iter 50/651 training loss = 0.079, time = 0.0701, time for 50 iter: 182.5823
Iter 100/651 training loss = 0.074, time = 0.0653, time for 50 iter: 155.1308
Iter 150/651 training loss = 0.080, time = 0.0701, time for 50 iter: 157.5971
Iter 200/651 training loss = 0.075, time = 0.0794, time for 50 iter: 156.6913
Iter 250/651 training loss = 0.077, time = 0.0712, time for 50 iter: 156.8357
Iter 300/651 training loss = 0.077, time = 0.0743, time for 50 iter: 155.7414
Iter 350/651 training loss = 0.075, time = 0.0746, time for 50 iter: 156.7062
Iter 400/651 training loss = 0.073, time = 0.0788, time for 50 iter: 157.0637
Iter 450/651 training loss = 0.075, time = 0.0598, time for 50 iter: 156.0358
Iter 500/651 training loss = 0.075, time = 0.0737, time for 50 iter: 156.7313
Iter 550/651 training loss = 0.076, time = 0.0693, time for 50 iter: 157.6530
Iter 600/651 training loss = 0.077, time = 0.0714, time for 50 iter: 157.0919
Iter 650/651 training loss = 0.073, time = 0.1320, time for 50 iter: 134.9983
------------------------------------
epoch 8 total training loss = 0.076
Total epoch time = 2041.00
------------------------------------
Init worker 0 with seed 968602591
Init worker 1 with seed 968602592
Init worker 2 with seed 968602593
Init worker 3 with seed 968602594
Init worker 4 with seed 968602595
Init worker 5 with seed 968602596
Iter 50 val loss = 0.086 , time = 0.06
------------------------------------
total val loss = 0.087
total traslation error: 7.831870926867706 cm
total rotation error: 0.6775783565280143 °
------------------------------------
Model saved as ./checkpoints/kitti/odom/val_seq_00/models/checkpoint_r5.00_t0.50_e8_0.087.tar
This is 9-th epoch
Init worker 0 with seed 968602691
Init worker 1 with seed 968602692
Init worker 2 with seed 968602693
Init worker 3 with seed 968602694
Init worker 4 with seed 968602695
Init worker 5 with seed 968602696
Iter 50/651 training loss = 0.079, time = 0.0772, time for 50 iter: 183.9697
Iter 100/651 training loss = 0.076, time = 0.0752, time for 50 iter: 156.5958
Iter 150/651 training loss = 0.077, time = 0.0692, time for 50 iter: 156.0164
Iter 200/651 training loss = 0.070, time = 0.0659, time for 50 iter: 158.0195
Iter 250/651 training loss = 0.076, time = 0.0745, time for 50 iter: 157.7461
Iter 300/651 training loss = 0.074, time = 0.0703, time for 50 iter: 154.6737
Iter 350/651 training loss = 0.072, time = 0.0812, time for 50 iter: 157.5792
Iter 400/651 training loss = 0.072, time = 0.0739, time for 50 iter: 157.1050
Iter 450/651 training loss = 0.074, time = 0.0794, time for 50 iter: 155.9157
Iter 500/651 training loss = 0.074, time = 0.0698, time for 50 iter: 154.8241
Iter 550/651 training loss = 0.072, time = 0.0776, time for 50 iter: 156.8397
Iter 600/651 training loss = 0.070, time = 0.0694, time for 50 iter: 157.2722
Iter 650/651 training loss = 0.075, time = 0.0915, time for 50 iter: 136.1803
------------------------------------
epoch 9 total training loss = 0.074
Total epoch time = 2042.87
------------------------------------
Init worker 0 with seed 968602691
Init worker 1 with seed 968602692
Init worker 2 with seed 968602693
Init worker 3 with seed 968602694
Init worker 4 with seed 968602695
Init worker 5 with seed 968602696
Iter 50 val loss = 0.088 , time = 0.05
------------------------------------
total val loss = 0.089
total traslation error: 8.530963863262755 cm
total rotation error: 0.6613214207598618 °
------------------------------------
This is 10-th epoch
Init worker 0 with seed 968602791
Init worker 1 with seed 968602792
Init worker 2 with seed 968602793
Init worker 3 with seed 968602794
Init worker 4 with seed 968602795
Init worker 5 with seed 968602796
Iter 50/651 training loss = 0.072, time = 0.0682, time for 50 iter: 184.8163
Iter 100/651 training loss = 0.069, time = 0.0710, time for 50 iter: 157.9242
Iter 150/651 training loss = 0.074, time = 0.0701, time for 50 iter: 157.5522
Iter 200/651 training loss = 0.073, time = 0.0737, time for 50 iter: 158.4745
Iter 250/651 training loss = 0.071, time = 0.0734, time for 50 iter: 157.7772
Iter 300/651 training loss = 0.070, time = 0.0676, time for 50 iter: 156.8239
Iter 350/651 training loss = 0.069, time = 0.0746, time for 50 iter: 157.8414
Iter 400/651 training loss = 0.071, time = 0.0611, time for 50 iter: 155.4537
Iter 450/651 training loss = 0.070, time = 0.0672, time for 50 iter: 157.1612
Iter 500/651 training loss = 0.070, time = 0.0795, time for 50 iter: 158.4274
Iter 550/651 training loss = 0.068, time = 0.0777, time for 50 iter: 156.7720
Iter 600/651 training loss = 0.071, time = 0.0811, time for 50 iter: 155.5649
Iter 650/651 training loss = 0.070, time = 0.0930, time for 50 iter: 136.5492
------------------------------------
epoch 10 total training loss = 0.070
Total epoch time = 2051.28
------------------------------------
Init worker 0 with seed 968602791
Init worker 1 with seed 968602792
Init worker 2 with seed 968602793
Init worker 3 with seed 968602794
Init worker 4 with seed 968602795
Init worker 5 with seed 968602796
Iter 50 val loss = 0.099 , time = 0.06
------------------------------------
total val loss = 0.101
total traslation error: 11.304136873105369 cm
total rotation error: 0.7081005896169481 °
------------------------------------
full training time = 7.44 HR
INFO - LCCNet - Result: 7.831870926867706
INFO - LCCNet - Completed after 7:26:25Training took 37 minutes for first epoch with a batch size of 60. So for 120 epochs of the first iteration it should take 3 days. For the next four iterations the paper runs LCCNet for 50 epochs each. So it is 320 epochs in total for the five iterations which should take 8 days.
Batch size of 120 was not used as it requires more than 12 GB VRAM.
Logs
python train_with_sacred.py
WARNING - LCCNet - No observers have been added to this run
INFO - LCCNet - Running command 'main'
INFO - LCCNet - Started
Loss Function Choice: combined
Val Sequence: 00
Ground truth poses are not avaialble for sequence 11.
Ground truth poses are not avaialble for sequence 12.
Ground truth poses are not avaialble for sequence 13.
Ground truth poses are not avaialble for sequence 14.
Ground truth poses are not avaialble for sequence 15.
Ground truth poses are not avaialble for sequence 16.
Ground truth poses are not avaialble for sequence 17.
Ground truth poses are not avaialble for sequence 18.
Ground truth poses are not avaialble for sequence 19.
Ground truth poses are not avaialble for sequence 20.
Ground truth poses are not avaialble for sequence 21.
Ground truth poses are not avaialble for sequence 11.
Ground truth poses are not avaialble for sequence 12.
Ground truth poses are not avaialble for sequence 13.
Ground truth poses are not avaialble for sequence 14.
Ground truth poses are not avaialble for sequence 15.
Ground truth poses are not avaialble for sequence 16.
Ground truth poses are not avaialble for sequence 17.
Ground truth poses are not avaialble for sequence 18.
Ground truth poses are not avaialble for sequence 19.
Ground truth poses are not avaialble for sequence 20.
Ground truth poses are not avaialble for sequence 21.
VAL SET: Using this file: /home/jay/datasets/KITTI/odometry_color/sequences/val_RT_left_seq00_20.00_1.50.csv
Number of the train dataset: 39011
Number of the val dataset: 4541
651
76
Number of model parameters: 69987887
This is -1-th epoch
Init worker 0 with seed 746881877
Init worker 1 with seed 746881878
Init worker 2 with seed 746881879
Init worker 3 with seed 746881880
Init worker 4 with seed 746881881
Init worker 5 with seed 746881882
/home/jay/mambaforge/envs/lcc/lib/python3.6/site-packages/torch/nn/functional.py:2423: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
"See the documentation of nn.Upsample for details.".format(mode))
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument
Iter 50/651 training loss = 3.922, time = 0.0649, time for 50 iter: 191.9447
Iter 100/651 training loss = 2.347, time = 0.0892, time for 50 iter: 165.8744
Iter 150/651 training loss = 2.043, time = 0.0810, time for 50 iter: 165.8079
Iter 200/651 training loss = 1.863, time = 0.0919, time for 50 iter: 169.8028
Iter 250/651 training loss = 1.690, time = 0.0942, time for 50 iter: 170.6948
Iter 300/651 training loss = 1.602, time = 0.0788, time for 50 iter: 168.4231
Iter 350/651 training loss = 1.447, time = 0.0775, time for 50 iter: 167.1476
Iter 400/651 training loss = 1.401, time = 0.0878, time for 50 iter: 170.6031
Iter 450/651 training loss = 1.322, time = 0.0905, time for 50 iter: 169.9958
Iter 500/651 training loss = 1.287, time = 0.0820, time for 50 iter: 170.5932
Iter 550/651 training loss = 1.221, time = 0.0705, time for 50 iter: 168.3159
Iter 600/651 training loss = 1.189, time = 0.0765, time for 50 iter: 169.4320
Iter 650/651 training loss = 1.183, time = 0.0991, time for 50 iter: 147.9759
------------------------------------
epoch -1 total training loss = 1.730
Total epoch time = 2196.74
------------------------------------
``
</details>