Thank you very much for providing this project. I fine-tuned the pre-trained model dino_deitsmall8_pretrain.pth on my own dataset (approximately 70,000 samples), but after fine-tuning, the performance is worse than the pre-trained model. Both the visualized attention maps and similarity calculations are inferior to the original dino_deitsmall8 model. This issue persists whether I fine-tuned for 3 epochs or 300 epochs. Interestingly, fine-tuning for 3 epochs yields better results than 300 epochs, which I suspect is due to severe overfitting with longer training. How can I improve the fine-tuning results to surpass the pre-trained model? Below are my training parameters and logs.
arch: vit_small
batch_size_per_gpu: 20
clip_grad: 3.0
data_path: /data01/wujiacheng/data/mini-imagenet100/train
dist_url: env://
drop_path_rate: 0.1
epochs: 300
freeze_last_layer: 1
global_crops_scale: (0.4, 1.0)
gpu: 0
local_crops_number: 8
local_crops_scale: (0.05, 0.4)
local_rank: 0
lr: 0.0005
min_lr: 1e-06
momentum_teacher: 0.996
norm_last_layer: True
num_workers: 10
optimizer: adamw
out_dim: 65536
output_dir: ./output
patch_size: 8
rank: 0
saveckp_freq: 20
seed: 0
teacher_temp: 0.04
use_bn_in_head: False
use_fp16: True
warmup_epochs: 10
warmup_teacher_temp: 0.04
warmup_teacher_temp_epochs: 0
weight_decay: 0.04
weight_decay_end: 0.4
world_size: 8
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Data loaded: there are 72448 images.
Student and Teacher are built: they are both vit_small network.
Loss, optimizer and schedulers ready.
Found checkpoint at ./output/checkpoint.pth
=> loaded 'student' from checkpoint './output/checkpoint.pth' with msg
=> loaded 'teacher' from checkpoint './output/checkpoint.pth' with msg
=> key 'optimizer' not found in checkpoint: './output/checkpoint.pth'
=> key 'fp16_scaler' not found in checkpoint: './output/checkpoint.pth'
=> key 'dino_loss' not found in checkpoint: './output/checkpoint.pth'
Starting DINO training !
{"train_loss": 7.3975777105156295, "train_lr": 1.5593881389687975e-05, "train_wd": 0.040003278940542016, "epoch": 0}
{"train_loss": 5.733514768491804, "train_lr": 4.68507966364243e-05, "train_wd": 0.04002299577417647, "epoch": 1}
{"train_loss": 5.2813909558200205, "train_lr": 7.810771188315966e-05, "train_wd": 0.04006244911448072, "epoch": 2}
{"train_loss": 4.931780147671172, "train_lr": 0.000109364627129896, "train_wd": 0.04012163463495158, "epoch": 3}
{"train_loss": 4.612698832708123, "train_lr": 0.00014062154237663252, "train_wd": 0.04020054584522951, "epoch": 4}
{"train_loss": 4.365842029254521, "train_lr": 0.00017187845762336835, "train_wd": 0.04029917409181127, "epoch": 5}
{"train_loss": 4.147426321377269, "train_lr": 0.00020313537287010427, "train_wd": 0.040417508558998075, "epoch": 6}
{"train_loss": 3.9338953712608964, "train_lr": 0.00023439228811683975, "train_wd": 0.040555536270082154, "epoch": 7}
{"train_loss": 3.751991960459051, "train_lr": 0.00026564920336357545, "train_wd": 0.040713242088769516, "epoch": 8}
{"train_loss": 3.5828048398273182, "train_lr": 0.0002969061186103117, "train_wd": 0.04089060872083998, "epoch": 9}
{"train_loss": 3.4191214270823824, "train_lr": 0.0003124969637646053, "train_wd": 0.04108761671604358, "epoch": 10}
{"train_loss": 3.2559533344367435, "train_lr": 0.0003124787063868354, "train_wd": 0.04130424447023352, "epoch": 11}
{"train_loss": 3.102740913596565, "train_lr": 0.0003124421735549047, "train_wd": 0.04154046822773548, "epoch": 12}
{"train_loss": 2.9655698861234483, "train_lr": 0.0003123873695561043, "train_wd": 0.041796262083952464, "epoch": 13}
{"train_loss": 2.8421991929675627, "train_lr": 0.000312314300821925, "train_wd": 0.04207159798820551, "epoch": 14}
{"train_loss": 2.737220007464685, "train_lr": 0.0003122229759273076, "train_wd": 0.04236644574680994, "epoch": 15}
{"train_loss": 2.6325242110915417, "train_lr": 0.00031211340558963266, "train_wd": 0.04268077302638645, "epoch": 16}
{"train_loss": 2.543589166044134, "train_lr": 0.0003119856026674665, "train_wd": 0.04301454535740672, "epoch": 17}
{"train_loss": 2.453971084536968, "train_lr": 0.00031183958215904727, "train_wd": 0.043367726137973274, "epoch": 18}
{"train_loss": 2.389527200201971, "train_lr": 0.00031167536120052953, "train_wd": 0.043740276637833655, "epoch": 19}
{"train_loss": 2.319972539567842, "train_lr": 0.0003114929590639708, "train_wd": 0.04413215600262723, "epoch": 20}
{"train_loss": 2.270206752373318, "train_lr": 0.0003112923971550707, "train_wd": 0.044543321258365624, "epoch": 21}
{"train_loss": 2.2110306938071695, "train_lr": 0.000311073699010659, "train_wd": 0.04497372731614514, "epoch": 22}
{"train_loss": 2.167464166599433, "train_lr": 0.000310836890295931, "train_wd": 0.04542332697709117, "epoch": 23}
{"train_loss": 2.1112963704771435, "train_lr": 0.0003105819988014412, "train_wd": 0.045892070937534446, "epoch": 24}
{"train_loss": 2.0732129593833095, "train_lr": 0.0003103090544398354, "train_wd": 0.04637990779441747, "epoch": 25}
{"train_loss": 2.0303954672793636, "train_lr": 0.00031001808924234546, "train_wd": 0.04688678405093154, "epoch": 26}
{"train_loss": 1.991264530828437, "train_lr": 0.00030970913735502576, "train_wd": 0.04741264412238291, "epoch": 27}
{"train_loss": 1.955514863116947, "train_lr": 0.0003093822350347523, "train_wd": 0.047957430342289026, "epoch": 28}
{"train_loss": 1.9239578501082364, "train_lr": 0.0003090374206449606, "train_wd": 0.048521082968701675, "epoch": 29}
{"train_loss": 1.8921751010569587, "train_lr": 0.0003086747346511482, "train_wd": 0.049103540190758584, "epoch": 30}
{"train_loss": 1.8520111252380684, "train_lr": 0.00030829421961612414, "train_wd": 0.049704738135461564, "epoch": 31}
{"train_loss": 1.827290213477295, "train_lr": 0.00030789592019501565, "train_wd": 0.05032461087468126, "epoch": 32}
{"train_loss": 1.7935898394645315, "train_lr": 0.0003074798831300259, "train_wd": 0.05096309043238634, "epoch": 33}
{"train_loss": 1.76907227727362, "train_lr": 0.00030704615724494824, "train_wd": 0.05162010679209833, "epoch": 34}
{"train_loss": 1.7515027310072848, "train_lr": 0.0003065947934394387, "train_wd": 0.052295587904569535, "epoch": 35}
{"train_loss": 1.725606787768481, "train_lr": 0.00030612584468304115, "train_wd": 0.05298945969568383, "epoch": 36}
{"train_loss": 1.7048891793310115, "train_lr": 0.0003056393660089718, "train_wd": 0.0537016460745804, "epoch": 37}
{"train_loss": 1.6839283264810796, "train_lr": 0.00030513541450765866, "train_wd": 0.05443206894199684, "epoch": 38}
{"train_loss": 1.6673894567761274, "train_lr": 0.0003046140493200464, "train_wd": 0.05518064819883493, "epoch": 39}
{"train_loss": 1.6475110412204423, "train_lr": 0.00030407533163065025, "train_wd": 0.05594730175494325, "epoch": 40}
{"train_loss": 1.6299118762588607, "train_lr": 0.00030351932466038016, "train_wd": 0.056731945538120054, "epoch": 41}
{"train_loss": 1.6132429480651576, "train_lr": 0.0003029460936591196, "train_wd": 0.057534493503332285, "epoch": 42}
{"train_loss": 1.5983621466707076, "train_lr": 0.0003023557058980677, "train_wd": 0.0583548576421517, "epoch": 43}
{"train_loss": 1.5876780218255204, "train_lr": 0.00030174823066184505, "train_wd": 0.05919294799240588, "epoch": 44}
{"train_loss": 1.5697981989324357, "train_lr": 0.0003011237392403656, "train_wd": 0.060048672648043463, "epoch": 45}
{"train_loss": 1.5534752824599236, "train_lr": 0.00030048230492046553, "train_wd": 0.060921937769212856, "epoch": 46}
{"train_loss": 1.5501195708767765, "train_lr": 0.0002998240029773068, "train_wd": 0.06181264759255256, "epoch": 47}
{"train_loss": 1.5323287861471155, "train_lr": 0.00029914891066554086, "train_wd": 0.06272070444169328, "epoch": 48}
{"train_loss": 1.5253395079181784, "train_lr": 0.00029845710721024385, "train_wd": 0.0636460087379685, "epoch": 49}
{"train_loss": 1.516570071788686, "train_lr": 0.0002977486737976179, "train_wd": 0.06458845901133499, "epoch": 50}
{"train_loss": 1.506209128495075, "train_lr": 0.00029702369356546644, "train_wd": 0.06554795191149967, "epoch": 51}
{"train_loss": 1.4922195377108534, "train_lr": 0.0002962822515934326, "train_wd": 0.06652438221925358, "epoch": 52}
{"train_loss": 1.48735602203328, "train_lr": 0.00029552443489302044, "train_wd": 0.06751764285801042, "epoch": 53}
{"train_loss": 1.4792325724476734, "train_lr": 0.0002947503323973786, "train_wd": 0.0685276249055477, "epoch": 54}
{"train_loss": 1.462175175745403, "train_lr": 0.0002939600349508677, "train_wd": 0.06955421760595262, "epoch": 55}
{"train_loss": 1.4598413309065905, "train_lr": 0.00029315363529839743, "train_wd": 0.07059730838176684, "epoch": 56}
{"train_loss": 1.455925998368622, "train_lr": 0.000292331228074542, "train_wd": 0.07165678284633235, "epoch": 57}
{"train_loss": 1.4439415440655652, "train_lr": 0.00029149290979243493, "train_wd": 0.07273252481633499, "epoch": 58}
{"train_loss": 1.434533314977204, "train_lr": 0.000290638778832445, "train_wd": 0.07382441632454499, "epoch": 59}
{"train_loss": 1.4319354517929321, "train_lr": 0.0002897689354306258, "train_wd": 0.0749323376327542, "epoch": 60}
{"train_loss": 1.4208135404418, "train_lr": 0.0002888834816669586, "train_wd": 0.07605616724490595, "epoch": 61}
{"train_loss": 1.412368931749885, "train_lr": 0.00028798252145336724, "train_wd": 0.07719578192041875, "epoch": 62}
{"train_loss": 1.4076655350576064, "train_lr": 0.0002870661605215287, "train_wd": 0.07835105668770134, "epoch": 63}
{"train_loss": 1.3964597912159114, "train_lr": 0.00028613450641045953, "train_wd": 0.07952186485785653, "epoch": 64}
{"train_loss": 1.3945600225763246, "train_lr": 0.0002851876684539026, "train_wd": 0.08070807803857477, "epoch": 65}
{"train_loss": 1.3920739725462894, "train_lr": 0.0002842257577674886, "train_wd": 0.08190956614821343, "epoch": 66}
{"train_loss": 1.3859488471716115, "train_lr": 0.0002832488872357018, "train_wd": 0.08312619743006168, "epoch": 67}
{"train_loss": 1.3832804730194992, "train_lr": 0.00028225717149863057, "train_wd": 0.08435783846678938, "epoch": 68}
{"train_loss": 1.3744480382337902, "train_lr": 0.0002812507269385135, "train_wd": 0.08560435419507723, "epoch": 69}
{"train_loss": 1.37167049871873, "train_lr": 0.0002802296716660824, "train_wd": 0.08686560792042894, "epoch": 70}
{"train_loss": 1.3687594512526968, "train_lr": 0.0002791941255066998, "train_wd": 0.08814146133216055, "epoch": 71}
{"train_loss": 1.3585772369617382, "train_lr": 0.00027814420998629975, "train_wd": 0.08943177451856754, "epoch": 72}
{"train_loss": 1.3593620024786321, "train_lr": 0.00027708004831712355, "train_wd": 0.0907364059822688, "epoch": 73}
{"train_loss": 1.3534520602562523, "train_lr": 0.00027600176538326106, "train_wd": 0.09205521265572247, "epoch": 74}
{"train_loss": 1.3473127303496661, "train_lr": 0.0002749094877259954, "train_wd": 0.09338804991691528, "epoch": 75}
{"train_loss": 1.3404909679830996, "train_lr": 0.00027380334352895234, "train_wd": 0.0947347716052214, "epoch": 76}
{"train_loss": 1.3364480797745353, "train_lr": 0.0002726834626030585, "train_wd": 0.09609523003743195, "epoch": 77}
{"train_loss": 1.3373393587421922, "train_lr": 0.0002715499763713055, "train_wd": 0.09746927602394875, "epoch": 78}
{"train_loss": 1.330167546637554, "train_lr": 0.00027040301785332836, "train_wd": 0.09885675888514518, "epoch": 79}
{"train_loss": 1.3304106952639017, "train_lr": 0.000269242721649794, "train_wd": 0.10025752646789025, "epoch": 80}
{"train_loss": 1.324928660160013, "train_lr": 0.00026806922392660714, "train_wd": 0.1016714251622331, "epoch": 81}
{"train_loss": 1.3187058581022826, "train_lr": 0.00026688266239892756, "train_wd": 0.10309829991824865, "epoch": 82}
{"train_loss": 1.3178141796631755, "train_lr": 0.0002656831763150128, "train_wd": 0.10453799426304057, "epoch": 83}
{"train_loss": 1.3142834861836998, "train_lr": 0.00026447090643987216, "train_wd": 0.10599035031790024, "epoch": 84}
{"train_loss": 1.3069883510387972, "train_lr": 0.00026324599503875094, "train_wd": 0.10745520881561951, "epoch": 85}
{"train_loss": 1.3104732318130214, "train_lr": 0.0002620085858604326, "train_wd": 0.10893240911795657, "epoch": 86}
{"train_loss": 1.3061490738517152, "train_lr": 0.0002607588241203705, "train_wd": 0.11042178923325201, "epoch": 87}
{"train_loss": 1.3041534660614829, "train_lr": 0.00025949685648364666, "train_wd": 0.11192318583419206, "epoch": 88}
{"train_loss": 1.3003195396632747, "train_lr": 0.0002582228310477579, "train_wd": 0.11343643427572052, "epoch": 89}
{"train_loss": 1.2948374743532158, "train_lr": 0.0002569368973252383, "train_wd": 0.1149613686130927, "epoch": 90}
{"train_loss": 1.2913228781086155, "train_lr": 0.00025563920622611143, "train_wd": 0.1164978216200744, "epoch": 91}
{"train_loss": 1.2854255201641174, "train_lr": 0.00025432991004018164, "train_wd": 0.11804562480727886, "epoch": 92}
{"train_loss": 1.2903377073858164, "train_lr": 0.00025300916241916146, "train_wd": 0.11960460844064469, "epoch": 93}
{"train_loss": 1.2885303561765271, "train_lr": 0.00025167711835864, "train_wd": 0.12117460156004825, "epoch": 94}
{"train_loss": 1.280728243736435, "train_lr": 0.0002503339341798936, "train_wd": 0.12275543199805172, "epoch": 95}
{"train_loss": 1.278711299519096, "train_lr": 0.0002489797675115412, "train_wd": 0.12434692639878328, "epoch": 96}
{"train_loss": 1.2754140556913562, "train_lr": 0.0002476147772710459, "train_wd": 0.12594891023694751, "epoch": 97}
{"train_loss": 1.2739259430940832, "train_lr": 0.00024623912364606353, "train_wd": 0.1275612078369634, "epoch": 98}
{"train_loss": 1.274091598437687, "train_lr": 0.0002448529680756468, "train_wd": 0.12918364239223065, "epoch": 99}
{"train_loss": 1.2714416869808878, "train_lr": 0.00024345647323129792, "train_wd": 0.1308160359845169, "epoch": 100}
{"train_loss": 1.2743088392359494, "train_lr": 0.00024204980299787825, "train_wd": 0.13245820960346946, "epoch": 101}
{"train_loss": 1.265813046703692, "train_lr": 0.00024063312245437699, "train_wd": 0.13410998316624534, "epoch": 102}
{"train_loss": 1.2606743107162481, "train_lr": 0.00023920659785453676, "train_wd": 0.13577117553725934, "epoch": 103}
{"train_loss": 1.2638890175396098, "train_lr": 0.00023777039660734473, "train_wd": 0.13744160454804835, "epoch": 104}
{"train_loss": 1.261428320722704, "train_lr": 0.0002363246872573844, "train_wd": 0.13912108701724746, "epoch": 105}
{"train_loss": 1.2582485414959028, "train_lr": 0.0002348696394650582, "train_wd": 0.14080943877067723, "epoch": 106}
{"train_loss": 1.2537645654159442, "train_lr": 0.0002334054239866761, "train_wd": 0.14250647466154262, "epoch": 107}
{"train_loss": 1.2553655015535454, "train_lr": 0.00023193221265441623, "train_wd": 0.14421200859073383, "epoch": 108}
{"train_loss": 1.2516674658168205, "train_lr": 0.00023045017835616045, "train_wd": 0.14592585352723625, "epoch": 109}
{"train_loss": 1.2444890861078042, "train_lr": 0.00022895949501520492, "train_wd": 0.14764782152863917, "epoch": 110}
{"train_loss": 1.2527960584276414, "train_lr": 0.00022746033756984853, "train_wd": 0.1493777237617461, "epoch": 111}
{"train_loss": 1.2495508431827864, "train_lr": 0.00022595288195286493, "train_wd": 0.15111537052328367, "epoch": 112}
{"train_loss": 1.2447310118251933, "train_lr": 0.00022443730507085396, "train_wd": 0.1528605712607023, "epoch": 113}
{"train_loss": 1.2446837277126919, "train_lr": 0.0002229137847834831, "train_wd": 0.15461313459307524, "epoch": 114}
{"train_loss": 1.2448710897212139, "train_lr": 0.00022138249988261222, "train_wd": 0.1563728683320825, "epoch": 115}
{"train_loss": 1.2402263224553483, "train_lr": 0.00021984363007131492, "train_wd": 0.15813957950309007, "epoch": 116}
{"train_loss": 1.2423123800714986, "train_lr": 0.00021829735594278523, "train_wd": 0.1599130743663086, "epoch": 117}
{"train_loss": 1.2355392041360647, "train_lr": 0.00021674385895914847, "train_wd": 0.161693158438041, "epoch": 118}
{"train_loss": 1.234681655380842, "train_lr": 0.0002151833214301642, "train_wd": 0.16347963651200906, "epoch": 119}
{"train_loss": 1.237066503746056, "train_lr": 0.0002136159264918294, "train_wd": 0.16527231268076012, "epoch": 120}
{"train_loss": 1.2333445499148386, "train_lr": 0.00021204185808489087, "train_wd": 0.1670709903571507, "epoch": 121}
{"train_loss": 1.2259560344180307, "train_lr": 0.00021046130093325377, "train_wd": 0.16887547229590466, "epoch": 122}
{"train_loss": 1.2298583858440408, "train_lr": 0.000208874440522309, "train_wd": 0.17068556061524207, "epoch": 123}
{"train_loss": 1.2289611470547661, "train_lr": 0.00020728146307716081, "train_wd": 0.17250105681858127, "epoch": 124}
{"train_loss": 1.22763606739453, "train_lr": 0.00020568255554077546, "train_wd": 0.17432176181630496, "epoch": 125}
{"train_loss": 1.230241806180406, "train_lr": 0.00020407790555204087, "train_wd": 0.17614747594759253, "epoch": 126}
{"train_loss": 1.219652475368858, "train_lr": 0.00020246770142374928, "train_wd": 0.17797799900231495, "epoch": 127}
{"train_loss": 1.2224865589719431, "train_lr": 0.00020085213212049332, "train_wd": 0.17981313024299198, "epoch": 128}
{"train_loss": 1.2223800104935612, "train_lr": 0.00019923138723649427, "train_wd": 0.1816526684268027, "epoch": 129}
{"train_loss": 1.220009788648283, "train_lr": 0.0001976056569733503, "train_wd": 0.18349641182765564, "epoch": 130}
{"train_loss": 1.2187102031605566, "train_lr": 0.0001959751321177154, "train_wd": 0.18534415825830974, "epoch": 131}
{"train_loss": 1.2208756563558647, "train_lr": 0.00019434000401891157, "train_wd": 0.18719570509254668, "epoch": 132}
{"train_loss": 1.2170751430753586, "train_lr": 0.00019270046456647006, "train_wd": 0.18905084928739088, "epoch": 133}
{"train_loss": 1.2127445554608, "train_lr": 0.00019105670616761567, "train_wd": 0.1909093874053759, "epoch": 134}
{"train_loss": 1.2120006283316596, "train_lr": 0.00018940892172468463, "train_wd": 0.1927711156368536, "epoch": 135}
{"train_loss": 1.2069310646883287, "train_lr": 0.00018775730461248642, "train_wd": 0.1946358298223433, "epoch": 136}
{"train_loss": 1.206128834070951, "train_lr": 0.0001861020486556131, "train_wd": 0.196503325474921, "epoch": 137}
{"train_loss": 1.2064570540446886, "train_lr": 0.00018444334810568909, "train_wd": 0.19837339780264407, "epoch": 138}
{"train_loss": 1.210334176716881, "train_lr": 0.00018278139761857936, "train_wd": 0.2002458417310076, "epoch": 139}
{"train_loss": 1.2002466429206253, "train_lr": 0.00018111639223154162, "train_wd": 0.20212045192543432, "epoch": 140}
{"train_loss": 1.205855048459386, "train_lr": 0.0001794485273403404, "train_wd": 0.20399702281379176, "epoch": 141}
{"train_loss": 1.201404860886061, "train_lr": 0.00017777799867631547, "train_wd": 0.20587534860893503, "epoch": 142}
{"train_loss": 1.1957137008420133, "train_lr": 0.00017610500228341263, "train_wd": 0.20775522333127328, "epoch": 143}
{"train_loss": 1.1990967025337493, "train_lr": 0.00017442973449517609, "train_wd": 0.20963644083135952, "epoch": 144}
{"train_loss": 1.200380598763581, "train_lr": 0.00017275239191170895, "train_wd": 0.21151879481249541, "epoch": 145}
{"train_loss": 1.1959535594683437, "train_lr": 0.00017107317137659948, "train_wd": 0.21340207885335447, "epoch": 146}
{"train_loss": 1.193580395270106, "train_lr": 0.0001693922699538236, "train_wd": 0.21528608643061925, "epoch": 147}
{"train_loss": 1.1916153922891326, "train_lr": 0.00016770988490461545, "train_wd": 0.21717061094162746, "epoch": 148}
{"train_loss": 1.188758715888304, "train_lr": 0.00016602621366432015, "train_wd": 0.2190554457270306, "epoch": 149}
{"train_loss": 1.1935037238754134, "train_lr": 0.00016434145381922282, "train_wd": 0.22094038409345393, "epoch": 150}
{"train_loss": 1.1858069163672955, "train_lr": 0.0001626558030833612, "train_wd": 0.2228252193361643, "epoch": 151}
{"train_loss": 1.188781982469084, "train_lr": 0.00016096945927532226, "train_wd": 0.22470974476173702, "epoch": 152}
{"train_loss": 1.186776782657984, "train_lr": 0.00015928262029502968, "train_wd": 0.22659375371072257, "epoch": 153}
{"train_loss": 1.1848745166233419, "train_lr": 0.00015759548410051614, "train_wd": 0.22847703958030907, "epoch": 154}
{"train_loss": 1.179297057656258, "train_lr": 0.00015590824868469462, "train_wd": 0.23035939584697765, "epoch": 155}
{"train_loss": 1.1830243970621872, "train_lr": 0.00015422111205212237, "train_wd": 0.23224061608915209, "epoch": 156}
{"train_loss": 1.1825099390049556, "train_lr": 0.00015253427219576334, "train_wd": 0.23412049400983312, "epoch": 157}
{"train_loss": 1.177629784557276, "train_lr": 0.0001508479270737535, "train_wd": 0.23599882345922252, "epoch": 158}
{"train_loss": 1.1774124071348162, "train_lr": 0.00014916227458617015, "train_wd": 0.23787539845732933, "epoch": 159}
{"train_loss": 1.1733541360899675, "train_lr": 0.00014747751255180628, "train_wd": 0.23975001321655767, "epoch": 160}
{"train_loss": 1.1724049934903078, "train_lr": 0.00014579383868495676, "train_wd": 0.24162246216427447, "epoch": 161}
{"train_loss": 1.1707856529680765, "train_lr": 0.0001441114505722144, "train_wd": 0.24349253996535158, "epoch": 162}
{"train_loss": 1.1721347715722117, "train_lr": 0.00014243054564928406, "train_wd": 0.24536004154468552, "epoch": 163}
{"train_loss": 1.173259326998573, "train_lr": 0.00014075132117781082, "train_wd": 0.24722476210968197, "epoch": 164}
{"train_loss": 1.167583369279066, "train_lr": 0.00013907397422223205, "train_wd": 0.24908649717271747, "epoch": 165}
{"train_loss": 1.1612190202669759, "train_lr": 0.00013739870162664985, "train_wd": 0.2509450425735634, "epoch": 166}
{"train_loss": 1.1647622720049942, "train_lr": 0.00013572569999173103, "train_wd": 0.25280019450177227, "epoch": 167}
{"train_loss": 1.159944876517474, "train_lr": 0.00013405516565163502, "train_wd": 0.2546517495190289, "epoch": 168}
{"train_loss": 1.1632022514807439, "train_lr": 0.00013238729465097374, "train_wd": 0.25649950458146153, "epoch": 169}
{"train_loss": 1.163216542089935, "train_lr": 0.00013072228272180373, "train_wd": 0.2583432570619047, "epoch": 170}
{"train_loss": 1.1554313779981658, "train_lr": 0.00012906032526065745, "train_wd": 0.26018280477212197, "epoch": 171}
{"train_loss": 1.1548928123346605, "train_lr": 0.00012740161730561144, "train_wd": 0.2620179459849766, "epoch": 172}
{"train_loss": 1.1518638640131176, "train_lr": 0.0001257463535133987, "train_wd": 0.2638484794565549, "epoch": 173}
{"train_loss": 1.1477938883121959, "train_lr": 0.00012409472813656444, "train_wd": 0.2656742044482334, "epoch": 174}
{"train_loss": 1.1508836093630674, "train_lr": 0.0001224469350006693, "train_wd": 0.26749492074869263, "epoch": 175}
{"train_loss": 1.1505381642239152, "train_lr": 0.00012080316748154406, "train_wd": 0.26931042869587224, "epoch": 176}
{"train_loss": 1.145343760762001, "train_lr": 0.00011916361848259463, "train_wd": 0.2711205291988676, "epoch": 177}
{"train_loss": 1.1440409388459098, "train_lr": 0.00011752848041216616, "train_wd": 0.27292502375976047, "epoch": 178}
{"train_loss": 1.138714359990791, "train_lr": 0.00011589794516096053, "train_wd": 0.274723714495387, "epoch": 179}
{"train_loss": 1.1420929789938759, "train_lr": 0.00011427220407951908, "train_wd": 0.2765164041590392, "epoch": 180}
{"train_loss": 1.141322568355672, "train_lr": 0.00011265144795576587, "train_wd": 0.27830289616209325, "epoch": 181}
{"train_loss": 1.13428166183948, "train_lr": 0.0001110358669926178, "train_wd": 0.28008299459556923, "epoch": 182}
{"train_loss": 1.1396118809378384, "train_lr": 0.0001094256507856644, "train_wd": 0.28185650425161385, "epoch": 183}
{"train_loss": 1.1334863024623651, "train_lr": 0.00010782098830091678, "train_wd": 0.2836232306449081, "epoch": 184}
{"train_loss": 1.131073048816318, "train_lr": 0.00010622206785263222, "train_wd": 0.28538298003399337, "epoch": 185}
{"train_loss": 1.1310193280203154, "train_lr": 0.00010462907708121456, "train_wd": 0.2871355594425188, "epoch": 186}
{"train_loss": 1.1265483911651952, "train_lr": 0.00010304220293119388, "train_wd": 0.28888077668040196, "epoch": 187}
{"train_loss": 1.1252250401962516, "train_lr": 0.00010146163162928732, "train_wd": 0.29061844036490675, "epoch": 188}
{"train_loss": 1.1231817850117793, "train_lr": 9.988754866254527e-05, "train_wd": 0.29234835994162767, "epoch": 189}
{"train_loss": 1.1264584160088438, "train_lr": 9.832013875658285e-05, "train_wd": 0.29407034570538937, "epoch": 190}
{"train_loss": 1.120118377153325, "train_lr": 9.67595858539022e-05, "train_wd": 0.29578420882104756, "epoch": 191}
{"train_loss": 1.1155596222003212, "train_lr": 9.52060730923054e-05, "train_wd": 0.29748976134419786, "epoch": 192}
{"train_loss": 1.11375545139169, "train_lr": 9.365978278340321e-05, "train_wd": 0.29918681624178667, "epoch": 193}
{"train_loss": 1.1138898587035657, "train_lr": 9.212089639121896e-05, "train_wd": 0.30087518741262026, "epoch": 194}
{"train_loss": 1.1139203502870767, "train_lr": 9.058959451089418e-05, "train_wd": 0.3025546897077736, "epoch": 195}
{"train_loss": 1.1094318839795558, "train_lr": 8.906605684749376e-05, "train_wd": 0.30422513895089354, "epoch": 196}
{"train_loss": 1.1112034465145089, "train_lr": 8.755046219491763e-05, "train_wd": 0.30588635195839653, "epoch": 197}
{"train_loss": 1.1071973581163757, "train_lr": 8.60429884149179e-05, "train_wd": 0.3075381465595561, "epoch": 198}
{"train_loss": 1.099203700405768, "train_lr": 8.454381241622658e-05, "train_wd": 0.30918034161648, "epoch": 199}
{"train_loss": 1.099097526492139, "train_lr": 8.305311013379425e-05, "train_wd": 0.31081275704397476, "epoch": 200}
{"train_loss": 1.1004753984115292, "train_lr": 8.157105650814283e-05, "train_wd": 0.3124352138292931, "epoch": 201}
{"train_loss": 1.0972965773673051, "train_lr": 8.009782546483681e-05, "train_wd": 0.31404753405176444, "epoch": 202}
{"train_loss": 1.092984740729485, "train_lr": 7.863358989407081e-05, "train_wd": 0.3156495409023077, "epoch": 203}
{"train_loss": 1.0960404245116937, "train_lr": 7.717852163038118e-05, "train_wd": 0.31724105870281877, "epoch": 204}
{"train_loss": 1.0928613061612817, "train_lr": 7.573279143248025e-05, "train_wd": 0.318821912925436, "epoch": 205}
{"train_loss": 1.0920073322094648, "train_lr": 7.429656896321662e-05, "train_wd": 0.32039193021167894, "epoch": 206}
{"train_loss": 1.0849385423239617, "train_lr": 7.287002276966499e-05, "train_wd": 0.32195093839145994, "epoch": 207}
{"train_loss": 1.086971781842938, "train_lr": 7.145332026334629e-05, "train_wd": 0.3234987665019642, "epoch": 208}
{"train_loss": 1.0813274293874218, "train_lr": 7.004662770058076e-05, "train_wd": 0.32503524480639606, "epoch": 209}
{"train_loss": 1.0791152904584873, "train_lr": 6.865011016297773e-05, "train_wd": 0.3265602048125955, "epoch": 210}
{"train_loss": 1.0761697434910895, "train_lr": 6.726393153806212e-05, "train_wd": 0.3280734792915127, "epoch": 211}
{"train_loss": 1.0794404465622738, "train_lr": 6.588825450004132e-05, "train_wd": 0.32957490229554814, "epoch": 212}
{"train_loss": 1.0754581980275368, "train_lr": 6.45232404907155e-05, "train_wd": 0.33106430917674984, "epoch": 213}
{"train_loss": 1.0727163913898763, "train_lr": 6.316904970053076e-05, "train_wd": 0.33254153660487, "epoch": 214}
{"train_loss": 1.0726363459453645, "train_lr": 6.182584104978022e-05, "train_wd": 0.33400642258527274, "epoch": 215}
{"train_loss": 1.0661814050765428, "train_lr": 6.049377216995485e-05, "train_wd": 0.33545880647670384, "epoch": 216}
{"train_loss": 1.0637150971400262, "train_lr": 5.9172999385243596e-05, "train_wd": 0.33689852900890205, "epoch": 217}
{"train_loss": 1.0641397414481746, "train_lr": 5.786367769418887e-05, "train_wd": 0.33832543230006773, "epoch": 218}
{"train_loss": 1.0615192341033073, "train_lr": 5.6565960751496254e-05, "train_wd": 0.33973935987417536, "epoch": 219}
{"train_loss": 1.056078793456267, "train_lr": 5.528000085000293e-05, "train_wd": 0.34114015667813274, "epoch": 220}
{"train_loss": 1.0578468480452368, "train_lr": 5.400594890280478e-05, "train_wd": 0.3425276690987848, "epoch": 221}
{"train_loss": 1.054848074393792, "train_lr": 5.274395442554689e-05, "train_wd": 0.3439017449797591, "epoch": 222}
{"train_loss": 1.0511530837934233, "train_lr": 5.14941655188763e-05, "train_wd": 0.3452622336381513, "epoch": 223}
{"train_loss": 1.051826697667088, "train_lr": 5.025672885106267e-05, "train_wd": 0.3466089858810468, "epoch": 224}
{"train_loss": 1.046208256253956, "train_lr": 4.9031789640785355e-05, "train_wd": 0.3479418540218876, "epoch": 225}
{"train_loss": 1.0447273803380344, "train_lr": 4.781949164009193e-05, "train_wd": 0.3492606918966609, "epoch": 226}
{"train_loss": 1.0428121057957147, "train_lr": 4.661997711752777e-05, "train_wd": 0.3505653548799322, "epoch": 227}
{"train_loss": 1.044226668519586, "train_lr": 4.543338684144047e-05, "train_wd": 0.3518556999007031, "epoch": 228}
{"train_loss": 1.0414691346128826, "train_lr": 4.425986006346e-05, "train_wd": 0.35313158545810114, "epoch": 229}
{"train_loss": 1.038192836032043, "train_lr": 4.309953450215688e-05, "train_wd": 0.35439287163689714, "epoch": 230}
{"train_loss": 1.037451881948298, "train_lr": 4.195254632688042e-05, "train_wd": 0.3556394201228481, "epoch": 231}
{"train_loss": 1.0366401289119922, "train_lr": 4.08190301417784e-05, "train_wd": 0.3568710942178651, "epoch": 232}
{"train_loss": 1.028051767729025, "train_lr": 3.969911897000097e-05, "train_wd": 0.35808775885500443, "epoch": 233}
{"train_loss": 1.029641574421749, "train_lr": 3.859294423808953e-05, "train_wd": 0.3592892806132772, "epoch": 234}
{"train_loss": 1.0282824208622023, "train_lr": 3.7500635760553364e-05, "train_wd": 0.36047552773228403, "epoch": 235}
{"train_loss": 1.0262311554588048, "train_lr": 3.6422321724635415e-05, "train_wd": 0.36164637012665946, "epoch": 236}
{"train_loss": 1.0233004989588748, "train_lr": 3.535812867526872e-05, "train_wd": 0.36280167940034086, "epoch": 237}
{"train_loss": 1.0209214830424933, "train_lr": 3.430818150022615e-05, "train_wd": 0.36394132886064734, "epoch": 238}
{"train_loss": 1.0191017907376574, "train_lr": 3.3272603415463845e-05, "train_wd": 0.36506519353217254, "epoch": 239}
{"train_loss": 1.0170858623707189, "train_lr": 3.225151595066155e-05, "train_wd": 0.36617315017049085, "epoch": 240}
{"train_loss": 1.0177517406131447, "train_lr": 3.124503893496051e-05, "train_wd": 0.36726507727567204, "epoch": 241}
{"train_loss": 1.0154096724844612, "train_lr": 3.0253290482901096e-05, "train_wd": 0.36834085510560294, "epoch": 242}
{"train_loss": 1.013073643710695, "train_lr": 2.927638698056129e-05, "train_wd": 0.3694003656891218, "epoch": 243}
{"train_loss": 1.0102413621549784, "train_lr": 2.8314443071898497e-05, "train_wd": 0.3704434928389536, "epoch": 244}
{"train_loss": 1.0077356305303036, "train_lr": 2.7367571645295374e-05, "train_wd": 0.3714701221644522, "epoch": 245}
{"train_loss": 1.0055394085429081, "train_lr": 2.6435883820312106e-05, "train_wd": 0.37248014108414146, "epoch": 246}
{"train_loss": 1.0048262930013634, "train_lr": 2.5519488934645823e-05, "train_wd": 0.37347343883806683, "epoch": 247}
{"train_loss": 1.0000261023962063, "train_lr": 2.461849453129943e-05, "train_wd": 0.37444990649993415, "epoch": 248}
{"train_loss": 1.0023978275403511, "train_lr": 2.3733006345961033e-05, "train_wd": 0.3754094369890625, "epoch": 249}
{"train_loss": 0.9999833804209082, "train_lr": 2.286312829459514e-05, "train_wd": 0.37635192508211884, "epoch": 250}
{"train_loss": 0.995666092780905, "train_lr": 2.2008962461248032e-05, "train_wd": 0.37727726742466483, "epoch": 251}
{"train_loss": 0.9959817434191308, "train_lr": 2.1170609086067297e-05, "train_wd": 0.37818536254248347, "epoch": 252}
{"train_loss": 0.9986522022502876, "train_lr": 2.0348166553538718e-05, "train_wd": 0.37907611085271203, "epoch": 253}
{"train_loss": 0.9922528021754614, "train_lr": 1.95417313809399e-05, "train_wd": 0.37994941467476084, "epoch": 254}
{"train_loss": 0.9929798559195041, "train_lr": 1.8751398207013937e-05, "train_wd": 0.3808051782410248, "epoch": 255}
{"train_loss": 0.9924680613789015, "train_lr": 1.7977259780862987e-05, "train_wd": 0.3816433077073856, "epoch": 256}
{"train_loss": 0.9913087562097451, "train_lr": 1.721940695106367e-05, "train_wd": 0.3824637111635003, "epoch": 257}
{"train_loss": 0.9861864087180094, "train_lr": 1.647792865500572e-05, "train_wd": 0.3832662986428876, "epoch": 258}
{"train_loss": 0.9858563996571224, "train_lr": 1.5752911908454728e-05, "train_wd": 0.38405098213278427, "epoch": 259}
{"train_loss": 0.9831302939312516, "train_lr": 1.5044441795340485e-05, "train_wd": 0.38481767558380386, "epoch": 260}
{"train_loss": 0.9852392861405305, "train_lr": 1.435260145777192e-05, "train_wd": 0.3855662949193685, "epoch": 261}
{"train_loss": 0.98016772293939, "train_lr": 1.3677472086280127e-05, "train_wd": 0.386296758044933, "epoch": 262}
{"train_loss": 0.9772953797107051, "train_lr": 1.3019132910290201e-05, "train_wd": 0.38700898485698393, "epoch": 263}
{"train_loss": 0.9801677357182895, "train_lr": 1.2377661188823446e-05, "train_wd": 0.38770289725182544, "epoch": 264}
{"train_loss": 0.978979548220976, "train_lr": 1.1753132201430425e-05, "train_wd": 0.38837841913414245, "epoch": 265}
{"train_loss": 0.9772988662980826, "train_lr": 1.1145619239356877e-05, "train_wd": 0.38903547642535014, "epoch": 266}
{"train_loss": 0.9759149223368252, "train_lr": 1.0555193596942484e-05, "train_wd": 0.3896739970717106, "epoch": 267}
{"train_loss": 0.9747701879566024, "train_lr": 9.981924563254158e-06, "train_wd": 0.3902939110522397, "epoch": 268}
{"train_loss": 0.9737108684581729, "train_lr": 9.425879413954783e-06, "train_wd": 0.3908951503863841, "epoch": 269}
{"train_loss": 0.9721088271950726, "train_lr": 8.887123403407999e-06, "train_wd": 0.3914776491414759, "epoch": 270}
{"train_loss": 0.9686770744217906, "train_lr": 8.365719757020349e-06, "train_wd": 0.39204134343996105, "epoch": 271}
{"train_loss": 0.9698984814288539, "train_lr": 7.861729663821536e-06, "train_wd": 0.3925861714664091, "epoch": 272}
{"train_loss": 0.9686554931145392, "train_lr": 7.375212269283579e-06, "train_wd": 0.3931120734742857, "epoch": 273}
{"train_loss": 0.9700315492474927, "train_lr": 6.906224668379831e-06, "train_wd": 0.3936189917925105, "epoch": 274}
{"train_loss": 0.9679153520506766, "train_lr": 6.454821898884645e-06, "train_wd": 0.3941068708317773, "epoch": 275}
{"train_loss": 0.9665576846141005, "train_lr": 6.021056934914423e-06, "train_wd": 0.3945756570906511, "epoch": 276}
{"train_loss": 0.9680534396361791, "train_lr": 5.604980680710925e-06, "train_wd": 0.39502529916143564, "epoch": 277}
{"train_loss": 0.9642110913991928, "train_lr": 5.206641964667367e-06, "train_wd": 0.39545574773581194, "epoch": 278}
{"train_loss": 0.9634479530056758, "train_lr": 4.826087533598204e-06, "train_wd": 0.3958669556102421, "epoch": 279}
{"train_loss": 0.9647843381596021, "train_lr": 4.4633620472532234e-06, "train_wd": 0.39625887769114887, "epoch": 280}
{"train_loss": 0.9623740180807633, "train_lr": 4.118508073076496e-06, "train_wd": 0.39663147099986024, "epoch": 281}
{"train_loss": 0.962872574336629, "train_lr": 3.79156608121089e-06, "train_wd": 0.39698469467732067, "epoch": 282}
{"train_loss": 0.962310821745149, "train_lr": 3.482574439748737e-06, "train_wd": 0.397318509988573, "epoch": 283}
{"train_loss": 0.9587763018774248, "train_lr": 3.191569410229164e-06, "train_wd": 0.39763288032700783, "epoch": 284}
{"train_loss": 0.9591802291044619, "train_lr": 2.9185851433826347e-06, "train_wd": 0.39792777121837364, "epoch": 285}
{"train_loss": 0.9604082268300111, "train_lr": 2.663653675123216e-06, "train_wd": 0.39820315032456216, "epoch": 286}
{"train_loss": 0.9587262105732431, "train_lr": 2.426804922789014e-06, "train_wd": 0.3984589874471502, "epoch": 287}
{"train_loss": 0.9592475521304042, "train_lr": 2.2080666816312486e-06, "train_wd": 0.3986952545307147, "epoch": 288}
{"train_loss": 0.9577035252004862, "train_lr": 2.0074646215523584e-06, "train_wd": 0.3989119256659073, "epoch": 289}
{"train_loss": 0.9586081852625429, "train_lr": 1.8250222840935152e-06, "train_wd": 0.3991089770922966, "epoch": 290}
{"train_loss": 0.9587571681641965, "train_lr": 1.6607610796719286e-06, "train_wd": 0.39928638720097287, "epoch": 291}
{"train_loss": 0.9573075459421142, "train_lr": 1.5147002850682333e-06, "train_wd": 0.39944413653691935, "epoch": 292}
{"train_loss": 0.9561012626864609, "train_lr": 1.3868570411642798e-06, "train_wd": 0.39958220780114223, "epoch": 293}
{"train_loss": 0.9595896629781454, "train_lr": 1.2772463509315658e-06, "train_wd": 0.3997005858525723, "epoch": 294}
{"train_loss": 0.9575634766740939, "train_lr": 1.18588107767058e-06, "train_wd": 0.3997992577097222, "epoch": 295}
{"train_loss": 0.9555465568689449, "train_lr": 1.1127719435012293e-06, "train_wd": 0.3998782125521103, "epoch": 296}
{"train_loss": 0.9585663597800035, "train_lr": 1.0579275281045622e-06, "train_wd": 0.3999374417214491, "epoch": 297}
{"train_loss": 0.957040233805413, "train_lr": 1.0213542677158922e-06, "train_wd": 0.3999769387225926, "epoch": 298}
{"train_loss": 0.9557520198652241, "train_lr": 1.0030564543694825e-06, "train_wd": 0.39999669922424963, "epoch": 299}
Thank you very much for providing this project. I fine-tuned the pre-trained model dino_deitsmall8_pretrain.pth on my own dataset (approximately 70,000 samples), but after fine-tuning, the performance is worse than the pre-trained model. Both the visualized attention maps and similarity calculations are inferior to the original dino_deitsmall8 model. This issue persists whether I fine-tuned for 3 epochs or 300 epochs. Interestingly, fine-tuning for 3 epochs yields better results than 300 epochs, which I suspect is due to severe overfitting with longer training. How can I improve the fine-tuning results to surpass the pre-trained model? Below are my training parameters and logs.
arch: vit_small
batch_size_per_gpu: 20
clip_grad: 3.0
data_path: /data01/wujiacheng/data/mini-imagenet100/train
dist_url: env://
drop_path_rate: 0.1
epochs: 300
freeze_last_layer: 1
global_crops_scale: (0.4, 1.0)
gpu: 0
local_crops_number: 8
local_crops_scale: (0.05, 0.4)
local_rank: 0
lr: 0.0005
min_lr: 1e-06
momentum_teacher: 0.996
norm_last_layer: True
num_workers: 10
optimizer: adamw
out_dim: 65536
output_dir: ./output
patch_size: 8
rank: 0
saveckp_freq: 20
seed: 0
teacher_temp: 0.04
use_bn_in_head: False
use_fp16: True
warmup_epochs: 10
warmup_teacher_temp: 0.04
warmup_teacher_temp_epochs: 0
weight_decay: 0.04
weight_decay_end: 0.4
world_size: 8
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Data loaded: there are 72448 images.
Student and Teacher are built: they are both vit_small network.
Loss, optimizer and schedulers ready.
Found checkpoint at ./output/checkpoint.pth
=> loaded 'student' from checkpoint './output/checkpoint.pth' with msg
=> loaded 'teacher' from checkpoint './output/checkpoint.pth' with msg
=> key 'optimizer' not found in checkpoint: './output/checkpoint.pth'
=> key 'fp16_scaler' not found in checkpoint: './output/checkpoint.pth'
=> key 'dino_loss' not found in checkpoint: './output/checkpoint.pth'
Starting DINO training !
{"train_loss": 5.733514768491804, "train_lr": 4.68507966364243e-05, "train_wd": 0.04002299577417647, "epoch": 1}
{"train_loss": 5.2813909558200205, "train_lr": 7.810771188315966e-05, "train_wd": 0.04006244911448072, "epoch": 2}
{"train_loss": 4.931780147671172, "train_lr": 0.000109364627129896, "train_wd": 0.04012163463495158, "epoch": 3}
{"train_loss": 4.612698832708123, "train_lr": 0.00014062154237663252, "train_wd": 0.04020054584522951, "epoch": 4}
{"train_loss": 4.365842029254521, "train_lr": 0.00017187845762336835, "train_wd": 0.04029917409181127, "epoch": 5}
{"train_loss": 4.147426321377269, "train_lr": 0.00020313537287010427, "train_wd": 0.040417508558998075, "epoch": 6}
{"train_loss": 3.9338953712608964, "train_lr": 0.00023439228811683975, "train_wd": 0.040555536270082154, "epoch": 7}
{"train_loss": 3.751991960459051, "train_lr": 0.00026564920336357545, "train_wd": 0.040713242088769516, "epoch": 8}
{"train_loss": 3.5828048398273182, "train_lr": 0.0002969061186103117, "train_wd": 0.04089060872083998, "epoch": 9}
{"train_loss": 3.4191214270823824, "train_lr": 0.0003124969637646053, "train_wd": 0.04108761671604358, "epoch": 10}
{"train_loss": 3.2559533344367435, "train_lr": 0.0003124787063868354, "train_wd": 0.04130424447023352, "epoch": 11}
{"train_loss": 3.102740913596565, "train_lr": 0.0003124421735549047, "train_wd": 0.04154046822773548, "epoch": 12}
{"train_loss": 2.9655698861234483, "train_lr": 0.0003123873695561043, "train_wd": 0.041796262083952464, "epoch": 13}
{"train_loss": 2.8421991929675627, "train_lr": 0.000312314300821925, "train_wd": 0.04207159798820551, "epoch": 14}
{"train_loss": 2.737220007464685, "train_lr": 0.0003122229759273076, "train_wd": 0.04236644574680994, "epoch": 15}
{"train_loss": 2.6325242110915417, "train_lr": 0.00031211340558963266, "train_wd": 0.04268077302638645, "epoch": 16}
{"train_loss": 2.543589166044134, "train_lr": 0.0003119856026674665, "train_wd": 0.04301454535740672, "epoch": 17}
{"train_loss": 2.453971084536968, "train_lr": 0.00031183958215904727, "train_wd": 0.043367726137973274, "epoch": 18}
{"train_loss": 2.389527200201971, "train_lr": 0.00031167536120052953, "train_wd": 0.043740276637833655, "epoch": 19}
{"train_loss": 2.319972539567842, "train_lr": 0.0003114929590639708, "train_wd": 0.04413215600262723, "epoch": 20}
{"train_loss": 2.270206752373318, "train_lr": 0.0003112923971550707, "train_wd": 0.044543321258365624, "epoch": 21}
{"train_loss": 2.2110306938071695, "train_lr": 0.000311073699010659, "train_wd": 0.04497372731614514, "epoch": 22}
{"train_loss": 2.167464166599433, "train_lr": 0.000310836890295931, "train_wd": 0.04542332697709117, "epoch": 23}
{"train_loss": 2.1112963704771435, "train_lr": 0.0003105819988014412, "train_wd": 0.045892070937534446, "epoch": 24}
{"train_loss": 2.0732129593833095, "train_lr": 0.0003103090544398354, "train_wd": 0.04637990779441747, "epoch": 25}
{"train_loss": 2.0303954672793636, "train_lr": 0.00031001808924234546, "train_wd": 0.04688678405093154, "epoch": 26}
{"train_loss": 1.991264530828437, "train_lr": 0.00030970913735502576, "train_wd": 0.04741264412238291, "epoch": 27}
{"train_loss": 1.955514863116947, "train_lr": 0.0003093822350347523, "train_wd": 0.047957430342289026, "epoch": 28}
{"train_loss": 1.9239578501082364, "train_lr": 0.0003090374206449606, "train_wd": 0.048521082968701675, "epoch": 29}
{"train_loss": 1.8921751010569587, "train_lr": 0.0003086747346511482, "train_wd": 0.049103540190758584, "epoch": 30}
{"train_loss": 1.8520111252380684, "train_lr": 0.00030829421961612414, "train_wd": 0.049704738135461564, "epoch": 31}
{"train_loss": 1.827290213477295, "train_lr": 0.00030789592019501565, "train_wd": 0.05032461087468126, "epoch": 32}
{"train_loss": 1.7935898394645315, "train_lr": 0.0003074798831300259, "train_wd": 0.05096309043238634, "epoch": 33}
{"train_loss": 1.76907227727362, "train_lr": 0.00030704615724494824, "train_wd": 0.05162010679209833, "epoch": 34}
{"train_loss": 1.7515027310072848, "train_lr": 0.0003065947934394387, "train_wd": 0.052295587904569535, "epoch": 35}
{"train_loss": 1.725606787768481, "train_lr": 0.00030612584468304115, "train_wd": 0.05298945969568383, "epoch": 36}
{"train_loss": 1.7048891793310115, "train_lr": 0.0003056393660089718, "train_wd": 0.0537016460745804, "epoch": 37}
{"train_loss": 1.6839283264810796, "train_lr": 0.00030513541450765866, "train_wd": 0.05443206894199684, "epoch": 38}
{"train_loss": 1.6673894567761274, "train_lr": 0.0003046140493200464, "train_wd": 0.05518064819883493, "epoch": 39}
{"train_loss": 1.6475110412204423, "train_lr": 0.00030407533163065025, "train_wd": 0.05594730175494325, "epoch": 40}
{"train_loss": 1.6299118762588607, "train_lr": 0.00030351932466038016, "train_wd": 0.056731945538120054, "epoch": 41}
{"train_loss": 1.6132429480651576, "train_lr": 0.0003029460936591196, "train_wd": 0.057534493503332285, "epoch": 42}
{"train_loss": 1.5983621466707076, "train_lr": 0.0003023557058980677, "train_wd": 0.0583548576421517, "epoch": 43}
{"train_loss": 1.5876780218255204, "train_lr": 0.00030174823066184505, "train_wd": 0.05919294799240588, "epoch": 44}
{"train_loss": 1.5697981989324357, "train_lr": 0.0003011237392403656, "train_wd": 0.060048672648043463, "epoch": 45}
{"train_loss": 1.5534752824599236, "train_lr": 0.00030048230492046553, "train_wd": 0.060921937769212856, "epoch": 46}
{"train_loss": 1.5501195708767765, "train_lr": 0.0002998240029773068, "train_wd": 0.06181264759255256, "epoch": 47}
{"train_loss": 1.5323287861471155, "train_lr": 0.00029914891066554086, "train_wd": 0.06272070444169328, "epoch": 48}
{"train_loss": 1.5253395079181784, "train_lr": 0.00029845710721024385, "train_wd": 0.0636460087379685, "epoch": 49}
{"train_loss": 1.516570071788686, "train_lr": 0.0002977486737976179, "train_wd": 0.06458845901133499, "epoch": 50}
{"train_loss": 1.506209128495075, "train_lr": 0.00029702369356546644, "train_wd": 0.06554795191149967, "epoch": 51}
{"train_loss": 1.4922195377108534, "train_lr": 0.0002962822515934326, "train_wd": 0.06652438221925358, "epoch": 52}
{"train_loss": 1.48735602203328, "train_lr": 0.00029552443489302044, "train_wd": 0.06751764285801042, "epoch": 53}
{"train_loss": 1.4792325724476734, "train_lr": 0.0002947503323973786, "train_wd": 0.0685276249055477, "epoch": 54}
{"train_loss": 1.462175175745403, "train_lr": 0.0002939600349508677, "train_wd": 0.06955421760595262, "epoch": 55}
{"train_loss": 1.4598413309065905, "train_lr": 0.00029315363529839743, "train_wd": 0.07059730838176684, "epoch": 56}
{"train_loss": 1.455925998368622, "train_lr": 0.000292331228074542, "train_wd": 0.07165678284633235, "epoch": 57}
{"train_loss": 1.4439415440655652, "train_lr": 0.00029149290979243493, "train_wd": 0.07273252481633499, "epoch": 58}
{"train_loss": 1.434533314977204, "train_lr": 0.000290638778832445, "train_wd": 0.07382441632454499, "epoch": 59}
{"train_loss": 1.4319354517929321, "train_lr": 0.0002897689354306258, "train_wd": 0.0749323376327542, "epoch": 60}
{"train_loss": 1.4208135404418, "train_lr": 0.0002888834816669586, "train_wd": 0.07605616724490595, "epoch": 61}
{"train_loss": 1.412368931749885, "train_lr": 0.00028798252145336724, "train_wd": 0.07719578192041875, "epoch": 62}
{"train_loss": 1.4076655350576064, "train_lr": 0.0002870661605215287, "train_wd": 0.07835105668770134, "epoch": 63}
{"train_loss": 1.3964597912159114, "train_lr": 0.00028613450641045953, "train_wd": 0.07952186485785653, "epoch": 64}
{"train_loss": 1.3945600225763246, "train_lr": 0.0002851876684539026, "train_wd": 0.08070807803857477, "epoch": 65}
{"train_loss": 1.3920739725462894, "train_lr": 0.0002842257577674886, "train_wd": 0.08190956614821343, "epoch": 66}
{"train_loss": 1.3859488471716115, "train_lr": 0.0002832488872357018, "train_wd": 0.08312619743006168, "epoch": 67}
{"train_loss": 1.3832804730194992, "train_lr": 0.00028225717149863057, "train_wd": 0.08435783846678938, "epoch": 68}
{"train_loss": 1.3744480382337902, "train_lr": 0.0002812507269385135, "train_wd": 0.08560435419507723, "epoch": 69}
{"train_loss": 1.37167049871873, "train_lr": 0.0002802296716660824, "train_wd": 0.08686560792042894, "epoch": 70}
{"train_loss": 1.3687594512526968, "train_lr": 0.0002791941255066998, "train_wd": 0.08814146133216055, "epoch": 71}
{"train_loss": 1.3585772369617382, "train_lr": 0.00027814420998629975, "train_wd": 0.08943177451856754, "epoch": 72}
{"train_loss": 1.3593620024786321, "train_lr": 0.00027708004831712355, "train_wd": 0.0907364059822688, "epoch": 73}
{"train_loss": 1.3534520602562523, "train_lr": 0.00027600176538326106, "train_wd": 0.09205521265572247, "epoch": 74}
{"train_loss": 1.3473127303496661, "train_lr": 0.0002749094877259954, "train_wd": 0.09338804991691528, "epoch": 75}
{"train_loss": 1.3404909679830996, "train_lr": 0.00027380334352895234, "train_wd": 0.0947347716052214, "epoch": 76}
{"train_loss": 1.3364480797745353, "train_lr": 0.0002726834626030585, "train_wd": 0.09609523003743195, "epoch": 77}
{"train_loss": 1.3373393587421922, "train_lr": 0.0002715499763713055, "train_wd": 0.09746927602394875, "epoch": 78}
{"train_loss": 1.330167546637554, "train_lr": 0.00027040301785332836, "train_wd": 0.09885675888514518, "epoch": 79}
{"train_loss": 1.3304106952639017, "train_lr": 0.000269242721649794, "train_wd": 0.10025752646789025, "epoch": 80}
{"train_loss": 1.324928660160013, "train_lr": 0.00026806922392660714, "train_wd": 0.1016714251622331, "epoch": 81}
{"train_loss": 1.3187058581022826, "train_lr": 0.00026688266239892756, "train_wd": 0.10309829991824865, "epoch": 82}
{"train_loss": 1.3178141796631755, "train_lr": 0.0002656831763150128, "train_wd": 0.10453799426304057, "epoch": 83}
{"train_loss": 1.3142834861836998, "train_lr": 0.00026447090643987216, "train_wd": 0.10599035031790024, "epoch": 84}
{"train_loss": 1.3069883510387972, "train_lr": 0.00026324599503875094, "train_wd": 0.10745520881561951, "epoch": 85}
{"train_loss": 1.3104732318130214, "train_lr": 0.0002620085858604326, "train_wd": 0.10893240911795657, "epoch": 86}
{"train_loss": 1.3061490738517152, "train_lr": 0.0002607588241203705, "train_wd": 0.11042178923325201, "epoch": 87}
{"train_loss": 1.3041534660614829, "train_lr": 0.00025949685648364666, "train_wd": 0.11192318583419206, "epoch": 88}
{"train_loss": 1.3003195396632747, "train_lr": 0.0002582228310477579, "train_wd": 0.11343643427572052, "epoch": 89}
{"train_loss": 1.2948374743532158, "train_lr": 0.0002569368973252383, "train_wd": 0.1149613686130927, "epoch": 90}
{"train_loss": 1.2913228781086155, "train_lr": 0.00025563920622611143, "train_wd": 0.1164978216200744, "epoch": 91}
{"train_loss": 1.2854255201641174, "train_lr": 0.00025432991004018164, "train_wd": 0.11804562480727886, "epoch": 92}
{"train_loss": 1.2903377073858164, "train_lr": 0.00025300916241916146, "train_wd": 0.11960460844064469, "epoch": 93}
{"train_loss": 1.2885303561765271, "train_lr": 0.00025167711835864, "train_wd": 0.12117460156004825, "epoch": 94}
{"train_loss": 1.280728243736435, "train_lr": 0.0002503339341798936, "train_wd": 0.12275543199805172, "epoch": 95}
{"train_loss": 1.278711299519096, "train_lr": 0.0002489797675115412, "train_wd": 0.12434692639878328, "epoch": 96}
{"train_loss": 1.2754140556913562, "train_lr": 0.0002476147772710459, "train_wd": 0.12594891023694751, "epoch": 97}
{"train_loss": 1.2739259430940832, "train_lr": 0.00024623912364606353, "train_wd": 0.1275612078369634, "epoch": 98}
{"train_loss": 1.274091598437687, "train_lr": 0.0002448529680756468, "train_wd": 0.12918364239223065, "epoch": 99}
{"train_loss": 1.2714416869808878, "train_lr": 0.00024345647323129792, "train_wd": 0.1308160359845169, "epoch": 100}
{"train_loss": 1.2743088392359494, "train_lr": 0.00024204980299787825, "train_wd": 0.13245820960346946, "epoch": 101}
{"train_loss": 1.265813046703692, "train_lr": 0.00024063312245437699, "train_wd": 0.13410998316624534, "epoch": 102}
{"train_loss": 1.2606743107162481, "train_lr": 0.00023920659785453676, "train_wd": 0.13577117553725934, "epoch": 103}
{"train_loss": 1.2638890175396098, "train_lr": 0.00023777039660734473, "train_wd": 0.13744160454804835, "epoch": 104}
{"train_loss": 1.261428320722704, "train_lr": 0.0002363246872573844, "train_wd": 0.13912108701724746, "epoch": 105}
{"train_loss": 1.2582485414959028, "train_lr": 0.0002348696394650582, "train_wd": 0.14080943877067723, "epoch": 106}
{"train_loss": 1.2537645654159442, "train_lr": 0.0002334054239866761, "train_wd": 0.14250647466154262, "epoch": 107}
{"train_loss": 1.2553655015535454, "train_lr": 0.00023193221265441623, "train_wd": 0.14421200859073383, "epoch": 108}
{"train_loss": 1.2516674658168205, "train_lr": 0.00023045017835616045, "train_wd": 0.14592585352723625, "epoch": 109}
{"train_loss": 1.2444890861078042, "train_lr": 0.00022895949501520492, "train_wd": 0.14764782152863917, "epoch": 110}
{"train_loss": 1.2527960584276414, "train_lr": 0.00022746033756984853, "train_wd": 0.1493777237617461, "epoch": 111}
{"train_loss": 1.2495508431827864, "train_lr": 0.00022595288195286493, "train_wd": 0.15111537052328367, "epoch": 112}
{"train_loss": 1.2447310118251933, "train_lr": 0.00022443730507085396, "train_wd": 0.1528605712607023, "epoch": 113}
{"train_loss": 1.2446837277126919, "train_lr": 0.0002229137847834831, "train_wd": 0.15461313459307524, "epoch": 114}
{"train_loss": 1.2448710897212139, "train_lr": 0.00022138249988261222, "train_wd": 0.1563728683320825, "epoch": 115}
{"train_loss": 1.2402263224553483, "train_lr": 0.00021984363007131492, "train_wd": 0.15813957950309007, "epoch": 116}
{"train_loss": 1.2423123800714986, "train_lr": 0.00021829735594278523, "train_wd": 0.1599130743663086, "epoch": 117}
{"train_loss": 1.2355392041360647, "train_lr": 0.00021674385895914847, "train_wd": 0.161693158438041, "epoch": 118}
{"train_loss": 1.234681655380842, "train_lr": 0.0002151833214301642, "train_wd": 0.16347963651200906, "epoch": 119}
{"train_loss": 1.237066503746056, "train_lr": 0.0002136159264918294, "train_wd": 0.16527231268076012, "epoch": 120}
{"train_loss": 1.2333445499148386, "train_lr": 0.00021204185808489087, "train_wd": 0.1670709903571507, "epoch": 121}
{"train_loss": 1.2259560344180307, "train_lr": 0.00021046130093325377, "train_wd": 0.16887547229590466, "epoch": 122}
{"train_loss": 1.2298583858440408, "train_lr": 0.000208874440522309, "train_wd": 0.17068556061524207, "epoch": 123}
{"train_loss": 1.2289611470547661, "train_lr": 0.00020728146307716081, "train_wd": 0.17250105681858127, "epoch": 124}
{"train_loss": 1.22763606739453, "train_lr": 0.00020568255554077546, "train_wd": 0.17432176181630496, "epoch": 125}
{"train_loss": 1.230241806180406, "train_lr": 0.00020407790555204087, "train_wd": 0.17614747594759253, "epoch": 126}
{"train_loss": 1.219652475368858, "train_lr": 0.00020246770142374928, "train_wd": 0.17797799900231495, "epoch": 127}
{"train_loss": 1.2224865589719431, "train_lr": 0.00020085213212049332, "train_wd": 0.17981313024299198, "epoch": 128}
{"train_loss": 1.2223800104935612, "train_lr": 0.00019923138723649427, "train_wd": 0.1816526684268027, "epoch": 129}
{"train_loss": 1.220009788648283, "train_lr": 0.0001976056569733503, "train_wd": 0.18349641182765564, "epoch": 130}
{"train_loss": 1.2187102031605566, "train_lr": 0.0001959751321177154, "train_wd": 0.18534415825830974, "epoch": 131}
{"train_loss": 1.2208756563558647, "train_lr": 0.00019434000401891157, "train_wd": 0.18719570509254668, "epoch": 132}
{"train_loss": 1.2170751430753586, "train_lr": 0.00019270046456647006, "train_wd": 0.18905084928739088, "epoch": 133}
{"train_loss": 1.2127445554608, "train_lr": 0.00019105670616761567, "train_wd": 0.1909093874053759, "epoch": 134}
{"train_loss": 1.2120006283316596, "train_lr": 0.00018940892172468463, "train_wd": 0.1927711156368536, "epoch": 135}
{"train_loss": 1.2069310646883287, "train_lr": 0.00018775730461248642, "train_wd": 0.1946358298223433, "epoch": 136}
{"train_loss": 1.206128834070951, "train_lr": 0.0001861020486556131, "train_wd": 0.196503325474921, "epoch": 137}
{"train_loss": 1.2064570540446886, "train_lr": 0.00018444334810568909, "train_wd": 0.19837339780264407, "epoch": 138}
{"train_loss": 1.210334176716881, "train_lr": 0.00018278139761857936, "train_wd": 0.2002458417310076, "epoch": 139}
{"train_loss": 1.2002466429206253, "train_lr": 0.00018111639223154162, "train_wd": 0.20212045192543432, "epoch": 140}
{"train_loss": 1.205855048459386, "train_lr": 0.0001794485273403404, "train_wd": 0.20399702281379176, "epoch": 141}
{"train_loss": 1.201404860886061, "train_lr": 0.00017777799867631547, "train_wd": 0.20587534860893503, "epoch": 142}
{"train_loss": 1.1957137008420133, "train_lr": 0.00017610500228341263, "train_wd": 0.20775522333127328, "epoch": 143}
{"train_loss": 1.1990967025337493, "train_lr": 0.00017442973449517609, "train_wd": 0.20963644083135952, "epoch": 144}
{"train_loss": 1.200380598763581, "train_lr": 0.00017275239191170895, "train_wd": 0.21151879481249541, "epoch": 145}
{"train_loss": 1.1959535594683437, "train_lr": 0.00017107317137659948, "train_wd": 0.21340207885335447, "epoch": 146}
{"train_loss": 1.193580395270106, "train_lr": 0.0001693922699538236, "train_wd": 0.21528608643061925, "epoch": 147}
{"train_loss": 1.1916153922891326, "train_lr": 0.00016770988490461545, "train_wd": 0.21717061094162746, "epoch": 148}
{"train_loss": 1.188758715888304, "train_lr": 0.00016602621366432015, "train_wd": 0.2190554457270306, "epoch": 149}
{"train_loss": 1.1935037238754134, "train_lr": 0.00016434145381922282, "train_wd": 0.22094038409345393, "epoch": 150}
{"train_loss": 1.1858069163672955, "train_lr": 0.0001626558030833612, "train_wd": 0.2228252193361643, "epoch": 151}
{"train_loss": 1.188781982469084, "train_lr": 0.00016096945927532226, "train_wd": 0.22470974476173702, "epoch": 152}
{"train_loss": 1.186776782657984, "train_lr": 0.00015928262029502968, "train_wd": 0.22659375371072257, "epoch": 153}
{"train_loss": 1.1848745166233419, "train_lr": 0.00015759548410051614, "train_wd": 0.22847703958030907, "epoch": 154}
{"train_loss": 1.179297057656258, "train_lr": 0.00015590824868469462, "train_wd": 0.23035939584697765, "epoch": 155}
{"train_loss": 1.1830243970621872, "train_lr": 0.00015422111205212237, "train_wd": 0.23224061608915209, "epoch": 156}
{"train_loss": 1.1825099390049556, "train_lr": 0.00015253427219576334, "train_wd": 0.23412049400983312, "epoch": 157}
{"train_loss": 1.177629784557276, "train_lr": 0.0001508479270737535, "train_wd": 0.23599882345922252, "epoch": 158}
{"train_loss": 1.1774124071348162, "train_lr": 0.00014916227458617015, "train_wd": 0.23787539845732933, "epoch": 159}
{"train_loss": 1.1733541360899675, "train_lr": 0.00014747751255180628, "train_wd": 0.23975001321655767, "epoch": 160}
{"train_loss": 1.1724049934903078, "train_lr": 0.00014579383868495676, "train_wd": 0.24162246216427447, "epoch": 161}
{"train_loss": 1.1707856529680765, "train_lr": 0.0001441114505722144, "train_wd": 0.24349253996535158, "epoch": 162}
{"train_loss": 1.1721347715722117, "train_lr": 0.00014243054564928406, "train_wd": 0.24536004154468552, "epoch": 163}
{"train_loss": 1.173259326998573, "train_lr": 0.00014075132117781082, "train_wd": 0.24722476210968197, "epoch": 164}
{"train_loss": 1.167583369279066, "train_lr": 0.00013907397422223205, "train_wd": 0.24908649717271747, "epoch": 165}
{"train_loss": 1.1612190202669759, "train_lr": 0.00013739870162664985, "train_wd": 0.2509450425735634, "epoch": 166}
{"train_loss": 1.1647622720049942, "train_lr": 0.00013572569999173103, "train_wd": 0.25280019450177227, "epoch": 167}
{"train_loss": 1.159944876517474, "train_lr": 0.00013405516565163502, "train_wd": 0.2546517495190289, "epoch": 168}
{"train_loss": 1.1632022514807439, "train_lr": 0.00013238729465097374, "train_wd": 0.25649950458146153, "epoch": 169}
{"train_loss": 1.163216542089935, "train_lr": 0.00013072228272180373, "train_wd": 0.2583432570619047, "epoch": 170}
{"train_loss": 1.1554313779981658, "train_lr": 0.00012906032526065745, "train_wd": 0.26018280477212197, "epoch": 171}
{"train_loss": 1.1548928123346605, "train_lr": 0.00012740161730561144, "train_wd": 0.2620179459849766, "epoch": 172}
{"train_loss": 1.1518638640131176, "train_lr": 0.0001257463535133987, "train_wd": 0.2638484794565549, "epoch": 173}
{"train_loss": 1.1477938883121959, "train_lr": 0.00012409472813656444, "train_wd": 0.2656742044482334, "epoch": 174}
{"train_loss": 1.1508836093630674, "train_lr": 0.0001224469350006693, "train_wd": 0.26749492074869263, "epoch": 175}
{"train_loss": 1.1505381642239152, "train_lr": 0.00012080316748154406, "train_wd": 0.26931042869587224, "epoch": 176}
{"train_loss": 1.145343760762001, "train_lr": 0.00011916361848259463, "train_wd": 0.2711205291988676, "epoch": 177}
{"train_loss": 1.1440409388459098, "train_lr": 0.00011752848041216616, "train_wd": 0.27292502375976047, "epoch": 178}
{"train_loss": 1.138714359990791, "train_lr": 0.00011589794516096053, "train_wd": 0.274723714495387, "epoch": 179}
{"train_loss": 1.1420929789938759, "train_lr": 0.00011427220407951908, "train_wd": 0.2765164041590392, "epoch": 180}
{"train_loss": 1.141322568355672, "train_lr": 0.00011265144795576587, "train_wd": 0.27830289616209325, "epoch": 181}
{"train_loss": 1.13428166183948, "train_lr": 0.0001110358669926178, "train_wd": 0.28008299459556923, "epoch": 182}
{"train_loss": 1.1396118809378384, "train_lr": 0.0001094256507856644, "train_wd": 0.28185650425161385, "epoch": 183}
{"train_loss": 1.1334863024623651, "train_lr": 0.00010782098830091678, "train_wd": 0.2836232306449081, "epoch": 184}
{"train_loss": 1.131073048816318, "train_lr": 0.00010622206785263222, "train_wd": 0.28538298003399337, "epoch": 185}
{"train_loss": 1.1310193280203154, "train_lr": 0.00010462907708121456, "train_wd": 0.2871355594425188, "epoch": 186}
{"train_loss": 1.1265483911651952, "train_lr": 0.00010304220293119388, "train_wd": 0.28888077668040196, "epoch": 187}
{"train_loss": 1.1252250401962516, "train_lr": 0.00010146163162928732, "train_wd": 0.29061844036490675, "epoch": 188}
{"train_loss": 1.1231817850117793, "train_lr": 9.988754866254527e-05, "train_wd": 0.29234835994162767, "epoch": 189}
{"train_loss": 1.1264584160088438, "train_lr": 9.832013875658285e-05, "train_wd": 0.29407034570538937, "epoch": 190}
{"train_loss": 1.120118377153325, "train_lr": 9.67595858539022e-05, "train_wd": 0.29578420882104756, "epoch": 191}
{"train_loss": 1.1155596222003212, "train_lr": 9.52060730923054e-05, "train_wd": 0.29748976134419786, "epoch": 192}
{"train_loss": 1.11375545139169, "train_lr": 9.365978278340321e-05, "train_wd": 0.29918681624178667, "epoch": 193}
{"train_loss": 1.1138898587035657, "train_lr": 9.212089639121896e-05, "train_wd": 0.30087518741262026, "epoch": 194}
{"train_loss": 1.1139203502870767, "train_lr": 9.058959451089418e-05, "train_wd": 0.3025546897077736, "epoch": 195}
{"train_loss": 1.1094318839795558, "train_lr": 8.906605684749376e-05, "train_wd": 0.30422513895089354, "epoch": 196}
{"train_loss": 1.1112034465145089, "train_lr": 8.755046219491763e-05, "train_wd": 0.30588635195839653, "epoch": 197}
{"train_loss": 1.1071973581163757, "train_lr": 8.60429884149179e-05, "train_wd": 0.3075381465595561, "epoch": 198}
{"train_loss": 1.099203700405768, "train_lr": 8.454381241622658e-05, "train_wd": 0.30918034161648, "epoch": 199}
{"train_loss": 1.099097526492139, "train_lr": 8.305311013379425e-05, "train_wd": 0.31081275704397476, "epoch": 200}
{"train_loss": 1.1004753984115292, "train_lr": 8.157105650814283e-05, "train_wd": 0.3124352138292931, "epoch": 201}
{"train_loss": 1.0972965773673051, "train_lr": 8.009782546483681e-05, "train_wd": 0.31404753405176444, "epoch": 202}
{"train_loss": 1.092984740729485, "train_lr": 7.863358989407081e-05, "train_wd": 0.3156495409023077, "epoch": 203}
{"train_loss": 1.0960404245116937, "train_lr": 7.717852163038118e-05, "train_wd": 0.31724105870281877, "epoch": 204}
{"train_loss": 1.0928613061612817, "train_lr": 7.573279143248025e-05, "train_wd": 0.318821912925436, "epoch": 205}
{"train_loss": 1.0920073322094648, "train_lr": 7.429656896321662e-05, "train_wd": 0.32039193021167894, "epoch": 206}
{"train_loss": 1.0849385423239617, "train_lr": 7.287002276966499e-05, "train_wd": 0.32195093839145994, "epoch": 207}
{"train_loss": 1.086971781842938, "train_lr": 7.145332026334629e-05, "train_wd": 0.3234987665019642, "epoch": 208}
{"train_loss": 1.0813274293874218, "train_lr": 7.004662770058076e-05, "train_wd": 0.32503524480639606, "epoch": 209}
{"train_loss": 1.0791152904584873, "train_lr": 6.865011016297773e-05, "train_wd": 0.3265602048125955, "epoch": 210}
{"train_loss": 1.0761697434910895, "train_lr": 6.726393153806212e-05, "train_wd": 0.3280734792915127, "epoch": 211}
{"train_loss": 1.0794404465622738, "train_lr": 6.588825450004132e-05, "train_wd": 0.32957490229554814, "epoch": 212}
{"train_loss": 1.0754581980275368, "train_lr": 6.45232404907155e-05, "train_wd": 0.33106430917674984, "epoch": 213}
{"train_loss": 1.0727163913898763, "train_lr": 6.316904970053076e-05, "train_wd": 0.33254153660487, "epoch": 214}
{"train_loss": 1.0726363459453645, "train_lr": 6.182584104978022e-05, "train_wd": 0.33400642258527274, "epoch": 215}
{"train_loss": 1.0661814050765428, "train_lr": 6.049377216995485e-05, "train_wd": 0.33545880647670384, "epoch": 216}
{"train_loss": 1.0637150971400262, "train_lr": 5.9172999385243596e-05, "train_wd": 0.33689852900890205, "epoch": 217}
{"train_loss": 1.0641397414481746, "train_lr": 5.786367769418887e-05, "train_wd": 0.33832543230006773, "epoch": 218}
{"train_loss": 1.0615192341033073, "train_lr": 5.6565960751496254e-05, "train_wd": 0.33973935987417536, "epoch": 219}
{"train_loss": 1.056078793456267, "train_lr": 5.528000085000293e-05, "train_wd": 0.34114015667813274, "epoch": 220}
{"train_loss": 1.0578468480452368, "train_lr": 5.400594890280478e-05, "train_wd": 0.3425276690987848, "epoch": 221}
{"train_loss": 1.054848074393792, "train_lr": 5.274395442554689e-05, "train_wd": 0.3439017449797591, "epoch": 222}
{"train_loss": 1.0511530837934233, "train_lr": 5.14941655188763e-05, "train_wd": 0.3452622336381513, "epoch": 223}
{"train_loss": 1.051826697667088, "train_lr": 5.025672885106267e-05, "train_wd": 0.3466089858810468, "epoch": 224}
{"train_loss": 1.046208256253956, "train_lr": 4.9031789640785355e-05, "train_wd": 0.3479418540218876, "epoch": 225}
{"train_loss": 1.0447273803380344, "train_lr": 4.781949164009193e-05, "train_wd": 0.3492606918966609, "epoch": 226}
{"train_loss": 1.0428121057957147, "train_lr": 4.661997711752777e-05, "train_wd": 0.3505653548799322, "epoch": 227}
{"train_loss": 1.044226668519586, "train_lr": 4.543338684144047e-05, "train_wd": 0.3518556999007031, "epoch": 228}
{"train_loss": 1.0414691346128826, "train_lr": 4.425986006346e-05, "train_wd": 0.35313158545810114, "epoch": 229}
{"train_loss": 1.038192836032043, "train_lr": 4.309953450215688e-05, "train_wd": 0.35439287163689714, "epoch": 230}
{"train_loss": 1.037451881948298, "train_lr": 4.195254632688042e-05, "train_wd": 0.3556394201228481, "epoch": 231}
{"train_loss": 1.0366401289119922, "train_lr": 4.08190301417784e-05, "train_wd": 0.3568710942178651, "epoch": 232}
{"train_loss": 1.028051767729025, "train_lr": 3.969911897000097e-05, "train_wd": 0.35808775885500443, "epoch": 233}
{"train_loss": 1.029641574421749, "train_lr": 3.859294423808953e-05, "train_wd": 0.3592892806132772, "epoch": 234}
{"train_loss": 1.0282824208622023, "train_lr": 3.7500635760553364e-05, "train_wd": 0.36047552773228403, "epoch": 235}
{"train_loss": 1.0262311554588048, "train_lr": 3.6422321724635415e-05, "train_wd": 0.36164637012665946, "epoch": 236}
{"train_loss": 1.0233004989588748, "train_lr": 3.535812867526872e-05, "train_wd": 0.36280167940034086, "epoch": 237}
{"train_loss": 1.0209214830424933, "train_lr": 3.430818150022615e-05, "train_wd": 0.36394132886064734, "epoch": 238}
{"train_loss": 1.0191017907376574, "train_lr": 3.3272603415463845e-05, "train_wd": 0.36506519353217254, "epoch": 239}
{"train_loss": 1.0170858623707189, "train_lr": 3.225151595066155e-05, "train_wd": 0.36617315017049085, "epoch": 240}
{"train_loss": 1.0177517406131447, "train_lr": 3.124503893496051e-05, "train_wd": 0.36726507727567204, "epoch": 241}
{"train_loss": 1.0154096724844612, "train_lr": 3.0253290482901096e-05, "train_wd": 0.36834085510560294, "epoch": 242}
{"train_loss": 1.013073643710695, "train_lr": 2.927638698056129e-05, "train_wd": 0.3694003656891218, "epoch": 243}
{"train_loss": 1.0102413621549784, "train_lr": 2.8314443071898497e-05, "train_wd": 0.3704434928389536, "epoch": 244}
{"train_loss": 1.0077356305303036, "train_lr": 2.7367571645295374e-05, "train_wd": 0.3714701221644522, "epoch": 245}
{"train_loss": 1.0055394085429081, "train_lr": 2.6435883820312106e-05, "train_wd": 0.37248014108414146, "epoch": 246}
{"train_loss": 1.0048262930013634, "train_lr": 2.5519488934645823e-05, "train_wd": 0.37347343883806683, "epoch": 247}
{"train_loss": 1.0000261023962063, "train_lr": 2.461849453129943e-05, "train_wd": 0.37444990649993415, "epoch": 248}
{"train_loss": 1.0023978275403511, "train_lr": 2.3733006345961033e-05, "train_wd": 0.3754094369890625, "epoch": 249}
{"train_loss": 0.9999833804209082, "train_lr": 2.286312829459514e-05, "train_wd": 0.37635192508211884, "epoch": 250}
{"train_loss": 0.995666092780905, "train_lr": 2.2008962461248032e-05, "train_wd": 0.37727726742466483, "epoch": 251}
{"train_loss": 0.9959817434191308, "train_lr": 2.1170609086067297e-05, "train_wd": 0.37818536254248347, "epoch": 252}
{"train_loss": 0.9986522022502876, "train_lr": 2.0348166553538718e-05, "train_wd": 0.37907611085271203, "epoch": 253}
{"train_loss": 0.9922528021754614, "train_lr": 1.95417313809399e-05, "train_wd": 0.37994941467476084, "epoch": 254}
{"train_loss": 0.9929798559195041, "train_lr": 1.8751398207013937e-05, "train_wd": 0.3808051782410248, "epoch": 255}
{"train_loss": 0.9924680613789015, "train_lr": 1.7977259780862987e-05, "train_wd": 0.3816433077073856, "epoch": 256}
{"train_loss": 0.9913087562097451, "train_lr": 1.721940695106367e-05, "train_wd": 0.3824637111635003, "epoch": 257}
{"train_loss": 0.9861864087180094, "train_lr": 1.647792865500572e-05, "train_wd": 0.3832662986428876, "epoch": 258}
{"train_loss": 0.9858563996571224, "train_lr": 1.5752911908454728e-05, "train_wd": 0.38405098213278427, "epoch": 259}
{"train_loss": 0.9831302939312516, "train_lr": 1.5044441795340485e-05, "train_wd": 0.38481767558380386, "epoch": 260}
{"train_loss": 0.9852392861405305, "train_lr": 1.435260145777192e-05, "train_wd": 0.3855662949193685, "epoch": 261}
{"train_loss": 0.98016772293939, "train_lr": 1.3677472086280127e-05, "train_wd": 0.386296758044933, "epoch": 262}
{"train_loss": 0.9772953797107051, "train_lr": 1.3019132910290201e-05, "train_wd": 0.38700898485698393, "epoch": 263}
{"train_loss": 0.9801677357182895, "train_lr": 1.2377661188823446e-05, "train_wd": 0.38770289725182544, "epoch": 264}
{"train_loss": 0.978979548220976, "train_lr": 1.1753132201430425e-05, "train_wd": 0.38837841913414245, "epoch": 265}
{"train_loss": 0.9772988662980826, "train_lr": 1.1145619239356877e-05, "train_wd": 0.38903547642535014, "epoch": 266}
{"train_loss": 0.9759149223368252, "train_lr": 1.0555193596942484e-05, "train_wd": 0.3896739970717106, "epoch": 267}
{"train_loss": 0.9747701879566024, "train_lr": 9.981924563254158e-06, "train_wd": 0.3902939110522397, "epoch": 268}
{"train_loss": 0.9737108684581729, "train_lr": 9.425879413954783e-06, "train_wd": 0.3908951503863841, "epoch": 269}
{"train_loss": 0.9721088271950726, "train_lr": 8.887123403407999e-06, "train_wd": 0.3914776491414759, "epoch": 270}
{"train_loss": 0.9686770744217906, "train_lr": 8.365719757020349e-06, "train_wd": 0.39204134343996105, "epoch": 271}
{"train_loss": 0.9698984814288539, "train_lr": 7.861729663821536e-06, "train_wd": 0.3925861714664091, "epoch": 272}
{"train_loss": 0.9686554931145392, "train_lr": 7.375212269283579e-06, "train_wd": 0.3931120734742857, "epoch": 273}
{"train_loss": 0.9700315492474927, "train_lr": 6.906224668379831e-06, "train_wd": 0.3936189917925105, "epoch": 274}
{"train_loss": 0.9679153520506766, "train_lr": 6.454821898884645e-06, "train_wd": 0.3941068708317773, "epoch": 275}
{"train_loss": 0.9665576846141005, "train_lr": 6.021056934914423e-06, "train_wd": 0.3945756570906511, "epoch": 276}
{"train_loss": 0.9680534396361791, "train_lr": 5.604980680710925e-06, "train_wd": 0.39502529916143564, "epoch": 277}
{"train_loss": 0.9642110913991928, "train_lr": 5.206641964667367e-06, "train_wd": 0.39545574773581194, "epoch": 278}
{"train_loss": 0.9634479530056758, "train_lr": 4.826087533598204e-06, "train_wd": 0.3958669556102421, "epoch": 279}
{"train_loss": 0.9647843381596021, "train_lr": 4.4633620472532234e-06, "train_wd": 0.39625887769114887, "epoch": 280}
{"train_loss": 0.9623740180807633, "train_lr": 4.118508073076496e-06, "train_wd": 0.39663147099986024, "epoch": 281}
{"train_loss": 0.962872574336629, "train_lr": 3.79156608121089e-06, "train_wd": 0.39698469467732067, "epoch": 282}
{"train_loss": 0.962310821745149, "train_lr": 3.482574439748737e-06, "train_wd": 0.397318509988573, "epoch": 283}
{"train_loss": 0.9587763018774248, "train_lr": 3.191569410229164e-06, "train_wd": 0.39763288032700783, "epoch": 284}
{"train_loss": 0.9591802291044619, "train_lr": 2.9185851433826347e-06, "train_wd": 0.39792777121837364, "epoch": 285}
{"train_loss": 0.9604082268300111, "train_lr": 2.663653675123216e-06, "train_wd": 0.39820315032456216, "epoch": 286}
{"train_loss": 0.9587262105732431, "train_lr": 2.426804922789014e-06, "train_wd": 0.3984589874471502, "epoch": 287}
{"train_loss": 0.9592475521304042, "train_lr": 2.2080666816312486e-06, "train_wd": 0.3986952545307147, "epoch": 288}
{"train_loss": 0.9577035252004862, "train_lr": 2.0074646215523584e-06, "train_wd": 0.3989119256659073, "epoch": 289}
{"train_loss": 0.9586081852625429, "train_lr": 1.8250222840935152e-06, "train_wd": 0.3991089770922966, "epoch": 290}
{"train_loss": 0.9587571681641965, "train_lr": 1.6607610796719286e-06, "train_wd": 0.39928638720097287, "epoch": 291}
{"train_loss": 0.9573075459421142, "train_lr": 1.5147002850682333e-06, "train_wd": 0.39944413653691935, "epoch": 292}
{"train_loss": 0.9561012626864609, "train_lr": 1.3868570411642798e-06, "train_wd": 0.39958220780114223, "epoch": 293}
{"train_loss": 0.9595896629781454, "train_lr": 1.2772463509315658e-06, "train_wd": 0.3997005858525723, "epoch": 294}
{"train_loss": 0.9575634766740939, "train_lr": 1.18588107767058e-06, "train_wd": 0.3997992577097222, "epoch": 295}
{"train_loss": 0.9555465568689449, "train_lr": 1.1127719435012293e-06, "train_wd": 0.3998782125521103, "epoch": 296}
{"train_loss": 0.9585663597800035, "train_lr": 1.0579275281045622e-06, "train_wd": 0.3999374417214491, "epoch": 297}
{"train_loss": 0.957040233805413, "train_lr": 1.0213542677158922e-06, "train_wd": 0.3999769387225926, "epoch": 298}
{"train_loss": 0.9557520198652241, "train_lr": 1.0030564543694825e-06, "train_wd": 0.39999669922424963, "epoch": 299}