💊 CI failures summary and remediations

As of commit 92ae7dd (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

@xiaohu2015 Can you please share the logs with @jdsgomes and all the information that will allow us to reproduce your experiment (for example the git commit hashcodes you used etc). It's also unclear to me whether you used TorchVision's reference scripts or something else in your experiments. Could you please clarify?

Hi @xiaohu2015
Finally I get the final results:

• swin_t: 81.204 (batch 1 testing) vs 81.204 (results reported after training) vs 81.3 (paper)
• swin_s: 82.602 (batch 1 testing) vs 82.598 (results reported after training) vs 83 (paper)
• swin_b: 82.750 (batch 1 testing) vs 82.940 (results reported after training) vs 83.5 (paper)

As you see there are some small discrepancies in the final testing vs the results reported in the paper. I will reach out offline to discuss what could be the causes of the discrepancies
ps: corrected the results after @xiaohu2015 spotted the missing pre-processing transforms in the Weights.

thanks, it seems that the swin_t can match the offical result, but for swin_s and swin_b, it drops about 0.5 point. I checked the training logs, I found the result can match the result with https://github.com/open-mmlab/mmclassification/tree/master/configs/swin_transformer before 200 epoch (for swint_b) and 250 epoch (for swint_s). I think maybe there is some difference about the lr scheduler. Torchvision use epoch-based lr scheduler, but timm and mmcls use iteration-based lr scheduler. @datumbox @jdsgomes what your thoughts?

Despite they use iteration-based lr scheduler, in the official repo they convert parameterise the scheduler builder with the number of epochs and just convert the number of epochs to iterations so I think it should be equivalent.

@xiaohu2015 Can you please share the logs with @jdsgomes and all the information that will allow us to reproduce your experiment (for example the git commit hashcodes you used etc). It's also unclear to me whether you used TorchVision's reference scripts or something else in your experiments. Could you please clarify?

for swin_t, I have shared the training logs (just use TorchVision's reference script ) with @jdsgomes, I got Acc@1 81.222 Acc@5 95.332, @jdsgomes can reproduce the training (81.204), the result can match the offical result. But for swin_s and swin_b, it drops about 0.5 point compared to the offical result, in fact, I only trained swin_t. I plan to check these two models. thanks to the training logs from @jdsgomes, I found that these two models seem to get behind in the last few epochs.

@jdsgomes yes, iteration-based lr scheduler and epoch-based lr scheduler should be equivalent. I just suspect that might be the reason, because some minor difference can make a result difference.

@xiaohu2015 Can you please share the logs with @jdsgomes and all the information that will allow us to reproduce your experiment (for example the git commit hashcodes you used etc). It's also unclear to me whether you used TorchVision's reference scripts or something else in your experiments. Could you please clarify?

for swin_t, I have shared the training logs (just use TorchVision's reference script ) with @jdsgomes, I got Acc@1 81.222 Acc@5 95.332, @jdsgomes can reproduce the training (81.204), the result can match the offical result. But for swin_s and swin_b, it drops about 0.5 point compared to the offical result, in fact, I only trained swin_t. I plan to check these two models. thanks to the training logs from @jdsgomes, I found that these two models seem to get behind in the last few epochs.

@jdsgomes yes, iteration-based lr scheduler and epoch-based lr scheduler should be equivalent. I just suspect that might be the reason, because some minor difference can make a result difference.

After discussing offline with @datumbox I think we should proceed to merge the PR with the swin_t only since it is clear that we can reproduce the result, so great work @xiaohu2015 !

After that we can continue investigations to close the gap and aim to merge the other variants in a different PR.

I will do the final cleanups between today and tomorrow.

Just wanted to echo what Joao said. Big massive thank you @xiaohu2015 for your awesome contribution. Top notch code and excellent research reproduction skills. Also apologies for taking us long to review and reproduce the PR; it's something we want to improve upon.

Looking forward seeing this merged!

@datumbox @jdsgomes Thanks very much！

for other models, I can convert the offical weight to torchvision version just like efficientnet.

to

@xiaohu2015 I understand that would be useful to include pre-trained weights from the initial implementation, but we would prefer to include the other variants once we can replicate the results fully. I am running a few experiments now, and hopefully we can get good results, but for now I will remove even the constructors so this PR can be merged.