Performance discrepancy between local runs and hugging face demo

[vietpho]

Hello, first of all, thank you for sharing your amazing work. However, I've noticed a difference in the results between the Hugging Face demo and running the demo.py code from the cloned GitHub repository. Even when using the same images, the performance on the Hugging Face demo appears to be much better. Could you possibly share the config settings used for the Hugging Face demo?

One change I made was to disable augmentation; I set it to accept the original image sizes directly. This adjustment improved the performance of demo.py slightly, though it still doesn't match the Hugging Face demo. It seems all the pretrained models you provided perform differently compared to HF.

I'm quite frustrated because the performance locally doesn't match the impressive results of the Hugging Face demo. I haven't modified your code except for the later tweak with the augmentation. I'm wondering if you might know why there is such a significant performance difference with the same images on the Hugging Face demo. Thanks!

[praeclarumjj3]

Hi @vietpho, Could you share the task you are testing and your results? The HF demo code is the same as the GitHub demo code. I don't remember making any changes.

This could be due to the difference between the package version on HF and the local runs. I remember having difficulties setting up the environment, as suggested in this repo on HF, so I had to change a few package versions. You can find the exact setup details from the files below:

https://huggingface.co/spaces/shi-labs/OneFormer/blob/main/requirements.txt
https://huggingface.co/spaces/shi-labs/OneFormer/blob/main/Dockerfile

Btw, the augmentation is happening in the HF demo as well.
https://huggingface.co/spaces/shi-labs/OneFormer/blob/main/demo/defaults.py#L70