num_classes issue

[virgile-fsr]

rf-detr/rfdetr/main.py

Line 92 in 6ca1b58

self.reinitialize_detection_head(checkpoint_num_classes)

If user creates a new model with:
model = RFDETRBase(num_classes=123)
The code linked above will overwrite the num classes specified by the users by the one in the checkpoint (90 by default). I don't think this is expected behavior. More worrisome, the config of the object stays with num_classes=123 which will cause issues in the train function.

If you call: model.train(dataset_dir=...) with the model above, it will check the number of classes in the dataset and compare it with self.model_config.num_classes which is still 123 in our example (even though the actual model has a 90 sized output layer). So if the number of classes in the dataset is indeed 123, the check will pass even though it should not.

A different issue it that if you try a dataset with a single class, the labels from the dataloader will have size zero.

[isaacrob-roboflow]

single class is being discussed here #48, the issue is due to our working mainly with datasets exported from roboflow which have a filler 0 class causing the issue you're observing with non-roboflow format datasets.

good point re using number of classes as an excuse to reinitialize the head! do you have an idea of what a preferred interface would look like? maybe something like having num_classes default to None and if it is not None, we reinitialize the head with that number of classes?

[anyaflya]

I encounter a similar issue, when loading from pretrained weights. I want the model to predict two classes and thus instantiate it with:
model = RFDETRLarge(num_classes=2).
I get printed:

Loading pretrain weights

WARNING:rfdetr.main:num_classes mismatch: pretrain weights has 90 classes, but your model has 2 classes
reinitializing detection head with 90 classes

It thus trained with 90 classes. Moreover, when predicting with the trained 90-class model
model = RFDETRBase(pretrain_weights="checkpoint_best_total.pth", num_classes=2)
I have:
num_classes mismatch: pretrain weights has 1 classes, but your model has 2 classes.

[panagiotamoraiti]

I have a similar issue. I train with 10 classes and i get a warning during training that says num_classes mismatch: pretrain weights has 90 classes, but your model has 10 classes reinitializing detection head with 2 classes.

Then in prediction i get num_classes mismatch: pretrain weights has 9 classes, but your model has 90 classes
reinitializing detection head with 9 classes. Why in prediction it finds one less class? But i get predictions from 0 to 9, so the results are correct.

I dont specify num_classes when defining the model. Which is the correct way? Do you think i will have a problem with this?

[anyaflya]

On my side, I managed to make the training and predictions work with the following:

1. training: instantiating the model without any arguments, and give the dataset when training.

   model = RFDETRBase()
   model.train(
               dataset_dir=dataset_dir,
               output_dir=output_dir,
           )
   

   In my case, my dataset contains two classes.
2. predictions: instantiate the model with the checkpoint path and the number of classes.

   model = RFDETRBase(
                   pretrain_weights=str(checkpoint_path), num_classes=num_classes
               )
   model.predict(image, threshold=0.5)
   

   It prints the warning

   WARNING - num_classes mismatch: pretrain weights has 1 classes, but your model has 2 classes
   reinitializing detection head with 1 classes
   

   but the model indeed predicts 2 classes.

There is clearly a bug with the warnings, would be nice to check that out @isaacrob-roboflow.

[panagiotamoraiti]

I've done the same as you suggested. I don't use num_classes argument. During prediction it gives a warning that it initializes the model with one class less than my actual number of classes, but predictions are done for all classes.

[isaacrob-roboflow]

yeah I think there's a bug in the warnings .. @SkalskiP ?

[SkalskiP]

I see two main issues here:

• The warning message is misleading and suggests that users did something wrong. Since the default value of num_classes is 90, anyone trying to load a fine-tuned model gets the warning: num_classes mismatch: pretrain weights has 10 classes, but your model has 90 classes.

• Since we're logging checkpoint_num_classes - 1, the reported number of classes in the checkpoint is likely lower than the actual value. This looks like a bug.

@isaacrob-roboflow In general, I believe num_classes should not be a user-configurable argument. It should be inferred either from the checkpoint or the dataset. I don’t see any case where manually setting it should be the user’s responsibility. The comments from @panagiotamoraiti and @anyaflya are the best evidence that num_classes is misleading for users.

• When training a model, num_classes should match the number of classes in the dataset. Optionally, we can inform the user if the pretrained checkpoint has a different number of classes.

• When loading a model, num_classes should be set based on the value in the checkpoint.

[isaacrob-roboflow]

I don't have strong feelings about this! Although I would like there to be a mechanism to load a model with a set number of classes without having to have a dataset exist. This is useful for people who want to train with custom data pipelines.

@SkalskiP feel free to make a change you think is best as long as it is still possible to create the model without a dataset present :)

[SkalskiP]

@isaacrob-roboflow I don’t know anyone using RF-DETR to train with custom data pipelines. Could you give me a bit more context on how and why someone would want to do that?

[isaacrob-roboflow]

I mean I have very often built my own data pipeline. useful for implementing certain augmentations, plugging into large datasets that can't be entirely stored locally, plugging into existing data pipelines .. I think it is more likely to be an issue in research cases and more mature enterprise pipelines. also useful for benchmarking for a given use case without committing to having data present in the appropriate format. I definitely think it should be possible to instantiate a model with an arbitrary configuration without relying on having a dataset .. separation of computation and transaction, and all that

[malaccan]

I have the exact same issue where I fine tuned with my own custom dataset with just 1 class - and it gives a warning that the model has 0 classes when loading the model. I thought I did something wrong and re-checked all the code. but predictions did work as expected. and after reading about this here, it does look like a genuine bug (or at worse a misleading warning that cause unnecessary alarm).

[cduong-a]

I was also confused by this 😅. I have a couple of questions:

1. In lwdetr.py, the build_model() function sets num_classes=args.num_classes + 1 when constructing the model. Apparently this is carried over from the original DETR model, which adds a no-object class and sets its ID to max_id + 1. Does RF-DETR actually use a no-object class? The pre-trained models have a class_embed size of 91 (whereas COCO has 90 classes). Fine-tuning seems to work without the extra class, but is this how we expect the model to behave?
2. I agree the warnings are misleading. Because reinitialize_detection_head() is called before load_state_dict(), it doesn't affect inference, but at first I thought my trained weights had been overridden. When loading a model, I expect to call something like RFDetrBase.from_checkpoint(pretrain_weights=...) and have the classes inferred. At the moment, the only way to suppress the warning is to explicitly pass num_classes=<actual num classes minus 1>. Of course, this is assuming the model is not meant to have the no-object class.

[GabrieleGiudic]

yeah I think there's a bug in the warnings .. @SkalskiP ?

Same problem me too