Inflexible score function of CheckpointSaver

[sandylaker]

MONAI/monai/handlers/checkpoint_saver.py

Lines 148 to 160 in 9f9b63a

	def _score_func(engine: Engine):
	if isinstance(key_metric_name, str):
	metric_name = key_metric_name
	elif hasattr(engine.state, "key_metric_name") and isinstance(engine.state.key_metric_name, str):
	metric_name = engine.state.key_metric_name
	else:
	raise ValueError(
	f"Incompatible values: save_key_metric=True and key_metric_name={key_metric_name}."
	)
	return round(engine.state.metrics[metric_name], 4)
	if key_metric_filename is not None and key_metric_n_saved > 1:
	raise ValueError("if using fixed filename to save the best metric model, we should only save 1 model.")

Currently the CheckpointSaver saves the check point that has the largest score value. However, this is not desirable when a lower score indicates better performance, e.g. validation loss. We can either allow the user to pass a custom score function (as in the ignite.handlers), or add a flag to control returning round(engine.state.metrics[metric_name], 4) or - round(engine.state.metrics[metric_name], 4)

[Nic-Ma]

Hi @sandylaker , thanks for raising this question, I think it's a question for ignite Checkpoint handler.
@vfdev-5 , could you please help share some comments? I think you have encountered this question.
Maybe adding a ”-“ before the metrics value if the smaller is better?

Thanks.

[vfdev-5]

Hi @Nic-Ma , in ignite we provide (since v0.4.3) a default score function from metric : https://pytorch.org/ignite/generated/ignite.handlers.checkpoint.Checkpoint.html#ignite.handlers.checkpoint.Checkpoint.get_default_score_fn in addition to score_function argument such that both cases: larger-better or lower-better are covered:

from ignite.handlers import Checkpoint

best_acc_score = Checkpoint.get_default_score_fn("accuracy")
best_model_handler = Checkpoint(
    to_save, save_handler, score_name="val_accuracy", score_function=best_acc_score
)
evaluator.add_event_handler(Events.COMPLETED, best_model_handler)

and

from ignite.handlers import Checkpoint

neg_loss_score = Checkpoint.get_default_score_fn("loss", -1.0)
best_model_handler = Checkpoint(
    to_save, save_handler, score_name="val_neg_loss", score_function=neg_loss_score
)
evaluator.add_event_handler(Events.COMPLETED, best_model_handler)

Maybe, something similar could be done from MONAI's side ?

[Nic-Ma]

Hi @vfdev-5 ,

Cool! I will try to update MONAI side to leverage this new ignite feature soon.
Let's keep this ticket open to track it.

Thanks.

[Nic-Ma]

Hi @vfdev-5 ,

I submitted PR #2487 to add support for this feature request.
I think there is a minor drawback in ignite: the saved checkpoint will have a "-" in the filename:
net_validation_loss=-0.1234.pt
Maybe you can consider to correct it back to positive number in the filename in future ignite version?

Thanks.

[vfdev-5]

@Nic-Ma thanks ! Yes, I agree that it is better to show actual value instead of negative one. We had previously abs(score) such that negative values become positive but we removed that. Now, the main pain point is that score function return negative value as well...

[Nic-Ma]

Hi @vfdev-5 , so do you plan to add back abs(score) logic in next ignite version?

Thanks.