optimizer_name() is None when used client optimizer

[vfdev-5]

There is a bug encountered when using client optimizer (Adam) and zero optimization:

class Args:
    pass

model = wide_resnet50_2(num_classes=100).to(device)
optimizer = Adam(model.parameters(), lr=0.001)

ds_args = Args()
ds_args.local_rank = 0
ds_args.deepspeed_config = None
ds_config_params = {
        "train_batch_size": batch_size,
        "steps_per_print": len(train_loader),
        "fp16": {
            "enabled": "true",
        },
        "zero_optimization": {
            "stage": 2,
        },
        # "zero_allow_untested_optimizer": "true"    
}
model_engine = DeepSpeedLight(ds_args, model, optimizer=optimizer, config_params=ds_config_params)

In this case self.optimizer_name() is None and the following check is incorrect:

https://github.com/microsoft/DeepSpeed/blob/4d64b5ba055b663a1e436dbc2d756350554dfe84/deepspeed/pt/deepspeed_light.py#L489

'You are using an untested ZeRO Optimizer. Please add <"zero_allow_untested_optimizer": true> in the configuration file to use it.'
AssertionError: You are using an untested ZeRO Optimizer. Please add <"zero_allow_untested_optimizer": true> in the configuration file to use it.

[tjruwase]

Thanks for using DeepSpeed.

What happens when you set zero_allow_untested_optimizer to true?

[vfdev-5]

Yes, setting zero_allow_untested_optimizer to true solves the problem.

[ShadenSmith]

Good catch!

In cases where we don't have an optimizer name from our config, maybe we should next defer to self.optimizer.__class__.__name__ before returning None from optimizer_name() ? In this case I believe that would give us "Adam" and do the correct thing. If no optimizer was specified we should still return None, I imagine.

Here's optimizer_name():

https://github.com/microsoft/DeepSpeed/blob/4d64b5ba055b663a1e436dbc2d756350554dfe84/deepspeed/pt/deepspeed_light.py#L264-L265

[tjruwase]

Fixed by PR #349.