Accuracy degradation when converting weights to `--fp16` with `trtexec`.

[rodneyML]

I have successfully converted EfficientTAM to TensorRT (from ONNX) thanks for the tips @okideal in #7 , but I am seeing a very strange issue with the image_encoder if I use trtexec with --fp16. It degrades the accuracy by A LOT when --fp16 is used. This is the first time I am seeing this with a model converted to --fp16 (normally the accuracy degrades, but by a negligible margin).

To confirm this was indeed because of --fp16, I have built it with the normal (fp32) and --fp16. I also set the LayerNorm nodes and their inputs/outputs to trt.DataType.FLOAT of the --fp16 model, but still no luck. It seems that if I set all of the layers containing blocks in the name to trt.DataType.FLOAT, it works fine (but that is basically the whole image encoder).

Here are some screenshots of the accuracy degradation (on a still image):

FP16 version	FP32 version

@yformer Do you know what might be the issue with converting the image encoder weights to FP16 ?
@okideal Did you encounter similar issues with your setup or did you just leave it in fp32?

[okideal]

class ImageEncoder(nn.Module):
    def __init__(self, sam_model: SAM2Base) -> None:
        super().__init__()
        self.model = sam_model
        self.no_mem_embed = sam_model.no_mem_embed  # [1,1,256]
        self.image_encoder = sam_model.image_encoder
        self.num_feature_levels = sam_model.num_feature_levels
        self.prepare_backbone_features = sam_model._prepare_backbone_features

    @torch.no_grad()
    def forward(self, image: torch.Tensor):
        backbone_out = self.image_encoder(image)  # {"vision_features","vision_pos_enc","backbone_fpn"}

        vision_pos_enc = backbone_out["vision_pos_enc"]  # 有1个tensor
        pix_feat = backbone_out["vision_features"]  # 有1个tensor
        backbone_fpn = [pix_feat]  # 有1个tensor

        expanded_backbone_out = {
            "backbone_fpn": backbone_fpn,
            "vision_pos_enc": vision_pos_enc,
        }
        for i, feat in enumerate(expanded_backbone_out["backbone_fpn"]):
            expanded_backbone_out["backbone_fpn"][i] = feat.expand(1, -1, -1, -1)
        for i, pos in enumerate(expanded_backbone_out["vision_pos_enc"]):
            expanded_backbone_out["vision_pos_enc"][i] = pos.expand(1, -1, -1, -1)

        # 均为1x256x64x64
        (_, current_vision_feats, current_vision_pos_embeds, _) = self.prepare_backbone_features(expanded_backbone_out)

        current_vision_feat = current_vision_feats[-1] + self.no_mem_embed
        current_vision_feat = current_vision_feat.permute(1, 2, 0).view(1, 256, 64, 64)  # [1,256,64,64]

        # pix_feat              [1, 256, 64, 64]
        # current_vision_feat   [1, 256, 64, 64]
        # current_vision_pos_embed [4096, 1, 256]
        return pix_feat, current_vision_feat, current_vision_pos_embeds[-1]````

Above is my definition of image encoder. As for conversion precision of TensorRT, I use --fp16 for all models. There is no significant downgration in my exp (but note that I have only tested the box-based prompt. Point prompt one not tested which in your case.)

[rodneyML]

Thanks @okideal . I have the exact same implementation of the image encoder as the above. I still however experience accuracy degradation, with both box and point prompts. Would you mind sharing what version of TensorRT did you use to perform onnx to engine conversion with trtexec? I am currently on v10.7.0.23-1+cuda12.6.

&&&& PASSED TensorRT.trtexec [TensorRT v100700] [b23] # /usr/src/tensorrt/bin/trtexec

Apt-cache policy output:

tensorrt:
  Installed: 10.7.0.23-1+cuda12.6
  Candidate: 10.7.0.23-1+cuda12.6

Note: I upgraded to the above version of tensorrt (previous version: 8.5.0.2) after failing to convert the memory attention model that resulted in operation.cpp:203: DCHECK(!i->is_use_only()) failed. This error does not pop up when using the newer version (10.7).

Another note: Would you mind sharing the build process from ONNX -> TRT ? Did you just use use trtexec or did you build it using a C++ or Python API ? I think that might be a big differentiator between my degraded --fp16 model and yours. Did you force some layers to run in fp32 ?

[okideal]

Hello @reynhardtstef , I am using the TensorRT-8.6.1.6.Linux.x86_64-gnu.cuda-12.0 version.
After some searching, I found that version 8.5.0.2 indeed has the bug you mentioned, operation.cpp:203: DCHECK(!i->is_use_only()). Maybe you could try the 8.6 version.
For the conversion, I used trtexec with ONNX opset 17, applied the --fp16 flag, and set some dynamic shapes. I didn’t use Python or C++ scripts for the conversion. However, as far as I know, in terms of precision under fp16/32, the differences between these methods should not be significant.

[rodneyML]

Thanks @okideal . I found that TensorRT v8.6.2 successfully converts the correct image_encoder, but unfortunately I encountered NVRTC compilation failed with the memory_attention model. Additionally I cannot use other TensorRT versions as I am working with a Nvidia Jetson AGX Orin which has limits on what versions of TensorRT can be run. However, I found that using ONNX Runtime currently works the best for me. Thanks so much for the valuable insights though!

[yformer]

I have successfully converted EfficientTAM to TensorRT (from ONNX) thanks for the tips @okideal in #7 , but I am seeing a very strange issue with the image_encoder if I use trtexec with --fp16. It degrades the accuracy by A LOT when --fp16 is used. This is the first time I am seeing this with a model converted to --fp16 (normally the accuracy degrades, but by a negligible margin).

To confirm this was indeed because of --fp16, I have built it with the normal (fp32) and --fp16. I also set the LayerNorm nodes and their inputs/outputs to trt.DataType.FLOAT of the --fp16 model, but still no luck. It seems that if I set all of the layers containing blocks in the name to trt.DataType.FLOAT, it works fine (but that is basically the whole image encoder).

Here are some screenshots of the accuracy degradation (on a still image):

FP16 version FP32 version

@yformer Do you know what might be the issue with converting the image encoder weights to FP16 ? @okideal Did you encounter similar issues with your setup or did you just leave it in fp32?

@reynhardtstef, I did not see any issue with fp16 setting. Are you able to identify why this happens?

[rodneyML]

Hi @yformer , I believe the reason behind the degradation has to do with an underlying issue in TensorRT v10.7.0.23-1+cuda12.6. I found that using TensorRT v8.6.2 converts the correct image encoder, but ran into other issues (NVRTC compilation failed) with converting the memory attention model on that version of TensorRT. Currently I am using torch.compile to perform inference on all of the models which yielded a faster inference time than TensorRT --fp16. The only downside with torch.compile is that one needs to perform warmup and its not currently serializable. Note that I am performing all of this on a Nvidia Jetson AGX Orin, so these TensorRT versions might have different behaviours on other platforms (x86 Linux or Windows).