Add Relation DETR by xiuqhou · Pull Request #34900 · huggingface/transformers

xiuqhou · 2024-11-24T14:08:34Z

What does this PR do?

This PR adds Relation-DETR as introduced in Relation DETR: Exploring Explicit Position Relation Prior for Object Detection. Checkpoint for Relation-DETR (ResNet50) converted from original repo https://github.com/xiuqhou/Relation-DETR has been uploaded to https://huggingface.co/xiuqhou/relation-detr-resnet50

Related issues in original repo:
xiuqhou/Relation-DETR#25
xiuqhou/Relation-DETR#21

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

TODO:

Make more checkpoints with Swin-L and Focal-L backbones available on HF.
Update the document about Relation-DETR.

Who can review?

@amyeroberts @qubvel

qubvel · 2024-11-25T12:00:53Z

Hi @xiuqhou! Congratulations on the paper, awesome work! And thanks for working on transformers implementation! Feel free to ping me when it's ready for review or if you have any questions!

xiuqhou · 2024-11-28T07:55:49Z

Hi @qubvel Thanks for your support! The code is now ready for review—I'd greatly appreciate it if you could take a look and share your feedback. Please let me know if there’s anything that needs improvement.

…s_attentions

xiuqhou · 2025-02-15T07:46:13Z

@ArthurZucker @qubvel @stevhliu Sorry for the late reply! Thanks for your detailed review and suggestions. I have pushed commits according to the reviews except #34900 (comment) about the modular style.
I have finished converting the code to the modular style but met a bug #36208 in utils/modular_model_converter.py.
Once it is fixed, I will commit this part of code.

CI errors seem not related to changes.

qubvel · 2025-02-15T11:27:48Z

Hey @xiuqhou thanks for working on the PR and opening the issue!

I think we can use the following workaround while the issue is not fixed.
Just override the function in modular file to avoid local imports (we can leave numpy and torch only)

def get_numpy_to_framework_fn(arr) -> Callable:
    """
    Returns a function that converts a numpy array to the framework of the input array.

    Args:
        arr (`np.ndarray`): The array to convert.
    """
    if isinstance(arr, np.ndarray):
        return np.array
    elif is_torch_available() and is_torch_tensor(arr):
        return torch.tensor
    else:
        raise RuntimeError("...")

Let me know if that works for you!

xiuqhou · 2025-02-15T12:35:03Z

@qubvel thanks for your suggestion!

I tried to overwrite the function in modular_relation_detr.py but the error still exists. If I override the original function inherited from src/transformers/models/detr/image_processing_detr.py, it will work. But this will change the code of other models, which is beyond the code scope of this PR.

I have a question about whether this function is necessary. I searched all the code of transformers, and found that this function has been defined many times in DETR related models, but it has never been called. Can we remove this unused function?

qubvel · 2025-02-15T13:02:16Z

Oh, it might be a redundant function that was copied from one model to another 😄 In that case we can open a separate PR to remove it everywhere

daniel-bogdoll · 2025-02-26T19:14:11Z

Hey @xiuqhou , thanks so much for your work so far! Since this will be (most likely) the new best DETR model on HF, I wanted to ask if you think that a merge is likely in the upcoming months? I saw that the issue you mentioned #36208 was fixed by #36279

xiuqhou · 2025-03-20T04:16:40Z

Hi @daniel-bogdoll Sorry for the late reply! As #36208 has been fixed, I have committed code about the modular style for relation-detr. Now I think the PR is fully prepared for review or merge. Please let me know if there are any remaining improvements needed before merging.

ArthurZucker

Great work isolating the differences! 🤗
Need a comment from @yonigozlan but good otherwise!

ArthurZucker · 2025-04-11T15:36:39Z

+        do_resize = self.do_resize if do_resize is None else do_resize
+        size = self.size if size is None else size
+        size = get_size_dict(size=size, default_to_square=False)
+        resample = self.resample if resample is None else resample
+        do_rescale = self.do_rescale if do_rescale is None else do_rescale
+        rescale_factor = self.rescale_factor if rescale_factor is None else rescale_factor
+        do_normalize = self.do_normalize if do_normalize is None else do_normalize
+        image_mean = self.image_mean if image_mean is None else image_mean
+        image_std = self.image_std if image_std is None else image_std
+        do_convert_annotations = (
+            self.do_convert_annotations if do_convert_annotations is None else do_convert_annotations
+        )
+        do_pad = self.do_pad if do_pad is None else do_pad
+        pad_size = self.pad_size if pad_size is None else pad_size
+        format = self.format if format is None else format


cc @yonigozlan we don't need these anymore do we?

This is for the slow image processor so I guess we do...
However @xiuqhou looking at the code for the fast image processor, we can definitely use modular for that too!
Also you should know that we are now prioritizing the use of fast image processors vs slow, so if you could save the checkpoints of this model with a fast image processor that'd be great!

vncntcmnn · 2025-06-19T16:53:43Z

Hey,

Really promising model and I was wondering if there are any updates on getting this PR merged ? 👀

No rush, just curious about the status.

Thanks !

xiuqhou changed the title ~~Add Relation DETR~~ [WIP] Add Relation DETR Nov 24, 2024

xiuqhou marked this pull request as draft November 24, 2024 15:16

xiuqhou force-pushed the add_relation_detr branch from 7867221 to 1f0465c Compare November 25, 2024 07:59

xiuqhou marked this pull request as ready for review November 25, 2024 08:37

xiuqhou changed the title ~~[WIP] Add Relation DETR~~ Add Relation DETR Nov 25, 2024

xiuqhou force-pushed the add_relation_detr branch from 37959ac to d114fc7 Compare November 25, 2024 08:53

qubvel added New model Vision run-slow labels Nov 25, 2024

xiuqhou force-pushed the add_relation_detr branch 5 times, most recently from 14308cf to ce63725 Compare November 28, 2024 07:42

qubvel self-requested a review November 28, 2024 09:19