Fixed VitDet for non-squre Images by cjfghk5697 · Pull Request #35969 · huggingface/transformers

cjfghk5697 · 2025-01-30T09:40:18Z

What does this PR do?

Fixes #35280 (issue)

The VitDet model doesn't support tuple-type inputs for image_size and patch_size. I’ve fixed it so that tuples can now be used.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ x ] Did you read the contributor guideline,
Pull Request section?
[ x ] Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
[ x ] Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@qubvel @amyeroberts

qubvel · 2025-01-31T09:39:19Z

Hey @cjfghk5697! Thanks for opening a PR 🤗 Feel free to ping me when it's ready for review

cjfghk5697 · 2025-02-04T04:43:29Z

@qubvel Thanks for your response! Would you like me to provide a mock code to check the tuple case?

qubvel

Thanks! Can you please add a test to tests/models/test_modeling_vitdet.py which allows passing non-square image through the model

qubvel · 2025-02-04T10:03:26Z

+        if isinstance(config.image_size, tuple) and isinstance(config.patch_size, tuple):
+            input_size = tuple(i // p for i, p in zip(config.image_size, config.patch_size))
+        elif isinstance(config.image_size, tuple) and not isinstance(config.patch_size, tuple):
+            input_size = tuple(i // config.patch_size for i in config.image_size)
+        elif not isinstance(config.image_size, tuple) and isinstance(config.patch_size, tuple):
+            input_size = tuple(config.image_size // p for p in config.patch_size)
+        else:
+            input_size = config.image_size // config.patch_size


Suggested change

if isinstance(config.image_size, tuple) and isinstance(config.patch_size, tuple):

input_size = tuple(i // p for i, p in zip(config.image_size, config.patch_size))

elif isinstance(config.image_size, tuple) and not isinstance(config.patch_size, tuple):

input_size = tuple(i // config.patch_size for i in config.image_size)

elif not isinstance(config.image_size, tuple) and isinstance(config.patch_size, tuple):

input_size = tuple(config.image_size // p for p in config.patch_size)

else:

input_size = config.image_size // config.patch_size

image_size = config.image_size

image_size = image_size if isinstance(image_size, (list, tuple)) else (image_size, image_size)

patch_size = config.patch_size

patch_size = patch_size if isinstance(patch_size, (list, tuple)) else (patch_size, patch_size)

input_size = (config.image_size[0] // config.patch_size[0], config.image_size[1] // config.patch_size[1])

Co-authored-by: Pavel Iakubovskii <qubvel@gmail.com>

…7/transformers into issue/vitDet-nonsquare

cjfghk5697 · 2025-02-04T10:43:43Z

@qubvel, thank you for reviewing it🤗. I've finished adding the test code. Could you please check it again?

qubvel

Thanks, to ensure CI is green, please run make modified_only_fixup to reformat the code 🤗

cjfghk5697 · 2025-02-06T10:07:19Z

@qubvel I’ve finished running make modified_only_fixup—thanks for your help!

qubvel · 2025-02-06T10:13:31Z

hm... something goes wrong, probably you have older/newer version of ruff? Can you please check and update accordingly?

cjfghk5697 · 2025-02-06T10:48:38Z

hm... something goes wrong, probably you have older/newer version of ruff? Can you please check and update accordingly?

I tried running it with ruff==0.5.1, but the results look really strange. Could you let me know which version you're using?

qubvel · 2025-02-06T11:29:00Z

It's ruff 0.5.1 in my env, it might be that you are running with no pyproject.toml configuration?

qubvel · 2025-02-06T11:30:18Z

It's better to delete 3 last commits, apply a modified fixup, and then force push

cjfghk5697 · 2025-02-07T05:55:32Z

@qubvel Sorry about that! I created a new environment and deleted some commits, which ended up modifying the fixup. Let me know if anything needs to be adjusted.

cjfghk5697 · 2025-02-13T08:15:44Z

@qubvel Can you review it again?

qubvel · 2025-02-17T18:21:21Z

Hey @cjfghk5697 can you please fix quality issues? most probably make modified_only_fixup will resolve the issue

cjfghk5697 · 2025-02-18T01:55:29Z

@qubvel Thank you for your help. I checked and discovered that there was an issue with my environment. I've applied the fix!🤗

qubvel · 2025-02-19T11:25:28Z

Hey @cjfghk5697, with the recent modification we have a bunch of tests failed, can you please check what should be fixed?

FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_attention_outputs - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_backbone - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_batching_equivalence - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_can_use_safetensors - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_correct_missing_keys - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_determinism - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_from_pretrained_no_checkpoint - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_gradient_checkpointing_backward_compatibility - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_gradient_checkpointing_enable_disable - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_hidden_states_output - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_initialization - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_inputs_embeds_matches_input_ids - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_load_save_without_tied_weights - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_model - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_model_get_set_embeddings - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_model_is_small - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_model_outputs_equivalence - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_model_weights_reload_no_missing_tied_weights - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_peft_gradient_checkpointing_enable_disable - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_retain_grad_hidden_states_attentions - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_save_load - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_save_load_fast_init_from_base - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_save_load_fast_init_to_base - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_save_load_keys_to_ignore_on_save - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_tie_model_weights - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_tied_weights_keys - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetModelTest::test_torch_save_load - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetBackboneTest::test_backbone_common_attributes - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetBackboneTest::test_backbone_outputs - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetBackboneTest::test_backbone_stage_selection - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetBackboneTest::test_channels - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetBackboneTest::test_create_from_modified_config - TypeError: 'int' object is not subscriptable
FAILED tests/models/vitdet/test_modeling_vitdet.py::VitDetBackboneTest::test_forward_signature - TypeError: 'int' object is not subscriptable

qubvel · 2025-02-19T11:27:57Z

I think I got why tests failed, I hope you don't mind I've pushed the fix 🤗

HuggingFaceDocBuilderDev · 2025-02-19T11:53:32Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

qubvel · 2025-02-19T12:08:41Z

cc @yonigozlan can you take a look

cjfghk5697 · 2025-02-19T12:29:25Z

@qubvel I sincerely appreciate your help in successfully wrapping up this task🤗

yonigozlan

Looks great to me @cjfghk5697 ! Thanks for making the model more flexible 🤗

yonigozlan · 2025-02-19T17:18:26Z


+    def test_non_square_image(self):
+        non_square_image_size = (32, 40)
+        patch_size = (2, 2)


I don't know too much about this model, but do we also want to test a non-square patch size?

yeah, maybe a bit out of scope for this PR, but should also work

cjfghk5697 added 4 commits January 30, 2025 17:04

size tuple

826495c

delete original input_size

737d3f2

use zip

44222e0

process the other case

89b6f59

cjfghk5697 changed the title ~~Issue/vit det nonsquare~~ Fixed VitDet for non-squre Images Jan 30, 2025

qubvel reviewed Feb 4, 2025

View reviewed changes

cjfghk5697 and others added 4 commits February 4, 2025 19:15

Update src/transformers/models/vitdet/modeling_vitdet.py

b7f02d7

Co-authored-by: Pavel Iakubovskii <qubvel@gmail.com>

[VITDET] Test non-square image

34b4dc4

Merge branch 'huggingface:main' into issue/vitDet-nonsquare

d971b12

Merge branch 'issue/vitDet-nonsquare' of https://github.com/cjfghk569…

713e610

…7/transformers into issue/vitDet-nonsquare

qubvel reviewed Feb 4, 2025

View reviewed changes

cjfghk5697 force-pushed the issue/vitDet-nonsquare branch from fb2998f to 713e610 Compare February 7, 2025 05:52

[Fix] Make Quality

3dd528d

qubvel and others added 2 commits February 13, 2025 11:40

Merge branch 'main' into issue/vitDet-nonsquare

45298b8

Merge branch 'main' into issue/vitDet-nonsquare

ce57b9d

make fix style

d2aa275

qubvel reviewed Feb 19, 2025

View reviewed changes

Comment thread src/transformers/models/vitdet/modeling_vitdet.py Outdated

Update src/transformers/models/vitdet/modeling_vitdet.py

a06f355

qubvel approved these changes Feb 19, 2025

View reviewed changes

qubvel requested a review from yonigozlan February 19, 2025 12:08

yonigozlan reviewed Feb 19, 2025

View reviewed changes

shethaadit approved these changes Feb 25, 2025

View reviewed changes

qubvel merged commit 9ebfda3 into huggingface:main Feb 25, 2025

qubvel added the Vision label Feb 25, 2025

Uh oh!

Conversation

cjfghk5697 commented Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Before submitting

Who can review?

Uh oh!

qubvel commented Jan 31, 2025

Uh oh!

cjfghk5697 commented Feb 4, 2025

Uh oh!

qubvel left a comment

Choose a reason for hiding this comment

Uh oh!

qubvel Feb 4, 2025

Choose a reason for hiding this comment

Uh oh!

cjfghk5697 commented Feb 4, 2025

Uh oh!

qubvel left a comment

Choose a reason for hiding this comment

Uh oh!

cjfghk5697 commented Feb 6, 2025

Uh oh!

qubvel commented Feb 6, 2025

Uh oh!

cjfghk5697 commented Feb 6, 2025

Uh oh!

qubvel commented Feb 6, 2025

Uh oh!

qubvel commented Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cjfghk5697 commented Feb 7, 2025

Uh oh!

cjfghk5697 commented Feb 13, 2025

Uh oh!

qubvel commented Feb 17, 2025

Uh oh!

cjfghk5697 commented Feb 18, 2025

Uh oh!

qubvel commented Feb 19, 2025

Uh oh!

Uh oh!

qubvel commented Feb 19, 2025

Uh oh!

HuggingFaceDocBuilderDev commented Feb 19, 2025

Uh oh!

qubvel commented Feb 19, 2025

Uh oh!

cjfghk5697 commented Feb 19, 2025

Uh oh!

yonigozlan left a comment

Choose a reason for hiding this comment

Uh oh!

yonigozlan Feb 19, 2025

Choose a reason for hiding this comment

Uh oh!

qubvel Feb 25, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cjfghk5697 commented Jan 30, 2025 •

edited

Loading

qubvel commented Feb 6, 2025 •

edited

Loading