Add Encoder-only Mask Transformer (EoMT) (except integration test(s), pre- and post- processing and training code)

[tommiekerssies]

See #37171

As stated in the title, the integration test is still needed. The pre- and post- processing and training code is preferably for a later PR.

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the Ready for review button (at the bottom of the PR page). This will assign reviewers and trigger CI.

[Rocketknight1]

cc @NielsRogge @qubvel

[tommiekerssies]

Unfortunately, I do not have more time to work on this in the coming days. Not sure if this can already be reviewed/merged?

[qubvel]

Hey @tommiekerssies, thanks a lot for opening the PR!

Unfortunately, it's not quite in a state where we can merge it into Transformers just yet. To move forward, the model should at least:

1. Be usable end-to-end — including both pre- and post-processing (it looks like Mask2FormerImageProcessor might be compatible?).

2. Follow Transformers conventions: clear docstrings, use of common modules, no one-letter variable names (you'll probably want to make use of the modular Transformers structure for this).

3. Include a conversion script, common model tests, and integration tests.

Totally understand if you don’t have the bandwidth in the coming days — we’d love for you to pick it up again later if possible. Alternatively, we can add the “contribution welcome” label in case someone from the community wants to continue your work.

In any case, it’s a great model and we’d be happy to see it land in Transformers!

[tommiekerssies]

Hey @tommiekerssies, thanks a lot for opening the PR!

Unfortunately, it's not quite in a state where we can merge it into Transformers just yet. To move forward, the model should at least:

1. Be usable end-to-end — including both pre- and post-processing (it looks like Mask2FormerImageProcessor might be compatible?).
2. Follow Transformers conventions: clear docstrings, use of common modules, no one-letter variable names (you'll probably want to make use of the modular Transformers structure for this).
3. Include a conversion script, common model tests, and integration tests.

Totally understand if you don’t have the bandwidth in the coming days — we’d love for you to pick it up again later if possible. Alternatively, we can add the “contribution welcome” label in case someone from the community wants to continue your work.

In any case, it’s a great model and we’d be happy to see it land in Transformers!

Thanks for the feedback, much appreciated.

Let’s go ahead and add the “contribution welcome” label for now. If someone from the community is interested in picking this up, they can refer to our implementation at https://github.com/tue-mps/eomt.

Regarding pre- and post-processing: all our checkpoints on Hugging Face are based on DINOv2, and the model expects ImageNet normalization.
For panoptic and instance segmentation, we use padded inference: the longer side is resized to the target input size, the shorter side is zero-padded to form a square of shape (image_size, image_size), and normalization is applied afterward.
For semantic segmentation, we use sliding-window inference: the shorter side is resized to the target input size, the image is split into (potentially overlapping) crops of (image_size, image_size), and each crop is normalized individually.

After the forward pass, the logits are mapped back to the original image size. Mask2Former post-processing can then be reused to convert the mask and class logits into final predictions for each task.

Happy to support anyone who wants to continue the integration. I may also return to it myself if time allows.

[yaswanth19]

Hey @tommiekerssies, We can close this PR 😄