Fuyu Finetuning Example by ncoop57 · Pull Request #26997 · huggingface/transformers

ncoop57 · 2023-10-22T23:49:47Z

What does this PR do?

This PR is about contributing an example for finetuning the recent Adept Fuyu model architecture. It adds the ability to calculate the loss for the model as well as add a new folder for image-text examples. It is currently in DRAFT model as it still has a bug due to the data collator padding and an easy way of performing evaluation. I'm opening the PR in this state so that I can get help.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

@molbap would love some help since you initially created the model!

Models:

text models: @ArthurZucker and @younesbelkada
vision models: @amyeroberts

Library:

tokenizers: @ArthurZucker
trainer: @muellerzr and @pacman100

ncoop57 · 2023-10-22T23:52:09Z

The issue I am getting right now is that I am unable to pad the image patches correct it seems for batch training. The image processor doesn't seem to be doing it correct.

ncoop57 · 2023-10-23T00:05:02Z

I thought it would be as simple as modifying variable_size var to false: https://github.com/huggingface/transformers/blob/main/src/transformers/models/fuyu/processing_fuyu.py#L508C1-L508C37

And that seems to patch the image patches correct, but the image_patches_indices var doesn't seem to be updated properly

Junemengyuan · 2023-10-23T01:07:13Z

The issue I am getting right now is that I am unable to pad the image patches correct it seems for batch training. The image processor doesn't seem to be doing it correct.

Yeah, the current image processor seems only outputs batch-size=1 result

ArthurZucker · 2023-10-23T10:21:30Z

Hey! Thanks a lot for wanting to contribute! We usually share links to training scripts like in the ressource section (like this one), but we don't add them to the repo for every new model! @molbap is working on a fix for the image processor see #27007

ncoop57 · 2023-10-23T19:56:23Z

Heyya @ArthurZucker ! In that case, should I close this PR or should this PR be the first of multi-modal model training examples? Seems like the space is getting super interesting and so it might be nice for people to see example scripts in the official repo for huggingface

…uyu-example

gagan3012 · 2023-11-04T22:19:15Z

Hi Is this working now?

RefractAI · 2023-11-05T17:56:31Z

This is working for me, but only with batch size 1. Using 4-bit Lora training this does fit on a 24GB GPU.

Using higher than batch 1 gives the error I reported in: #27255

molbap · 2023-11-07T15:17:10Z

Hi @ncoop57! It was mentioned before not sure if this PR should be merged here but it's a great resource. We don't have official training scripts, but we updated the FuyuProcessor has been updated with batching + attention masks with left-padding, see https://github.com/huggingface/transformers/blob/88832c01c8a962b653874c4ce4ed8df5783ac5cd/src/transformers/models/fuyu/processing_fuyu.py#L335C1-L383C1. Is that factored in for your loss calculation? I'm not 100% sure about the code but I think it would need to be adjusted a bit, such as

def collate_fn(examples):
    texts = [e[text_column_name] for e in examples]
    images = [e[image_column_name] for e in examples]
    output = processor(
        text=texts,
        images=images,
        padding="max_length",
        truncation=True
    )
    first_non_padded = (output["input_ids"] != tokenizer.pad_token_id).nonzero(as_tuple=False)
    first_non_padded_indices = first_non_padded[:, 1].view(-1, 1)
    labels = torch.full_like(output["input_ids"], -100)
    for i, index in enumerate(first_non_padded_indices):
        labels[i, index:] = output["input_ids"][i, index:]
    output["labels"] = labels
    return output

let me know what you think!

amyeroberts · 2023-11-07T15:52:57Z

@molbap @ArthurZucker I would be pro adding a finetuning script for Fuyu to our examples because we yet have an example for this modality / task.

ncoop57 · 2023-11-08T02:56:29Z

Super happy to see more excitement around this example script! Thanks for the loss pointer, I'm looking into it. I had the same error as @Carolinabanana mentioned but only when using the Trainer object, not when doing it myself. I did try to account for padding/image patches because I don't think we want to train on the prediction of the image patch indices, but I did forget to take into account the left padding!

infamix · 2023-11-14T00:05:11Z

The script doesn't support VQA, does it?

cysmnl · 2023-11-22T00:19:13Z

Can it fit in a 80GB card with bfloat16 for tuning? I faced a CUDA memory issue when running run_fuyu_no_trainer.py on a single A100 card.

zhaoyukoon · 2023-11-22T09:45:57Z

Can anyone provide a train script? Thanks!

amyeroberts · 2023-12-19T21:13:02Z

Hi @ncoop57 Any update on the script and readiness for PR review? Already looks very good! Let us know if you need any help adding it to the library

ncoop57 · 2023-12-21T16:48:49Z

AH sadly haven't had a chance to look back into this. Luckily Ill be off all next week so gonna add this to my list to complete and will reach out with help that I need, ty!

z3ugma · 2023-12-24T20:24:12Z

Thanks @ncoop57 ! I would love an example training script to try finetuning Fuyu-8B for a domain specific image captioning dataset

ncoop57 · 2024-01-14T19:55:37Z

@amyeroberts think I got it fully in and ready for review. @molbap if you have some time, I'd love for you to checkout how I am handling the left padding:

def collate_fn(examples):
        texts = [e[text_column_name] for e in examples]
        images = [e[image_column_name] for e in examples]
        output = processor(
            text=texts, images=images, padding="max_length", truncation=True
        )
        position = (output["input_ids"] == tokenizer.vocab["<s>"]).nonzero(
            as_tuple=True
        )[0][
            0
        ]  # This gets the index of the first '1' in the tensor by passing the left padding
        output["labels"] = torch.full_like(
            output["input_ids"], -100
        )  # This creates a tensor filled with -100
        output["labels"][position:] = output["input_ids"][position:]
        return output

Let me know what else is needed to get this merged in 🤓

ncoop57 · 2024-01-14T19:56:56Z

I wasn't able to get the Trainer working so right now this is just with a standard training loop and uses accelerate to handle all the FSDP stuff

amyeroberts · 2024-01-15T15:51:32Z

@ncoop57 Great! For the quality checks, make sure toe run make fixup and resolve any issues it flags locally and push those changes. This should make the CI green. For example, I can see from the recent CI theres a few objects imported by never used in the script.

ncoop57 · 2024-01-15T23:19:08Z

@amyeroberts looks like I got all the CI stuff working 🤓 lemme know if any other stuff are needed

ncoop57 · 2024-01-15T23:20:36Z

maybe one tweak would be renaming the folder to "visual-language-modeling"?

ncoop57 · 2024-01-22T15:28:27Z

@amyeroberts any updates on this?

amyeroberts

Thanks for all the work adding this!

Overall looking good. Made some general comments on the structure. There's some places which have been copied across from other scripts but need adapting to this one.

Before we can publish this script, we'll need a demonstration of a model which has been trained for a few epochs and pushed to the hub successfully. If/when you have thant, could you share in the PR description?

amyeroberts · 2024-01-15T18:17:49Z

@@ -0,0 +1,51 @@
+<!---
+Copyright 2020 The HuggingFace Team and Nathan Cooper. All rights reserved.


Suggested change

Copyright 2020 The HuggingFace Team and Nathan Cooper. All rights reserved.

Copyright 2024 The HuggingFace Team and Nathan Cooper. All rights reserved.

amyeroberts · 2024-01-15T18:18:35Z

@@ -0,0 +1,712 @@
+#!/usr/bin/env python
+# coding=utf-8
+# Copyright 2020 The HuggingFace Inc. team. All rights reserved.


Suggested change

# Copyright 2020 The HuggingFace Inc. team. All rights reserved.

# Copyright 2024 The HuggingFace Inc. team. All rights reserved.

amyeroberts · 2024-01-18T13:50:30Z

Lets undo the style changes here - we prefer to keep things on one line as much as possible.

For some of the lines, adding an extra comma on the last element will cause the split i.e. (a, b, c,) will be split across three lines whereas (a, b, c) won't

amyeroberts · 2024-01-18T13:50:45Z

@@ -0,0 +1,778 @@
+#!/usr/bin/env python
+# coding=utf-8
+# Copyright 2021 The HuggingFace Inc. team and Nathan Cooper. All rights reserved.


Suggested change

# Copyright 2021 The HuggingFace Inc. team and Nathan Cooper. All rights reserved.

# Copyright 2024 The HuggingFace Inc. team and Nathan Cooper. All rights reserved.

amyeroberts · 2024-01-18T13:51:00Z

+Fine-tuning the library models for causal language modeling (GPT, GPT-2, CTRL, ...)
+on a text file or a dataset without using HuggingFace Trainer.
+
+Here is the full list of checkpoints on the hub that can be fine-tuned by this script:
+https://huggingface.co/models?filter=text-generation


To be updated

amyeroberts · 2024-01-23T18:38:37Z

+        valid_images = []
+        for i in range(0, len(examples[image_column_name])):
+            try:
+                processor(text="test", images=[examples[image_column_name][i]])


Why pass them through the processor here? Trying to open them with Image.open should be enough to check. A failure here doesn't mean the image itself is problematic.

amyeroberts · 2024-01-23T18:40:25Z

+        for i in range(0, len(examples[image_column_name])):
+            try:
+                processor(text="test", images=[examples[image_column_name][i]])


range automatically starts on the 0th index

You can iterate directly on the dataset

Suggested change

for i in range(0, len(examples[image_column_name])):

try:

processor(text="test", images=[examples[image_column_name][i]])

for image in examples[image_column_name]:

try:

Image.open(image)

amyeroberts · 2024-01-23T18:42:51Z

+            batched=True,
+        )
+        # resize images
+        resize = Resize((224, 224))


The size here should be taken from the image processor config:

size_dict = processor.image_processor.size size = (size_dict["height"], size_dict["width"]) resize = Resize(size)

Actually figured out the processor already handles resizing, so need to do something like:

processor = FuyuProcessor.from_pretrained(args.model_name_or_path, size = {"height": 480, "width": 480})

Yes, but it doesn't do augmentation. I'd recommend not using the processor to process the images.

amyeroberts · 2024-01-23T18:44:09Z

+            batched=True,
+        )
+        # resize images
+        resize = Resize((224, 224))


Why is this the only transformation applied? I'd expect some augmentation in a training script. You can look at the image classification script as an example.

amyeroberts · 2024-01-23T18:46:37Z

+        )
+
+    if args.block_size is None:
+        block_size = tokenizer.model_max_length


This should use the processor's tokenizer

ncoop57 · 2024-01-23T20:11:22Z

Awesome, thanks for the review and recommendations! 🤓 I've been working on finetuning using the https://huggingface.co/datasets/HuggingFaceM4/WebSight dataset, so once I have the model trained I'll make sure to add it here and to the README!

github-actions · 2024-02-17T08:05:21Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

ncoop57 and others added 5 commits October 21, 2023 11:24

add ability to pass labels to fuyu model for calculating loss

d08314c

update docstring to have labels descript

42b46db

Add a image-text causal visual language model training script using fuyu

a554eae

update readme and cvlm to run on the example dataset

b809e4c

Fix issues up to data collator

e54cb77

ncoop57 marked this pull request as draft October 22, 2023 23:49

Fix overwriting image_processor

3c90c76

dongxiaolong mentioned this pull request Oct 23, 2023

Support Fuyu-8B axolotl-ai-cloud/axolotl#777

Closed

5 tasks

ncoop57 and others added 2 commits November 3, 2023 23:30

Merge branch 'huggingface:main' into fuyu-example

06b9b90

Merge branch 'fuyu-example' of github.com:ncoop57/transformers into f…

d73f9f6

…uyu-example

Got training working with loss going down

32eb51e

molbap mentioned this pull request Nov 7, 2023

Fuyu training fails with a padding error for the same inputs as the test case #27255

Closed

4 tasks

jzhang38 mentioned this pull request Nov 21, 2023

[Fuyu and Persimmon] Add FA2 and fused kernels #27622

Closed

8 tasks

huggingface deleted a comment from github-actions Bot Dec 17, 2023

ncoop57 and others added 4 commits January 14, 2024 12:56

Merge branch 'huggingface:main' into fuyu-example

a9028e1

Refactor Fuyu model code

af3a2c6

Add torchvision to requirements.txt

366809a

Refactor image-text training script for Fuyu model

445cf5f

ncoop57 marked this pull request as ready for review January 14, 2024 19:55

Update copyright information

0d4be88

ncoop57 added 3 commits January 15, 2024 22:41

remove old code

bea57a0

Refactor code to improve performance and readability

ca54bba

Improve code readability

e0bc976

amyeroberts reviewed Jan 23, 2024

View reviewed changes

github-actions Bot closed this Feb 25, 2024

		@@ -0,0 +1,51 @@
		<!---
		Copyright 2020 The HuggingFace Team and Nathan Cooper. All rights reserved.

	Copyright 2020 The HuggingFace Team and Nathan Cooper. All rights reserved.
	Copyright 2024 The HuggingFace Team and Nathan Cooper. All rights reserved.

	# Copyright 2020 The HuggingFace Inc. team. All rights reserved.
	# Copyright 2024 The HuggingFace Inc. team. All rights reserved.

	# Copyright 2021 The HuggingFace Inc. team and Nathan Cooper. All rights reserved.
	# Copyright 2024 The HuggingFace Inc. team and Nathan Cooper. All rights reserved.

Uh oh!

Conversation

ncoop57 commented Oct 22, 2023

What does this PR do?

Before submitting

Who can review?

Uh oh!

ncoop57 commented Oct 22, 2023

Uh oh!

ncoop57 commented Oct 23, 2023

Uh oh!

Junemengyuan commented Oct 23, 2023

Uh oh!

ArthurZucker commented Oct 23, 2023

Uh oh!

ncoop57 commented Oct 23, 2023

Uh oh!

gagan3012 commented Nov 4, 2023

Uh oh!

RefractAI commented Nov 5, 2023

Uh oh!

molbap commented Nov 7, 2023

Uh oh!

amyeroberts commented Nov 7, 2023

Uh oh!

ncoop57 commented Nov 8, 2023

Uh oh!

infamix commented Nov 14, 2023

Uh oh!

cysmnl commented Nov 22, 2023

Uh oh!

zhaoyukoon commented Nov 22, 2023

Uh oh!

amyeroberts commented Dec 19, 2023

Uh oh!

ncoop57 commented Dec 21, 2023

Uh oh!

z3ugma commented Dec 24, 2023

Uh oh!

ncoop57 commented Jan 14, 2024

Uh oh!

ncoop57 commented Jan 14, 2024

Uh oh!

amyeroberts commented Jan 15, 2024

Uh oh!

ncoop57 commented Jan 15, 2024

Uh oh!

ncoop57 commented Jan 15, 2024

Uh oh!

ncoop57 commented Jan 22, 2024

Uh oh!

amyeroberts left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ncoop57 commented Jan 23, 2024

amyeroberts left a comment •

edited

Loading