Added example script for InternVL by quic-dhirajku · Pull Request #269 · quic/efficient-transformers

quic-dhirajku · 2025-02-14T11:53:02Z

No description provided.

quic-amitraj · 2025-02-17T08:00:34Z

+
+    ## STEP 2 -- EXPORT & COMPILE THE MODEL
+
+    model.compile(num_cores=14, num_devices=4, mxfp6_matmul=True)


Please update the num_cores to 16. We should always by default set the num_cores to 16 only.

Pass num_patches here that you will be using at the time of processor.

Updated, thanks.

quic-amitraj · 2025-02-17T08:02:26Z

+    ## STEP 3 -- SETUP THE PROCESSOR
+
+    # InternVL doesn't have an AutoProcessor yet, so we will use our own processor class "InternProcessor"
+    qeff_pt_model = model.model


Is there any need of this line?

Removed accrodingly.

quic-amitraj · 2025-02-17T08:30:06Z

+    pixel_values = internProcessor.load_image(image, max_num=12)
+    question = "<image>\n" + prompt
+    query = internProcessor(pixel_values, question, messages, roles)
+    inputs = tokenizer(query, return_tensors="pt", padding="max_length", max_length=3840, padding_side="right")


How you are setting max_length to 3840 if you are not setting the prefil_seq_length at the time of compilation?
There is two thing you can do-

Pass prefill seq as a arg to compile, so you will know the max_length value what to use at the time of tokenizer.

Else don't pass anything here, by default padding will be done till the 3840 as its the default value at the time of compilation.

Added prefill_seq_len as an added arguement now, thanks.

quic-amitraj · 2025-02-17T08:49:48Z

+
+    # Resizing the image to have num_crops=13
+    # We can fix a different size and the compile the model with appropriate `num_crops`
+    image = image.resize((1000, 747))


@quic-dhirajku How user will know which shape is best to resize to achive exact num_crops that we have used at the time of compilation? It should be taken care inside the processor as we are using own processor and by just passing the num_patches (As its already known because we have used at the time of compilation) and image (Same as you are doing currently) it should resize the image in function dynamic_preprocess. Please modify the function dynamic_preprocess to incorporate this logic. Let me know if you still have doubt.

Already discussed, keeping this as it is for now.

quic-amitraj · 2025-02-17T12:18:28Z

+
+    ## STEP 2 -- EXPORT & COMPILE THE MODEL
+
+    model.compile(num_cores=14, num_devices=4, mxfp6_matmul=True)


Pass num_patches here that you will be using at the time of processor.

quic-amitraj · 2025-02-17T12:19:29Z

+    image = image.resize((1000, 747))
+
+    # preprocess the resized image
+    pixel_values = internProcessor.load_image(image, max_num=12)


Change max_num to num_patches.

ochougul · 2025-02-17T13:02:53Z

+
+
+# Processor class for InternVL models
+class InternProcessor:


write a docstring explaining this is taken from model card on hf and summarize the changes done to original code.

Done, thanks

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com>

ochougul · 2025-02-20T08:10:37Z

 ---

 *Latest news* :fire: <br>
+- [02/2025] [VLMs support](https://github.com/quic/efficient-transformers/pull/267) added for the models [InternVL-1B](https://huggingface.co/OpenGVLab/InternVL2_5-1B), [Llava](https://huggingface.co/llava-hf/llava-1.5-7b-hf) (Single QPC) and [Mllama](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct) (Single + Dual QPCs)


Single/dual qpc is internal terminology. We expose this via kv_offload variable.
Remove reference to single/dual qpc term

Changes incorporated

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com>

ochougul · 2025-02-27T08:52:28Z

+    image = Image.open(BytesIO(img.content)).convert("RGB")
+
+    # Resizing the image to have num_crops=13
+    # We can fix a different size and the compile the model with appropriate `num_crops`


num_patches

ochougul · 2025-02-27T08:53:46Z

rename to text_to_image_inference.py

ochougul · 2025-02-27T08:55:18Z

+    InternVL model only has an AutoTokenizer so this class performs the processing tasks similar to an AutoProcessor.
+    The methods used here are borrowed from the original InternVL modelling files.
+    "https://huggingface.co/OpenGVLab/InternVL2_5-1B/"
+


remove line

remove which line?
I didn't get it.

ochougul · 2025-02-27T08:55:59Z

+        query = get_prompt(messages)
+        if verbose and pixel_values is not None:
+            image_bs = pixel_values.shape[0]
+            print(f"dynamic ViT batch size: {image_bs}")


why? use logger.info if needed.
Avoid using print statements

ochougul · 2025-02-27T08:57:29Z

+    pixel_values = internProcessor.load_image(image, max_num=12)
+    question = "<image>\n" + prompt
+    query = internProcessor(pixel_values, question, messages, roles)
+    # inputs = tokenizer(query, return_tensors="pt", padding="max_length", padding_side="right")


commented code?

ochougul · 2025-02-27T08:58:08Z

+    # Inputs for the model
+    prompt = "Please describe the image in detail."
+    image_url = "https://image.slidesharecdn.com/azureintroduction-191206101932/75/Introduction-to-Microsoft-Azure-Cloud-1-2048.jpg"
+    prefill_seq_len = 3840


Add the explanation of why this is put as 3840

ochougul · 2025-02-27T08:59:04Z

+    ## STEP - 1 Load the Processor and Model
+
+    processor = AutoProcessor.from_pretrained(model_name, token=token)
+    # `kv_offload` is used to decide if we wish to run Single QPC or 2 QPC setup


single/ dual qpc is internal term. can you explain properly here

ochougul · 2025-02-27T09:00:06Z

+        img_size=img_size,
+        num_cores=num_cores,
+        num_devices=num_devices,
+    )


What's the expected output for this model? can you put that here?

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com>

ochougul · 2025-02-28T10:27:20Z

+    ctx_len = 512
+    img_size = 560
+    num_cores = 16
+    num_devices = 1


can you please define generation_len here and pass it to run_model ?
everything else LGTM.
Will merge after this change

sure, will do, thanks.

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com>

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com> Signed-off-by: Hem Agnihotri <quic_hemagnih@quicinc.com>

quic-dhirajku requested review from ochougul and quic-rishinr as code owners February 14, 2025 11:53

quic-dhirajku requested a review from vbaddi February 14, 2025 11:55

quic-amitraj self-requested a review February 17, 2025 07:58

quic-amitraj requested changes Feb 17, 2025

View reviewed changes

quic-amitraj assigned quic-dhirajku Feb 17, 2025

ochougul reviewed Feb 17, 2025

View reviewed changes

quic-dhirajku added 3 commits February 20, 2025 05:28

Added example script for InternVL

0292e62

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com>

Addressed comments

8911640

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com>

Added docstrings, updated announcements in readme file

d6b19bd

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com>

quic-dhirajku force-pushed the intern_example branch from 25c7256 to d6b19bd Compare February 20, 2025 05:59

ochougul requested changes Feb 20, 2025

View reviewed changes

quic-amitraj marked this pull request as draft February 21, 2025 06:19

quic-dhirajku added 3 commits February 21, 2025 06:44

Added readme for example script, addressed further comments

398a13c

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com>

Added example vlm script, WIP

aadd3de

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com>

vlm inference example script finalized changes

43df6d5

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com>

quic-dhirajku marked this pull request as ready for review February 26, 2025 09:12

ochougul requested changes Feb 27, 2025

View reviewed changes

Addressed comments, minor edits in the script

8de3a62

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com>

quic-rishinr requested changes Feb 28, 2025

View reviewed changes

Comment thread examples/intern_example/internvl_inference.py Outdated

Comment thread examples/intern_example/internvl_inference.py

quic-dhirajku added 2 commits February 28, 2025 08:28

addressed comments

78db128

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com>

minor explanation edits

56f2b3e

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com>

quic-rishinr approved these changes Feb 28, 2025

View reviewed changes

ochougul reviewed Feb 28, 2025

View reviewed changes

added generation_len as a parameter

bedc855

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com>

ochougul approved these changes Feb 28, 2025

View reviewed changes

ochougul merged commit 040dab4 into quic:main Feb 28, 2025

quic-hemagnih pushed a commit to quic-hemagnih/efficient-transformers that referenced this pull request Mar 12, 2025

Added example script for InternVL (quic#269)

d5c5179

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com> Signed-off-by: Hem Agnihotri <quic_hemagnih@quicinc.com>

quic-hemagnih pushed a commit to quic-hemagnih/efficient-transformers that referenced this pull request Mar 12, 2025

Added example script for InternVL (quic#269)

bbfc4de

Signed-off-by: quic-dhirajku <quic_dhirajku@quicinc.com> Signed-off-by: Hem Agnihotri <quic_hemagnih@quicinc.com>


		## STEP 2 -- EXPORT & COMPILE THE MODEL

		model.compile(num_cores=14, num_devices=4, mxfp6_matmul=True)

Uh oh!

Conversation

quic-dhirajku commented Feb 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

49

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants