Upgrade transformers to 4.48.0 for llama2 by selenayang888 · Pull Request #24302 · microsoft/onnxruntime

selenayang888 · 2025-04-03T23:08:16Z

Description

Upgrade Transformers to 4.48.0 for llama2, this version deprecated the old format of past_key_value, the current format is DynamicCache. So, we need to add patches to dynamo exporter in llama2.

Thanks to @xadupre who made the changes to add the patches to dynamo exporter, and implements patches to transformers 4.48.0 which don't export and convert dynamic_axes into dynamic shapes.

…llama2

…xadupre/llama

…grade-llama2

 import numpy as np
+import packaging.version as pv
 import torch
+import transformers


To fix the issue, we will:

Remove the from transformers import AutoConfig statement on line 29.

Update all references to AutoConfig in the code to use transformers.AutoConfig instead.

Ensure that the functionality remains unchanged.

This approach eliminates the redundant import while maintaining the same behavior.

+def _catch_produce_guards_and_solve_constraints(
+    previous_function: Callable,
+    fake_mode: "FakeTensorMode",
+    gm: "torch.fx.GraphModule",
+    dynamic_shapes: dict[str, Any] | tuple[Any] | list[Any] | None,
+    equalities_inputs: "EqualityConstraint",  # noqa: F821
+    original_signature: inspect.Signature,
+    _is_torch_jit_trace: bool = False,
+    verbose: int = 0,
+):


To fix the problem, we need to add explicit return statements at the end of the functions _catch_produce_guards_and_solve_constraints and patch__check_input_constraints_for_graph. This will ensure that the functions always return a value explicitly, making the code easier to read and understand.

For _catch_produce_guards_and_solve_constraints, we will add return None at the end of the function.

For patch__check_input_constraints_for_graph, we will also add return None at the end of the function.

+def patch__check_input_constraints_for_graph(
+    previous_function: Callable,
+    input_placeholders: list[torch.fx.Node],
+    flat_args_with_path,
+    range_constraints,
+    verbose: int = 0,
+) -> None:


To fix the problem, we need to add an explicit return statement at the end of the function patch__check_input_constraints_for_graph. This ensures that the function consistently returns None when no exception is raised, making the code easier to read and understand.

Add an explicit return None statement at the end of the function patch__check_input_constraints_for_graph.

This change should be made in the file onnxruntime/python/tools/transformers/models/torch_export_patches/patches/patch_torch.py.

+            # if config.print_specializations:
+            #    self.log.warning(
+            #         "Specializing %s to %s", self.var_to_sources[a][0].name(), tgt


To fix the problem, we should remove the commented-out code entirely. This will clean up the code and eliminate any potential confusion for future developers. If the logging functionality is still required, it should be properly implemented with a configuration check.

Remove the commented-out code on lines 304-308.

Ensure that the removal does not affect the existing functionality of the code.

+            # if input_ids.shape[1] == 0:
+            #     inputs_embeds = inputs_embeds[:, -cache_position.shape[0] :]
+            # else:
+            #     if cache_position[-1] >= input_ids.shape[1]:
+            #         input_ids = input_ids[:, -cache_position.shape[0] :]
+            #     else:
+            #         if input_ids.shape[1] != cache_position.shape[0]:
+            #             input_ids = input_ids[:, cache_position]


To fix the problem, we should remove the commented-out code entirely. This will make the code cleaner and less confusing for future developers. The functionality of the code will remain unchanged as the commented-out code is not executed.

…grade-llama2

github-actions