Fix CLIP memory leak causing 600-800MB accumulation per batch

[goodhee]

What does this PR do?

This PR fixes a critical memory leak in CLIP model's get_image_features and get_text_features methods that was causing significant memory accumulation during repeated inference calls.

Problem

When using CLIP models for batch processing (e.g., in microservices), repeated calls to get_image_features or get_text_features caused memory to increase by 600-800MB per batch, eventually leading to OOM crashes. This was particularly problematic in production environments processing multiple batches sequentially.

Root Cause

The issue was caused by holding references to full BaseModelOutputWithPooling objects when only the pooler_output tensor was needed. These large intermediate objects contained all hidden states, attentions, and other tensors that were not being properly garbage collected.

Solution

• Explicitly delete intermediate vision_outputs and text_outputs objects after extracting the required pooler_output
• This allows Python's garbage collector to immediately free the memory used by unused intermediate tensors
• No API changes - existing code continues to work exactly the same

Testing

• Created test script that verifies tensor count remains constant across multiple batches
• Confirmed fix resolves the memory accumulation issue
• No performance impact on inference speed

Fixes #41178

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline, Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@amyeroberts @qubvel (vision models)

---

[UPDATE] Further Memory Analysis Results

After deeper profiling investigation, we discovered this is actually a system-level memory fragmentation issue, matching known PyTorch/glibc behavior (refs: pytorch/pytorch#68114, pytorch/pytorch#114455).

Detailed Memory Profiling

• Model (get_image_features): 94% of leak (374.2 MiB)
• Processor (preprocess): 6% of leak (22.7 MiB)
• Total: ~396.9 MiB per batch

Recommended Application-Level Solution

Users experiencing significant memory issues should handle memory management at the application level:

def trim_memory():
    try:
        import ctypes
        libc = ctypes.CDLL("libc.so.6")
        return libc.malloc_trim(0)
    except:
        return False

# Use after batch processing
features = model.get_image_features(**inputs)
del inputs, features
gc.collect()
trim_memory()

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: aimv2, clip, metaclip_2

[ydshieh]

Thank you for this PR @eun2ce . I would like @Adefey to confirm this solve the issue #41178.

The need of adding these del statements is a bit unsatisfying. But if this is the only way to fix (i.e. not possible to fix in user code), we could go for it.

[goodhee]

@ydshieh After reviewing my PR and running additional memory profiling, I need to revise my initial approach.

Current PR Review

My current fix:

# Extract pooled output and immediately delete the full output to free memory
pooled_output = text_outputs.pooler_output
del text_outputs  # Explicitly delete to help with memory management

While this helps with immediate Python object cleanup, I found it's not addressing the root cause.

Memory Profiling Results

I created a test script to isolate the issue:

Model Only (get_image_features):

Line 71: image_features = model.get_image_features(**inputs) → 514.2 MiB leak per batch

Processor Only (preprocess):

Line 99: inputs = processor(images=images, return_tensors="pt", padding=True) → 74.0 MiB leak per batch

Full Pipeline:

• Model: 374.2 MiB leak (94%)
• Processor: 22.7 MiB leak (6%)
• Total: 396.9 MiB per batch

Root Cause

After this analysis, I realize this is actually a system-level memory fragmentation issue, not a transformers library bug. This matches known PyTorch/glibc behavior (refs: pytorch/pytorch#68114, pytorch/pytorch#114455).

Recommendation

I suggest we:

1. Close this PR as the current approach isn't the right solution
2. Document this as a known system-level issue
3. Recommend users handle memory management at the application level:

def trim_memory():
    try:
        import ctypes
        libc = ctypes.CDLL("libc.so.6")
        return libc.malloc_trim(0)
    except:
        return False

# Use after batch processing
features = model.get_image_features(**inputs)
del inputs, features
gc.collect()
trim_memory()

I've attached my full analysis script and results for reference. Let me know if you'd like me to update the documentation instead to help users handle this properly at the application level.

[ydshieh]

Thank you a lot @eun2ce for taking a deeper analysis.

For the documentation, the discussions in this PR is already a good resource, we don't need to add a documentation within the codebase (which is a bit off-topic). But it would be nice if you can add [update] ... (what you mentioned in the last comment) to the PR description.

@Adefey, maybe you could try what @eun2ce suggests?

[goodhee]

I’ve updated the PR description with the details from my last comment. Please let me know if anything else should be added or clarified.