What version of transfomers _make_causal_mask was moved from modeling_clip.py

[bhosalems]

System Info

Name: transformers
Version: 4.28.0
Summary: State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow
Home-page: https://github.com/huggingface/transformers
Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors)
Author-email: transformers@huggingface.co
License: Apache 2.0 License
Location: anaconda3/envs/pathldm1/lib/python3.8/site-packages
Requires: tqdm, packaging, filelock, numpy, tokenizers, regex, huggingface-hub, pyyaml, requests
Required-by:

Who can help?

@ArthurZucker @younesbelkada

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Try to import
from transformers.models.clip.modeling_clip import _make_causal_mask, _expand_mask

Expected behavior

It should import both functions without errors. I see that several times the code has been refactored While I can see the below code in a version of the transformers I am not sure if I should just add this code in modeling_clip.py

def _make_causal_mask(
    input_ids_shape: torch.Size,
    dtype: torch.dtype,
    device: torch.device,
    past_key_values_length: int = 0,
    sliding_window: Optional[int] = None,
):
    """
    Make causal mask used for bi-directional self-attention.
    """
    bsz, tgt_len = input_ids_shape
    mask = torch.full((tgt_len, tgt_len), torch.finfo(dtype).min, device=device)
    mask_cond = torch.arange(mask.size(-1), device=device)
    mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)

    mask = mask.to(dtype)

    if past_key_values_length > 0:
        mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype, device=device), mask], dim=-1)

    # add lower triangular sliding window mask if necessary
    if sliding_window is not None:
        diagonal = past_key_values_length - sliding_window + 1

        context_mask = 1 - torch.triu(torch.ones_like(mask, dtype=torch.int), diagonal=diagonal)
        mask.masked_fill_(context_mask.bool(), torch.finfo(dtype).min)

    return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len + past_key_values_length)

[not-lain]

@bhosalems you will find them here so you will probably need this version of the transformers library : v4.32.0
just use

pip install transformers==4.32.0

and everything should go back to normal

[ArthurZucker]

_make_causal_mask, _expand_mask are private functions. There is no guaranty that they will be properly imported and are not part of the public api. They were removed as such!
I do not recommend you to go back to previous versions but rather use

transformers/src/transformers/modeling_attn_mask_utils.py

Line 21 in b1292bc

class AttentionMaskConverter:

and it's utilities

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.