Can't run bert_vocab_from_dataset without TypeError: Tensor is unhashable

[mihalt]

Issue type

Support

Have you reproduced the bug with TensorFlow Nightly?

No

Source

source

TensorFlow version

2.13

Custom code

Yes

OS platform and distribution

No response

Mobile device

No response

Python version

No response

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

This is the code from you manual and I really don't understans that I get this error. Why is it?

If I add
tf.compat.v1.disable_eager_execution()
tf.compat.v1.disable_v2_behavior()

I get
RuntimeError: input_dataset: Attempting to capture an EagerTensor without building a function.

Standalone code to reproduce the issue

data = tf.data.TextLineDataset([SENTENCES_PATH, TAGS_PATH])

from tensorflow_text.tools.wordpiece_vocab import bert_vocab_from_dataset as bert_vocab

tokens = bert_vocab.bert_vocab_from_dataset(
    data,
    # The target vocabulary size
    vocab_size = 50000,
    # Reserved tokens that must be included in the vocabulary
    reserved_tokens=["[PAD]", "[UNK]", "[START]", "[END]"],
    # Arguments for `text.BertTokenizer`
    bert_tokenizer_params=dict(lower_case=True),
    # Arguments for `wordpiece_vocab.wordpiece_tokenizer_learner_lib.learn`
    learn_params={},
)

Relevant log output

TypeError: Tensor is unhashable. Instead, use tensor.ref() as the key.

[sushreebarsa]

@mihalt I was able to run the manual code successfully, please find the gist here. I tried to replicate the issue reported here and faced a different error. Please find the gist here and share all the dependencies. Thank you!

[mihalt]

I was able to run the manual code successfully, please find the gist here.

Yes, I know, that on server of official documentation it is fine

I tried to replicate the issue reported here and faced a different error. Please find the gist here and share all the dependencies. Thank you!

You can create any txt files with any text and put its path instead SENTENCES_PATH, TAGS_PATH to prevent an error
NameError: name 'SENTENCES_PATH' is not defined

[mihalt]

I've just run my code in another pc and it looks fine. So, what a problem can be in my problematic pc? What do you think?

I have both

>>>tf.executing_eagerly()
True

[mihalt]

@sushreebarsa You can congratulate me, I've found an error. Looks like that this code incompatible with import trax. But I a bit not fully understand what to do now, if I would like to use them both.

[SuryanarayanaY]

@mihalt ,

I tried replicating the issue with a simple dataset (without trax) and its working fine.Refer attached gist.

It seems trax has different dependencies compared to tensorflow/tensorflow_text. I am not sure why trax being used by you. I request you to create different environments for both and then try.

[mihalt]

@mihalt ,

I tried replicating the issue with a simple dataset (without trax) and its working fine.Refer attached gist.

It seems trax has different dependencies compared to tensorflow/tensorflow_text. I am not sure why trax being used by you. I request you to create different environments for both and then try.

I tried replicating the issue with a simple dataset (without trax) and its working fine.Refer attached gist.

yes, it works without trax

And what if I need it in the same notebook? I am not sure that new environment should help in this case.