Readable token streaming support

[greshilov]

Hey folks,

Thank you for this amazing library! :)

I waited a bit for #311 to resolve, but eventually decided to help complete it.

The original @jeffra's commit is cherry-picked, so the commit history stays valid!

Description

While testing the original PR, I stumbled upon several issues:

• It trimmed redundant spaces (the example becomes the example), because it was based on the space splitting

• It sometimes replaced more characters than needed (if the previous token was found multiple times in the current token, like an anchor becomes an chor)

• It didn't work properly when more than two tokens are required to decode the symbol

For example, with tiiuae/falcon-7b tokenizer, 例如 becomes �如 (the first symbol is a Unicode REPLACEMENT CHARACTER)

My approach is slightly different then used before.

Two consequent tokens are always stored in the form of a token_ids sequence. The idea is to find the difference between previous and (previous + current) token decode results:

def example(...)
    ...
    # n - is the size of the previous token
    prev = tokenizer.decode(token_ids[:n])
    both = tokenizer.decode(token_ids)
    result = both.replace(prev, "", 1)
    token_ids = token_ids[n:]
    return result

I added unit tests for the ReadableStream for all supported models + one sanity test checking consistency between regular inference and streaming inference.

Should address: #306, #281, #347

[greshilov]

@microsoft-github-policy-service agree company="JetBrains"

[mrwyattii]

This is amazing! Thank you for the contribution @greshilov