Skip to content
This repository was archived by the owner on Feb 24, 2026. It is now read-only.
This repository was archived by the owner on Feb 24, 2026. It is now read-only.

Perplexity evaluation too high for 1bitLLM/bitnet_b1_58-3B #47

@MekkCyber

Description

@MekkCyber

Hello Everyone,

I am trying to evaluate the perplexity of 1bitLLM/bitnet_b1_58-3B, using the script available in integration/BitNet. However, I am getting a very high loss, and perplexity. Is it normal ?

avg_loss = 14.603133460411653: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 174/174 [02:04<00:00,  1.40it/s]
wikitext2 PPL: 24887.495944644015
[23059.264828634798, 24887.495944644015]
Avg PPL: 23973.38038663941

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions