[GenerateAITAgs2] Calculate the number of tokens

[scraly]

For the moment I defined a max_tokens equals to 30000 but it could be useful to know the exact max_tokens

      const res = await fetch(process.env.OVH_AI_ENDPOINTS_MODEL_URL, {
        method: 'POST',
        headers: {
          'Authorization': `Bearer ${process.env.OVH_AI_ENDPOINTS_ACCESS_TOKEN}`,
          'Content-Type': 'application/json'
        },
        body: JSON.stringify({
          model: process.env.OVH_AI_ENDPOINTS_MODEL_NAME,
          temperature: 0.0,
          top_P: 1.0,
          messages: [
            {
              //System prompt adds more stability to the inferences
              role: 'system', content: systemPrompt,
              role: 'user', content: userPrompt 
            }
          ],
          max_tokens: 30000
        })
    });

Our user prompt and system prompt are not "tiny" so it's necessary to count how many tokens we are sending and then calculate 32000 - this number of tokens.

Use for example: https://www.npmjs.com/package/tiktoken