Gemini fails to decode

[gtokman]

Describe the bug

When using Gemini flash models the stream is failing to decode. Looks like Gemini ommits a few fields from the response. Should these fields be optional?

  "id": "chatcmpl-B9Yydp5DMEQ0z5Hr7ujy0DqaBGZ2E",
  "service_tier": "default",
  "system_fingerprint": "fp_eb9dce56a8",

{
  "id": "chatcmpl-B9Yydp5DMEQ0z5Hr7ujy0DqaBGZ2E",
  "object": "chat.completion.chunk",
  "created": 1741619103,
  "model": "gpt-4o-2024-08-06",
  "service_tier": "default",
  "system_fingerprint": "fp_eb9dce56a8",
  "choices": [
    {
      "index": 0,
      "delta": {
        "role": "assistant",
        "content": "",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": null
    }
  ]
}

{
  "choices": [
    {
      "delta": {
        "content": "Hey",
        "role": "assistant"
      },
      "index": 0
    }
  ],
  "created": 1741619073,
  "model": "gemini-2.0-flash-lite",
  "object": "chat.completion.chunk"
}

[gtokman]

Also, I'm not sure if the goal of the library is to be agnostic across many different models. If not, happy to close this!

[nezhyborets]

@gtokman current decision is to support other providers without a harm to the main one - OpenAI. In this case (and there are other similar cases) making fields optional would somewhat drift the Swift structs away from official API Reference of OpenAI, because the reference would point out that a field is non-optional, but it would be optional in our SDK, which may be strange. But let's not close this, maybe we'll make a workaround for such a case without downgrading the experience of working with OpenAI

[gtokman]

@nezhyborets thanks for the context! What about in this case where the ChatCompletionMessageParam doesn't include the reasoning prop?

{
  "id": "chatcmpl-260011a1-d778-45bf-877c-46291ddeb5c3",
  "object": "chat.completion",
  "created": 1741629033,
  "model": "qwen-qwq-32b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "reasoning": "Okay, the user is asking...",
        "tool_calls": [
          {
            "id": "call_zgfy",
            "type": "function",
            "function": {
              "name": "get_linked_accounts",
              "arguments": "{}"
            }
          }
        ]
      },
      "logprobs": null,
      "finish_reason": "tool_calls"
    }
  ],
  "usage": {
    "queue_time": 0.21565806099999998,
    "prompt_tokens": 282,
    "prompt_time": 0.019009824,
    "completion_tokens": 181,
    "completion_time": 0.418294019,
    "total_tokens": 463,
    "total_time": 0.437303843
  },
  "system_fingerprint": "fp_c81bd6c71c",
  "x_groq": {
    "id": "req_01jp0jw2qwf6ar1kf7jbza1m7h"
  }
}

[nezhyborets]

@gtokman interesting! Sometime ago someone asked for the same field for DeepSeek. But it has reasoning_content key 😅 anyway, this addition doesn't interfere with OpenAI API spec and can be just added as an optional field to main Result structs set (ChatResult in this case). We'll just need to think about how to deal with different keys for the same thing. But maybe just add two optional fields.

[gtokman]

@nezhyborets also noticed the naming descrepencys across different APIs.

The node OpenAI client supports all of these from my testing:

- https://api.fireworks.ai/inference/v1
- https://api.together.xyz/v1
- https://api.perplexity.ai
- https://generativelanguage.googleapis.com/v1beta/openai
- https://api.x.ai/v1
- https://api.anthropic.com/v1
- https://api.mistral.ai/v1
- https://api.groq.com/openai/v1

[nezhyborets]

@gtokman you could use host and basePath fields of Configuration to build a path. As far as I can see it would work with any of the paths above. Let me know if it doesn't work for any

[gtokman]

@nezhyborets no I meant these providers all are OpenClient compatible. Although some of them ommit certain fields like the Gemini example above.

[batanus]

@nezhyborets Are there any updates regarding the optional id field and others initially mentioned? Because of this issue, this library doesn't work with Gemini API, and the fork is required to fix it.

UPD: After giving it more thought, I believe there are ways to work around this issue without making the fields optional:
1. According to OpenAI’s best practices, it’s not recommended to call the API directly from the client side, as it exposes the API key. Instead, you should use a backend server, where you can easily add any missing fields before forwarding the request to the OpenAI (or Gemini) API.
2. I opened a pull request yesterday introducing support for Middleware. With this in place, you can implement a custom middleware that automatically adds missing properties to the response data.

[gtokman]

@batanus good idea! so for the case with Gemini you'd check if the id with nil and then inject it with the middleware so that the decoding doesn't fail?

[batanus]

@gtokman Yes, something like this:

func interceptStreamingData(request: URLRequest?, _ data: Data) -> Data {
    guard var jsonObject = try? JSONSerialization.jsonObject(with: data) as? [String: Any], jsonObject["id"] == nil else { return data }
    jsonObject["id"] = someFallbackID
    return try? JSONSerialization.data(withJSONObject: jsonObject)
}

I didn't test this code, but it's how it is supposed to work 🙂

[gtokman]

@batanus sg! that works

[nezhyborets]

Hi guys. @batanus thanks for the PR! I believe with might implement that. But first I want all of you to check #297 to see if such solution may be beneficial in any way. I believe both solutions may have their pros and cons. We may eventually go with both.

[batanus]

Thank you for the response, @nezhyborets!

I feel that this approach might be a bit confusing and hard to understand for users. Almost always, most of the response parameters (I’d say 99% of parameters from different API providers) will be identical, and only in rare cases - such as with Gemini (which so far is the only exception) - will the server response differ.

Another concern is that anyone who wants to use Gemini with this library would need to define an almost identical ChatStreamResult model with just a couple of optional parameters.

Additionally, the application may not always know in advance which model will be used for an API call, which can increase the complexity of the codebase because of generics.

P.S. I initially opened the PR with middleware support for different purposes - mainly to add custom parameters to requests and enable logging - and only later realized it could also help with this issue 😄 I’m sure there are better and simpler solutions for especially this issue, but Middlewares could be useful in other scenarios as well 🙂

[gtokman]

I agree with @batanus. Defining duplicate models that have some and not all properties would be a bit annoying, but not the end of the world. I think I'd rather have this middleware layer vs using generics & models.

[nezhyborets]

Merged. I am not sure though that it fixes this issue and other issues of the SDK not being able to parse responses from different providers. Users of the SDK would probably want it to "just work" out of the box or with an easy additional config. Maybe we'd need some DefaultMiddleware that would adjust common use cases to our Swift structs