Performance of newer versions of codex with gpt-oss has fallen off a cliff

[bartlettroscoe]

What version of Codex is running?

0.73.0

What subscription do you have?

plus

Which model were you using?

gpt-oss-20b

What platform is your computer?

Linux RedHat8 container

What issue are you seeing?

Our project has been putting together a benchmark suite for refactoring and test generation of C++ code. We have to use open-weights models for now and the gpt-oss models are fast and perform well (according to public benchmarks). We were getting some pretty good results with codex 0.55.0. But when we tried upgrading to codex 0.73.0, the performance on our benchmark cases fell off a cliff. Our pass@1 for codex + gpt-oss-20b-medium went from 0.6 to 0.1 and impacted all of our benchmark cases!

My guess is that this is due to gutting the default system prompt in the file codex-rs/core/prompt.md between codex versions 0.55.0 and 0.73.0. (I would need to do some debugging to verify that is the prompt being used for our use case but I don't see any others except for specific versions of gpt-5.)

I can fork the codex repo and try reverting just the hard-coded codex system prompt from the version in 0.55.0 but I first want to get some feedback to see if this is the likely issue.

Does OpenAI do benchmarking with the gpt-oss models when it updates codex, or is OpenAI only focused on benchmarks with the frontier gpt-5+ models? Otherwise, what would explain this cratering of performance of the gpt-oss models with updated versions of codex?

And why can't we tweak the codex system prompt without having to fork the repo? If we need better control and if OpenAI is not running benchmarks with the gpt-oss models when it updates codex, should we move on to a different open-source agent like OpenCode or some other tool that allows us more control?

What steps can reproduce the bug?

We would need to publish our benchmark suite. We are actually going to start that open-source process soon but it will be several months before that will be out.

What is the expected behavior?

It looks like codex-0.73.0 + gpt-oss-20b is giving up after half the model calls needed by codex-0.55.0 + gpt-oss-20b. I expect the agent to act better and solve the problem.

Additional information

Is codex the right CLI tool to be using with the gpt-oss models or should we be looking elsewhere?

[etraut-openai]

The overwhelming majority of codex users are using the latest frontier models, so that's where almost all of the team's attention is focused. We do little or no benchmarking or testing with older models.

I'm somewhat surprised to hear that you're seeing such a large degradation. You hypothesized that it might be related to changes to the fallback system prompt. That's an interesting theory, and I'd be very interested to hear the result of the experiment you proposed. Depending on the results of that experiment, there are various ways we might mitigate the degradation.

One other idea that comes to mind. Codex has historically supported two different wire protocols: "chat/completions" and "responses". We recently announced that we will be discontinuing support for that older chat wire protocol. It wasn't designed for tool calls or agentic workflows, and we've struggled to keep it working as we've introduced new features. Some of the hacks we've added to keep it working could have resulted in performance degradations. If you haven't already switched to the "responses" API, give that a try.

[bartlettroscoe]

Hello @etraut-openai, thanks for the quick response 👍

You hypothesized that it might be related to changes to the fallback system prompt. That's an interesting theory, and I'd be very interested to hear the result of the experiment you proposed. Depending on the results of that experiment, there are various ways we might mitigate the degradation.

It is reasonable to just try copying the system prompt from codex 0.55.0 to 0.73.0 and see if that restores the performance? Are there other issues I may need to look out for? (I have not built codex from source yet but hopefully that is not too hard.)

If you haven't already switched to the "responses" API, give that a try.

Unfortunately, codex 0.73.0 + llama.cpp (as of 'master' a few days old) with the "responses" API seems to be broken. I had to explicitly set wire_api = "chat" in the config.toml file to get it codex-cli 0.73.0 + llama.cpp + gpt-oss-20b working at all.

The broader question: Is OpenAI going to be supporting the usability and performance of the open-source stack codex-cli + llama.cpp + gpt-oss? (Note that codex-cli+ vllm + gpt-oss still does not work.) I mean, could OpenAI run a benchmark suite with codex-cli + llama.cpp + gpt-oss-20b (or 120b) and make sure there are no major regressions before putting out a new codex-cli release? Otherwise, what was the point of putting out the gpt-oss models if you can't actually use them in an open-source stack locally? If there is no such testing with codex-cli + llama.cpp + gpt-oss for codex releases, should we start looking at other open-source tools to effectively use the gpt-oss models and better control the system prompts and other tooling?

NOTE: The gpt-oss models are pretty impressive given their smaller size and great speed, so it would be a shame to not be able to use them for agentic coding tasks. We just don't have the option of using external models. We have to run models locally for the time being. And we can't run Chinees open-weights models either, so gpt-oss is really the only viable open-weights models to be doing anything useful right now with local models.

[etraut-openai]

It is reasonable to just try copying the system prompt from codex 0.55.0 to 0.73.0 and see if that restores the performance?

Yes, that should work for experimentation purposes. You can find instructions for building codex here.

This is a rapidly evolving space, and we're focused on expanding the frontiers of agentic coding. Our priorities are guided by input from our user base, and the overwhelming majority are using the frontier models.

The gpt-oss models were released way back in August. Since then, we've released half a dozen major updates to the frontier models. Perhaps you haven't used these lately, but their coding abilities are far beyond those of just a few months ago — and significantly beyond what the gpt-oss models are capable of.

If the gpt-oss models are working well for you, here are some options:

1. You can file bugs (like this one) if you run into problems. We will try to keep gpt-oss running well, but as you can see, we have a lot of open bug reports and feature requests. We generally prioritize bug fixes based on user impact. We also accept PRs for bug fixes.
2. You can maintain your own fork of codex. We've intentionally made the code open source to allow users to adapt it for their own use cases.
3. You can pin to an older version of codex. I wouldn't generally recommend this if you were using the cloud-based models, but for local models, it's an option.

[bartlettroscoe]

The gpt-oss models were released way back in August. Since then, we've released half a dozen major updates to the frontier models. Perhaps you haven't used these lately, but their coding abilities are far beyond those of just a few months ago — and significantly beyond what the gpt-oss models are capable of.

No question, it is just that we can't use any of those models and are stuck with open-weights models for the foreseeable future.

If the gpt-oss models are working well for you, here are some options:

I will do some more investigating and comment back here

[dylan-hurd-oai]

@bartlettroscoe additional information here would be really helpful for debugging the specific failures you're seeing.

1. What config are you running with, exactly? model value, any specific features enabled/disabled?
2. Is there anything unusual about your setup? Are you just running standard ollama?
3. Do you have any diagnostics on the specific failure modes? e.g. are specific function calls failing more frequently? is the model getting stuck on a particular behavior?

[bartlettroscoe]

additional information here would be really helpful for debugging the specific failures you're seeing.

@dylan-hurd-oai, I am going to do some more debugging today after upgrading llama-swap/llama.cpp again and report back here with more details. I can likely provide access to a pre-built container image that will reproduce what we are seeing. Stay tuned.

[vt-alt]

gpt-oss models are frontier OpenAI OSS models, since there are no updates for them. So, why they would be neglected?

[bartlettroscoe]

Update: We have not had time to dig into debugging this issue. We are able to keep making progress for now on our short-term goals using an older version of Codex CLI (0.55.0). At some point soon, we will get back to this and try the most recent Codex CLI release and/or we may look at other open-source coding agents (like OpenCode or OpenHands) to see if they work correctly with the gpt-oss models served with llama-swap/llama.cpp.

[bfroemel]

@bartlettroscoe Note that llama.cpp recently started to partially (good enough for codex) support the completions API: ggml-org/llama.cpp#18486

After disabling web_search (web_search = "disabled" in ~/.codex/config.toml), switching to the responses API (wire_api = "responses", and setting model_reasoning_effort to "high" everything seems to work rather well for me (small to medium sized python coding tasks).

(On the chat completions API I had to rename the "reasoning" field in codex to "reasoning_content" to get good performance with codex and llama.cpp. The naming mismatch made tool calls unreliable/looping because the model wasn't fed back its reasoning trace correctly.)

In case your performance is still degraded compared to 0.55.0 you can set a custom system/developer prompt via model_instructions_file = "~/.codex/custom_prompt.md" in ~/.codex/config.toml; Just to try different prompts than the ones that had been optimized for the commercial/closed frontier models.
Finally for debugging it might be helpful to see the model reasoning text in the transcript which can be enabled via show_raw_agent_reasoning = true in ~/.codex/config.toml.

[bartlettroscoe]

@bfroemel, thanks for the tips.

you can set a custom system/developer prompt via model_instructions_file = "~/.codex/custom_prompt.md" in ~/.codex/config.toml; Just to try different prompts than the ones that had been optimized for the commercial/closed frontier models.

According to:

• https://developers.openai.com/codex/config-reference

it says:

model_instructions_file	string (path)	Replacement for built-in instructions instead of AGENTS.md.

That would suggest that this just replaces AGENTS.md file, not the hard-coded system prompt baked into codex?

However, then looking at:

codex/codex-rs/core/src/config/mod.rs

Lines 802 to 806 in 81a17bb

	/// Optional path to a file containing model instructions that will override
	/// the built-in instructions for the selected model. Users are STRONGLY
	/// DISCOURAGED from using this field, as deviating from the instructions
	/// sanctioned by Codex will likely degrade model performance.
	pub model_instructions_file: Option<AbsolutePathBuf>,

it seems to suggest this would replace the codex system prompt. This is confusing.

I guess I can experiment and examine the actual JSON passed to the model to see what system prompt is actually used.