feat(express): Introduce A2UI Express compiler, draft proposals, and evaluations

[gspencergoog]

Summary

This pull request introduces A2UI Express, an experimental, compact domain-specific language (DSL) that allows agents to express UI layouts with minimal token usage. It provides a complete compilation/decompilation pipeline, integrates it into the evaluations strategy suite, and documents the specification draft under proposals.

Evaluation Results

Below is a summary of the evaluations benchmark run (47 samples) comparing the layout generation strategies using the lightweight google/gemini-3.1-flash-lite model:

Strategy	a2ui_scorer Accuracy (Syntax)	measured_model_graded_qa Accuracy (Semantics)	Avg Latency	Median Latency	Output Tokens	Total Tokens
direct (Raw JSON)	87.23%	93.62%	4.99s (baseline)	3.57s (baseline)	35,022 (baseline)	527,882 (baseline)
express (A2UI Express DSL)	91.49%	84.04%	1.09s (78% reduction)	1.06s (70% reduction)	9,912 (72% reduction)	232,748 (56% reduction)

By compiling into inline components and inline data models, the Express strategy matches or exceeds the direct strategy's syntactic validity while reducing generation latencies by 70% (over 3.3x faster) and saving 56% of total generation tokens (and 72% of output tokens).

Changes

Python Agent SDK

• agent_sdks/python/a2ui_agent/src/a2ui/express/: Added the core package including:
  • compiler.py: Compiles flat Express DSL statements into standard A2UI wire JSON. Includes robust exception handling and positional argument validation.
    • Added support for bare $ path token resolving to relative root JSONPointer {"path": ""}.
    • Added recursive nested check compilation (e.g. ?and([?required, ?email])).
    • Added automatic unrolling of inline component constructor instantiations (e.g. Row([Text("Soup")]) -> _inline_1).
    • Added lexical scanner rules to ignore Python-style # and JS-style // comments outside string literals.
  • decompiler.py: Translates standard wire JSON layouts back into the flat DSL representation, with safety guards on missing properties.
  • prompt_generator.py: Generates LLM system instruction prompt contracts containing component signatures, enums, static property indicators, and nested schema requirements dynamically.
  • schema_helper.py: Crawls catalog JSON schemas to resolve positional parameter bounds and properties, with type checks to skip boolean schemas inside allOf loops.
• agent_sdks/python/a2ui_core/: Improved core validation routines to make them schema-driven:
  • node_graph.py & integrity_checker.py: Decoupled structural and static checks from hardcoded catalog definitions, resolving constraints dynamically from catalog schemas.
• Feature Flag Control: Added environment variable switches:
  • A2UI_EXPRESS_ENABLED: If not set to true, importing a2ui.express raises an ImportError.
  • A2UI_VERSION_1_0: Gates the new dynamic JSON schema validator for v1.0. Automatically enabled if A2UI_EXPRESS_ENABLED is true.
• SDK Tests: Added comprehensive test cases in tests/express/test_express.py covering DSL tokenization, parsing, expression nesting, map variable inlining, comments, and validator gating. Added tests/express/test_cli_tools.py to test all 4 CLI wrapper scripts under specification/proposals/express/, achieving 90% test coverage for the CLI utility suite.

Specification & Proposals

• specification/proposals/express/: Relocated and formalized the post-1.0 draft specification (a2ui_express.md, evolve_express.md, and sample configs) to prevent polluting the ratified v1_0 baseline.
• DSL Examples: Relocated all .a2ui evaluation layout examples to specification/proposals/express/examples/.

Evaluations Suite

• eval/a2ui_eval/strategies/express.py: Added the express evaluation solver that runs and merges layout accuracy scores. Express compiler results generate standard A2UI v1.0 JSON payloads directly.
• eval/tasks.py / main.py: Shifted the core baseline evaluation task to run against A2UI v0.9.1 (renamed to a2ui_v0_9_1_eval), while keeping the express strategy target isolated to A2UI v1.0.
• Evaluation Datasets: Introduced a new translated dataset eval/datasets/v1_0_prompts.yaml that adapts prompt texts and target descriptions to use v1.0 terminology (inline components, longText instead of multiline, and omission of returnType properties). This aligns target outputs with strategy versions for clean, accurate LLM grading.

Impact & Risks

• No Breaking Changes: The compilation pipeline is isolated and disabled by default under the A2UI_EXPRESS_ENABLED environment switch.
• No Regression: The existing baseline evaluation runs (direct, subagent_tool) are preserved as defaults.

Testing

• Unit Tests: Run uv run pytest inside agent_sdks/python/a2ui_agent/ folder.
• Evaluations Run: To run the new strategy in the benchmark runner, run uv run main.py --strategies express inside the eval/ folder.

[gemini-code-assist]

Code Review

This pull request introduces A2UI Express, a compact, token-efficient declarative DSL for generative user interfaces, complete with a compiler, decompiler, prompt generator, CLI tools, and integration into the evaluation framework. The code review highlights several critical issues, including a sentinel parsing bug that ignores statements on the same line as tags, a potential crash in the schema helper when handling boolean schemas in allOf, and a decompiler bug that incorrectly strips quotes from string literals matching component IDs. Additionally, the reviewer noted hardcoded absolute paths in the dataset translation script and a PEP 8 import style violation in the evaluation solver.

[jacobsimionato]

Exciting!!!

[jacobsimionato]

Next, let's combine ExpressPromptGenerator and ExpressCompiler into one ExpressInferenceFormat which implements AbstractInferenceFormat, so that people can implement other inference formats and reuse this inspect_ai strategy.

[gspencergoog]

Good idea. I'll leave that for another PR though.

[jacobsimionato]

Yes! Love the idea of having the reverse direction too, so we can make sure that the model also understands sees content through the DSL

[gspencergoog]

This was partly to make sure that I could "round trip" things to make sure that the express format could support everything that A2UI can. It was also helpful in converting examples in the catalog to use in the prompt.

[jacobsimionato]

I think this is also important to handle conversation history, assuming it is stored in the more standardized, stable A2UI format. If the agent is communicating back and forth with A2UI, we want it to always read and write A2UI in the same format, to avoid confusion (and maximise efficiency!). So I think we need the decompiler for this.

Steps for an inference in a multi-turn conversation:

1. Convert all A2UI messages in existing conversation history to Express format (required decompiler)
2. Convert Catalog to express format
3. Prompt agent with system prompt in express format, and conversation history also using express format.
4. Parse express output back to A2UI (requires compiler)
5. Persist conversation history in A2UI format.

[jacobsimionato]

Do these examples need to be checked in? Could we generate them on demand using the decompiler from the canonical A2UI examples?

[gspencergoog]

We could generate them, yes. That's how I produced them.

[jacobsimionato]

Amazing! Should this be checked in though?

[gspencergoog]

No, that is an artifact of the improvement loop I was running. I'll remove it.

[jacobsimionato]

Are you intending to land this PR as-is? Any chance of breaking it up a bit, to make it easier to do thorough reviewing?

[jiahaog]

What does "clean positional signatures" refer to?

[gspencergoog]

It just means that the DSL uses argument positions without named arguments. "Clean" is because they don't have the named arguments, I guess.

[jiahaog]

What about escaping?

[munificent]

And newlines in strings.

[gspencergoog]

Embedded newlines are allowed, they just have to be properly closed.
I added more rules about quoting and escaping too:

• Strings are represented in two formats:
  • Standard Strings: Enclosed in single double quotes (e.g., "Enter your name") or triple double quotes (e.g., """Line 1\nLine 2"""). Standard strings support common escape sequences: \n (newline), \t (tab), \\ (backslash), and \" (double quote). Embedded newlines are allowed.
  • Raw Strings: Prefaced by r (e.g., r"^[a-zA-Z]+$" or r"""Raw multi-line content"""). In raw strings, no escape sequences are processed, and backslashes are interpreted as literal characters. This is particularly useful for validation regex patterns containing backslashes. Embedded newlines are allowed.

[jiahaog]

The compilation pipeline runs on the host application.

Do we need to opine on this? I thought the compiler runs (and is an implementation detail of the agent). If it runs on the client, we have more problems like dealing with versioning

[gspencergoog]

It does. I think "host application" is the agent here. I'll reword it.

[jiahaog]

Some more general comments:

• For the benchmarks we should use at least use a non-lite model. It's unlikely that a lite model will be sufficient in the "direct" strategy since the agent needs to do domain specific things, in addition to generating output as A2UI. I'll send some changes to allow us to benchmark multiple providers besides just Gemini too.
• Is your intention for express to be an implementation detail of the agent SDK? If not, perhaps we should start with that first (also see my inline comment on where the compiler should run)

[gspencergoog]

Are you intending to land this PR as-is? Any chance of breaking it up a bit, to make it easier to do thorough reviewing?

No, I mainly wanted to get your (and Jia Hao's) impressions on it. I'll split it up today into at least two PRs, one with the proposal directory and docs, and one with the Python agent implementation and evals, taking into account both of your review comments.

I was maybe a little overexcited to share the results and it needs some more pruning and cleaning.

[gspencergoog]

Some more general comments:

• For the benchmarks we should use at least use a non-lite model. It's unlikely that a lite model will be sufficient in the "direct" strategy since the agent needs to do domain specific things, in addition to generating output as A2UI. I'll send some changes to allow us to benchmark multiple providers besides just Gemini too.

Yes, I agree. I did run the direct model on Lite too, so it's at least comparable, but you're right that there isn't a lot more reasoning overhead left there after building the UIs. I mainly wanted to see 1) that a Lite model could handle the reasoning needed (showing that it wasn't too reasoning heavy), and 2) to see how fast we could go if latency were the only driver.

It works well in Flash 3.5, but is of course slower. I found that limiting the thinking budget helped a bit there.

• Is your intention for express to be an implementation detail of the agent SDK? If not, perhaps we should start with that first (also see my inline comment on where the compiler should run)

Yes, that is my intention. The rendering clients should have no idea that Express was ever involved, and just see regular A2UI.

[yjbanov]

Are we sure we want to support names that contain "-" and other special characters? In the future, when models become more advanced, we might want to introduce features that would take advantage of special characters, but allowing them in identifier names could cause ambiguities in the grammar.

[gspencergoog]

Good point. We probably should limit it to the recommendations in https://www.unicode.org/reports/tr31/ so that we can allow Unicode characters in identifiers (I recently added this to A2UI as well).

[gspencergoog]

Done.

[yjbanov]

Another " can almost certainly appear inside a multiline string like this. How do we ensure we can still parse it correctly?

[gspencergoog]

"Naaah, that'll
"never" happen." :-)

Sure, we should specify quoting rules. I really like the """ rules that other languages like Dart have. I want to make sure we can keep the prompt small though, so it's easy to describe, but I don't want to open the door to something like just saying "Use Python string quoting rules" because that opens the door to supporting lots of other syntax (raw string quoting, multiple character escape formats, misinterpreting conflicting string interpolation formats, etc.). Maybe there's a way we can say it that is concise but doesn't mention a language.

e.g. "For quoting strings, surround with a double quote. If a double quote appears in the string, then use """ around the string. Individual double quotes may also be escaped with a preceding backslash. For strings that are raw strings interpreted literally, precede the string with an r: r" or r""" ".

Unlike other languages, one of the driving criteria here is that its rules are able to be concisely described in English prose.

[yjbanov]

Do we need formatString? Dart and JS can format strings without it. Just the ${ syntax is enough. It's also fewer tokens this way.

[gspencergoog]

formatString is implemented in the catalog. This makes it so that a catalog author can implement whatever string formatting they want to use (e.g. you could implement a printf function and use printf formatting instead). It also makes it a lot easier to implement a renderer, since the string formatting isn't part of the specification, and implementing formatString is actually pretty involved (it's reactive, and so generates new strings whenever the interpolated values, including interpolated function calls, change, for instance).

We provide an implementation in the core libraries that catalog writers can leverage so they can just include it.

I can see the argument for including it in the language, but if we do that for Express, it needs to be included in A2UI as well, and implemented in all renderers.

[jacobsimionato]

I agree with just using our existing formatString function call approach for the first cut of express, to keep it catalog-agnostic.

But in the future, I think it'd be interesting to pursue these optimizations which make the format catalog-specific in service of greater performance gains.

We might be able to generalize this - JSON render has a concept of "directives" which is something like this - https://json-render.dev/docs/directives.

So perhaps you create a generic ExpressInferenceFormat and then install a directive that transforms ${/number} to formatString(/number) etc

[munificent]

@yjbanov asked me to take a look with an eye to language design stuff. I have very little context on the overall problem being solved and I'm definitely not an expert at what kinds of code LLMs are good at writing and reading, so take all of this with a very large grain of salt.

Most of my comments are probably fine but I figured it's better to bring it up then potentially miss a problem.

Overall, this makes sense to me. Unlike a syntax where the entire thing is one monolithic tree which isn't meaningfully parsable until you have the whole thing, it gives you a way to break the code into smaller separately handle-able units.

That does raise questions around forward declarations and how names are resolved and managed. Name resolution in general is pretty vague here and is something you'll likely want to be pedantic about. It's a part of language design that has a lot of sharp edges.

Also, an explicit grammar in something like EBNF would be nice to see. I know it's not something that everyone loves, but the exercise of writing it will force you to answer a lot of things that might otherwise be left implicit and then become subtle parser bugs. (For example, it's not clear from the spec here if component definitions can have map literals as arguments or not. Can lists have trailing commas? Be empty? Be empty except for just a comma?)

I'm always excited to see people doing novel language design to approach a problem a different way! :D

[munificent]

It's surprising that this reduces token size so much. In theory, a directly nested syntax should be more concise than what's proposed here because it avoids repeating path components, as in:

foo/bar/baz/bang/a = 1
foo/bar/baz/bang/b = 2
foo/bar/baz/bang/c = 3
foo/bar/baz/bang/d = 4
foo/bar/baz/bang/e = 5
// 55 tokens ("token" in the PL sense, not LLM sense)

// Versus:
foo(
  bar(
    baz(
      bang(
        a = 1
        b = 2
        c = 3
        d = 4
        e = 5
      )
    )
  )
)
// 27 tokens

So is the improvement here really from the flattening, or from not having a nesting syntax that has a lot of other unnecessary boilerplate like quoted key names, argument names, comma separators, etc.?

If so, perhaps it would be beneficial for this notation to allow nesting too?

[gspencergoog]

There were two reasons I stayed away from this kind of nesting:

1. if we want to "stream" user interfaces, nesting like this is hard, since we have to wait until the end, or do some auto-closing magic in order to do intermediate states. If I can use adjacency lists to separate the components from the list, then it makes streaming much nicer: we ignore symbols that we don't recognize yet, and fill them in when they are defined. This lets the LLM be sloppy about ordering the definitions, and

2. the LLM then doesn't need to keep track of all the nesting paren levels, which it is not terribly good at.

[munificent]

One of the examples below looks like:

root = Card(main-column)
main-column = Column([icon, title, description, actions], _, "center")
icon = Icon($/icon)
title = Text($/title, "h3")
...

Lines here often refer to names declared on later lines. That implies that we can't always process lines as they come in, unless the system can gracefully handle references to unknown entities.

[gspencergoog]

Yes, this is on purpose to let the LLM control streaming behavior. The A2UI renderers already handle this by ignoring symbols that they don't recognize until they are defined, and also caching symbol definitions that aren't yet connected to anything until they get used. This lets us stream in a Column with identifiers for the children, and then fill in the children as the come in, or vice versa and have them all pop in at once when the Column definition arrives.

[munificent]

What happens if a user refers to root? What does this do:

root = Row([root])

Or is it "write-only" in some way?

Related: how are cyclic references handled?

[gspencergoog]

It is an error to have circular references.

The renderers will throw an error back to the agent if they catch circular references. This is a somewhat bad design in that in order to catch these (algorithmically) on the server before they hit the client, the server has to keep track of everything the client has seen. On the other hand, the renderer is the final say as to the actual state of things, so maybe it's appropriate there.

[munificent]

For what it's worth, I've seen various little DSLs and hobby languages over the years try to stake a claim like this in the name of simplicity (or because their authors aren't comfortable writing a full expression parser) and most usually end up dialing it back over time. It becomes really annoying if you can't do any computation in a nested expression.

If someone wants to do:

root = Framed(app-frame-thickness + (is-android ? android-frame-adjust : ios-frame-adjust) + 4)

Do you really want them to have to write something like:

a = is-android ? android-frame-adjust : ios-frame-adjust
b = app-frame-thickness + a
c = b + 4
root = Framed(c)

If this DSL is really only for authoring component trees, it's probably fine. But you do have literal values and even lists. Presumably it will be useful to add numbers, concatenate strings, or append to lists. Having to hoist all of that out to separate named declarations could get really annoying.

Though if this code will never be written or read by a human... 🤷

[gspencergoog]

Well, exactly, it will not be written by a human. Which is weird.

The prohibition there is actually there more to keep an LLM from writing an entire tree in one expression, so that streaming works better. It forces it to split it up into a bunch of lines that can be evaluated as they come in.

[munificent]

And newlines in strings.

[munificent]

So it seems like map literals can be used as expressions basically anywhere? If so, you probably want to add them to ### Core primitive types.

[gspencergoog]

Yeah, good point, I'll do that.

[munificent]

I don't have enough context to know what "validation" means here. But if the leading ? is just syntactic sugar for calling a function with that name, does it carry its weight?

[gspencergoog]

Validation here is in the context of "form validation", in the sense of wanting to check that a text field contains an email address, for instance. It is handled by defining a client side function to do the validation and return a boolean. Any function that returns a boolean can be used as a validation function.

The ? is just syntactic sugar for calling a boolean-returning function that takes an implicit first "value" first argument.

To be honest, I haven't really thought this part through that well. I think probably it could just be a regular function call syntax, but that does have slightly higher (LLM) token size (not much though).

Maybe it should instead be something like

username = TextInput($/form/username, [required(_), regex(_, "^[0-9]{5}$")])

and we can sub in the value for the _. Right now that would be:

username = TextInput($/form/username, [?required, ?regex("^[0-9]{5}$")])

[munificent]

So the system implicitly understands that a list containing validation checks is itself a validation check? What about:

[?required, "oops, not a validation check"]

Would it make more sense to do:

?[required, email]

[gspencergoog]

So the system implicitly understands that a list containing validation checks is itself a validation check? What about:

[?required, "oops, not a validation check"]

This would fail because the string isn't a boolean.

Would it make more sense to do:

?[required, email]

No, the values don't have to be functions, they could also be from the data model.

[munificent]

You do say an assignment can span multiple lines, so it's probably clearer to say that separate top-level assignments or standalone operations may be executed before later ones are parsed. Is that the intent here?

Are there comments?

[gspencergoog]

Yes, that's the intent.

There aren't explicitly comments, but I do actually ignore both # and // end of line comments in the parser because the LLM sometimes adds them anyhow. We don't want to mention or "allow" them because they just take up tokens we're not going to use.

[munificent]

What if the offending line is defining some name that is referred to elsewhere on other lines?

[gspencergoog]

The other place will just ignore the undefined value. It might not render because of that, but it would just wait until there's a valid value there, which could come in a an error correction update.

[gspencergoog]

@yjbanov asked me to take a look with an eye to language design stuff. I have very little context on the overall problem being solved and I'm definitely not an expert at what kinds of code LLMs are good at writing and reading, so take all of this with a very large grain of salt.

Thank you for taking a look! I really appreciate the feedback.

Overall, this makes sense to me. Unlike a syntax where the entire thing is one monolithic tree which isn't meaningfully parsable until you have the whole thing, it gives you a way to break the code into smaller separately handle-able units.

Yes, we want to be able to stream the data, make corrections, and recover from missing pieces.

That does raise questions around forward declarations and how names are resolved and managed. Name resolution in general is pretty vague here and is something you'll likely want to be pedantic about. It's a part of language design that has a lot of sharp edges.

Okay, point well taken. That makes a lot of sense. I'll see if I can lock that down.

Also, an explicit grammar in something like EBNF would be nice to see. I know it's not something that everyone loves, but the exercise of writing it will force you to answer a lot of things that might otherwise be left implicit and then become subtle parser bugs. (For example, it's not clear from the spec here if component definitions can have map literals as arguments or not. Can lists have trailing commas? Be empty? Be empty except for just a comma?)

Also a great idea. We won't be giving it to the LLM because that's too verbose, but we need it for the compiler and it would help formalize the language.

I'm always excited to see people doing novel language design to approach a problem a different way! :D

Thanks! This one is weird because

1. it's not meant to be authored, or even read, by a human
2. it's description needs to be easily, accurately, and completely describable in a minimum of English words in a prompt. and
3. it's throw-away code: we're not going to compile it to machine code, optimize it, etc. and
4. we're actively trying to stay away from making a Turing complete language so that we can avoid all the security and complexity implications of that.

[chenman9226]

One thing I’m curious about: since Express drops many of the JSON/property keys, have you observed any impact on generation quality from losing those semantic cues? For example, keys like label, value, children, action, or validation seem like they might help the model understand which argument is which, especially for less common components or longer positional signatures.

I can see the token-efficiency benefit, so I’m mostly wondering whether this showed up in practice, or whether the shorter format generally outweighed the loss of those semantic anchors.

[gspencergoog]

One thing I’m curious about: since Express drops many of the JSON/property keys, have you observed any impact on generation quality from losing those semantic cues? For example, keys like label, value, children, action, or validation seem like they might help the model understand which argument is which, especially for less common components or longer positional signatures.

I can see the token-efficiency benefit, so I’m mostly wondering whether this showed up in practice, or whether the shorter format generally outweighed the loss of those semantic anchors.

The thing I thought might be a problem, but doesn't appear to be, is that LLMs aren't great at counting, so I thought that positional parameters would be the issue (getting them in the wrong order, or inserting something between them).

The lack of property keys also doesn't seem to affect quality as long as the descriptions from the JSON schema in the catalog are included. For example, if the catalog item for TextField is converted to this:

• TextField(label, value?, placeholder?, variant? (static only), weight? (static only), checks? (static only))
  - label: The text label for the input field.
  - value: The value of the text field.
  - placeholder: The placeholder text for the input field.
  - variant: The type of input field to display. Must be one of: 'longText', 'number', 'shortText', 'obscured'
  - weight: The relative weight of this component within a Row or Column. This is similar to the CSS 'flex-grow' property. Note: this may ONLY be set when the component is a direct descendant of a Row or Column.

Then the LLM has enough context to be able to decide what each argument means when it writes the output. We may have to play with how the catalogs are converted to prompts to minimize the tokens in the prompt, but we will need to include the entire descriptions that are supplied by the catalog developers because they can include important instructions for how to use the components.

In fact, for a while I wasn't including any of the descriptions at all (which saved a lot of input tokens!), and as long as the parameter was something well named and intuitive, the LLM seemed to be able to extrapolate. If anything was vague or unconventional then it started to break down, however.

[gspencergoog]

FYI, I created a stack of three PRs that I split this PR into:

• feat(validation): Support A2UI v1.0 validation and nested reference path checks #1718
  • The core library and validation layers for the A2UI v1.0 and Express compiler outputs. It introduces support for v1.0 schemas and enhances component reference resolution for nested slot structures.
• feat(express): Implement A2UI Express compiler, decompiler and parser gspencergoog/A2UI#3
  • The A2UI Express technical specification, introducing the A2UI Express format. It includes the compiler, decompiler, schema helper, and parser modules, as well as a EBNF format description of the format.
• feat(eval): Integrate A2UI Express into evaluation suite gspencergoog/A2UI#4
  • Integrates the A2UI Express compiler and prompt strategies into the Inspect-ai evaluation suite. It updates the scorers, solvers, and task configurations to support v1.0 and Express DSL targets.

The last two are diffs on the first one and each other to make a stacked set of PRs. The last two are PRs on my fork, not on the main repo, until we roll the first one into the repo, and then I'll change their target to the main repo and land them too.

I converted this PR back to a draft and will close it once the other PRs land.