Review experiment status & potential next steps

[Locke0]

Review the finetuning experiments we've done from the relevant notes in this order:

• Blog outline finetuning experiment sections
• Analysis
• Raw experiment notes - This is the temp notes I kept while I did the experiments, some of the notes were outdated, use the blog outline and analysis as the high abstracted source of truth

Review papers:

• SpatialLadder
• GTA1 paper
• An Efficient Training Pipeline For Graphical User Interface Reasoning Models
• AutoGUI
• ZonUI
• Other relevant papers you find

Review and identify what are the next steps for improving 7B CUA models on our perturbation evaluation (especially on the spatial relational instructions). I have listed some directions such as improving the training data mix, and using RL training methods instead of SFT with LoRA.

You should investigate for all possible directions (breadth first) and find evidences and justifications / uncertainties for how to prioritize these investigation directions based on the risks and time required.

The next step after this ticket will be investigating more in depth on the top priority direction and design minimal rapid experiments to validate specific hypothesis (e.g., do this in this way will improve 7B model on our perturbation eval spatial relational instructions)

[Locke0]

Relevant to the LoRA path:

• Another paper on MLLM LoRA finetuning that seems helpful: https://www.alphaxiv.org/abs/2501.03012
• this paper: https://www.alphaxiv.org/abs/2405.18541 mentions "Choosing the location of LoRA modules requires careful consideration", which could another sub-direction under LoRA.

[Locke0]

@mathuryash5 is finalizing his literature review and structuring findings into notes here

before we break out specific next ticket

[mathuryash5]

• P0. Validity-gated “gold” relational set (DOM/AX)
  Filter perturbed samples where target/ref missing/occluded, relation no longer true, or candidate is non-unique; prevents negative scaling from broken supervision.
  Refs: Efficient Training Pipeline (https://arxiv.org/abs/2511.08172), AutoGUI (https://arxiv.org/abs/2502.01977)

• P0. RLFT > SFT for grounding (start GRPO/RLOO with verifiable rewards)
  Optimize directly for click-in-bbox + relational constraints (using stored target_id/ref_id/relation), rather than cross-entropy on a single coordinate.
  Refs: UI-R1 (https://arxiv.org/abs/2503.21620), GuirlVG (https://arxiv.org/abs/2508.04389)

• P0. Dense reward shaping (reduce sparse hit/miss)
  Swap binary reward for Gaussian point + coverage rewards; calibrate variance by element size to learn “near-miss → correct” under perturbations.
  Ref: GUI-G² (https://arxiv.org/abs/2507.15846)

• P0. Explicit equivariance / transform-consistency objective
  Enforce that predictions transform correctly under known perturbations (zoom/crop/resize); add as auxiliary loss or RL reward term.
  Ref: ReGUIDE (https://arxiv.org/abs/2505.15259)

• **P0. Test-time scaling (diagnostic) **
  Sample N predictions and aggregate via region-consistency voting; optionally do test-time RL on unlabeled screens.
  Ref: GUI-RC / GUI-RCPO (https://arxiv.org/abs/2508.05615)

• P1. Self-supervised transition learning (robust to non-causal UI changes)
  Train on GUI state transitions (inverse dynamics) to focus on action-relevant signals and ignore visual noise—complements perturbation robustness.
  Ref: UIShift (https://arxiv.org/abs/2505.12493)

• P1. Coordinate-free action head (avoid brittle coordinate tokenization)
  Predict regions/patches (and verify/select) instead of emitting raw coordinates; improves generalization across resolution/layout changes.
  Ref: GUI-Actor (https://arxiv.org/abs/2506.03143)

• P1. Explicit position-to-coordinate mapping (target high-res / zoom failures)
  Inject explicit coordinate markers / positional mapping so patch→pixel conversion is reliable under resolution/zoom shifts.
  Ref: Position→Coordinate Mapping (https://arxiv.org/abs/2510.03230)

• P2. Modern synth+filter pipeline (only if we need more data)
  If scaling data: use pipelines that explicitly address element/screen ratio, imbalance, implicit instructions + strong filtering; avoid “more noisy synthetic.”
  Refs: UI-E2I-Synth (https://arxiv.org/abs/2504.11257), Efficient Pipeline (https://arxiv.org/abs/2511.08172), Jedi/OSWorld-G (https://arxiv.org/abs/2505.13227)

• Research bet (if relational remains hard): spatial reasoning interventions
  Spatial reasoning is a known weak spot in VLMs; consider progressive spatial curriculum or modality-balance fixes if reward/data changes plateau.
  Refs: Beyond Semantics (https://arxiv.org/abs/2503.17349), SpatialLadder (https://arxiv.org/abs/2510.08531)

[Locke0]

Some follow-up questions and links:

Near term

• What are the limits to the small data is better claim? What apps / platforms does small data cover? does training on small data improve all of them?
• Related to Coordinate-free prediction: https://github.com/YXB-NKU/SE-GUI
• synthetic environments: https://x.com/tzafon_company/status/2027101372597072351?s=20. Maybe extremely simple synthetic environments but enable huge combinatorial scenarios (like the colored circles thing they did) could generate part of the training data mix to be used with SpatialLadder finetuning recipe.

Longer term

• Is Beyond semantics useful for improving the design of existing MRoPE and variants used in CUAs like Qwen2.5VL, Qwen3VL for better visual, functional, geometric understanding and visual "puzzle" solving in GUI environments?
  • Related to better vision encoder - https://x.com/Brian_Bo_Li/status/2021649265123373149
• GUI dynamics model best prediction / learning objectives & intermediate representations (semantic schemas, latent)
• dynamics model: https://si.inc/

I think the test-time scaling is also related to VLM judge model idea, which I think could be important for self-play, self-exploration direction. Since this self-exploration loop is bottlenecked by instability and need good process reward modeling and verification mechanisms (judge models). Anyways, we'll come back to these longer term ideas in a few weeks in our roadmap discussion.

[mathuryash5]

What are the limits to the small data is better claim? What apps / platforms does small data cover? does training on small data improve all of them?

I think we should treat this as an empirical question. I guess it will all depend on whether the small dataset has representation of the UI patterns (layout diversity, semantic diversity of instructions too ) allowing the model to learn and generalize.

Related to Coordinate-free prediction: https://github.com/YXB-NKU/SE-GUI

Interesting, thanks for the pointer.

synthetic environments: https://x.com/tzafon_company/status/2027101372597072351?s=20. Maybe extremely simple synthetic environments but enable huge combinatorial scenarios (like the colored circles thing they did) could generate part of the training data mix to be used with SpatialLadder finetuning recipe.

This is great. I think synthetic environments can be used to provide infinite layout diversity and potentially multi-point (target + reference + relation) supervision. We just have to figure out the unrealistic supervision (similar to the issue with the perturbed examples that we have currently).

[Locke0]

relevant: OpenApps

[Locke0]

Moving this back to backlog. Will resume after preprint. Meanwhile, I will review more closely and keep the discussion in mind as I prepare for our roadmap meeting

[Locke0]

Another relevant paper: UI-INS: ENHANCING GUI GROUNDING WITH MULTI-PERSPECTIVE INSTRUCTION-AS-REASONING