FRAME

Fusing UI Structure & Semantics for
Feature-Oriented App Screen
Retrieval & Clustering

Arun Krishna Vajjala  ·  Yanfu Yan  ·  Ajay Krishna Vajjala  ·  Shrunal Pothagoni  ·  Denys Poshyvanyk  ·  Kevin Moran
George Mason University  ·  Zhejiang University  ·  William & Mary  ·  University of Central Florida
ICSME 2026  ·  Benevento, Italy
Benevento. Some time in the 600s.
i.
A rite at the walnut tree.
Lombards · honoring Wotan
ii.
Their Christian neighbors watched.
iii.
What happened.
They saw witchcraft.
iv.
The legend travels.
v.
1273
first reports of the gatherings beneath the walnut
1486
Malleus Maleficarum — and Benevento becomes the meeting place named in confessions across Europe
Two centuries of trials.
A ritual misunderstood, leading to nine hundred years of consequences.
vi.
Mistaken identity.
The ritual was only judged on the surface.

Two screens. One feature.

A financial app login screen
Log in
A game app login screen
Log in

Same supported feature. Different palette, different type, different logo, different layout, different everything a pixel-grid model can see.

Mistaken identity. A model that reads purely visual differences sees them as different screens.

Screens from RICO.

One surface. Two features.

SIGN IN Email Password SIGN UP Email Password

Near-identical field layouts. The difference in functionality can be traced to a single word.

False identity. A model that reads structure alone sees them as the same screen.

what we want to measure
Semantic screen similarity
the extent to which a pair of UI screens
share similar supported features

Screen Similarity underpins many UI tools

screen retrieval & clustering Test transfer across apps Context-specific UI testing scenario selection Non-crashing bug detection visual & behavioral faults Duplicate UI bug reports report triage Fault localization from screen evidence Prototyping & UI evolution design reuse

Where current models fall short

WHAT IT ISWHAT IT ENCODESone featureone grid
01

Screens as monolithic pixel grids

SE tasks need the functional module — components that together provide a feature.

LABELLED CORPUSNEXT YEAR’S UIhand-labelled, app by appMODELout of distribution
02

Expensive, proprietary training

UI corpora are costly to label, and they go stale as design conventions move.

WHAT THE BAR ISnavigationWHAT THE MODEL SEESTextTextTextthree unrelated Text elements
03

No shared component context

Nothing models the joint context across related components.

our approach
FRAME
ReinForced UseR InterfAce Screen EMbedding
with Graphical Structural ComprEhension
Neuro-symbolic. A vision-language backbone for what the components are; a symbolic graph for how they relate.

A structurally infused embedding

CLIP EMBEDDING STRUCTURAL INFORMATION Structurally Infused Embedding

Four steps

  1. Edge detectiongreyscale + contrast, then UIED to find components
  2. Graph creationone node per component: CLIP image + BERT text embedding
  3. Embedding propagationneighbors exchange context along the graph
  4. Rips complexradius-based connection; triangle area weights the average
augmented UI graph representation triangle area as embedding weight

Putting it together

global CLIP screen embedding what the screen looks like + propagated, weighted component embeddings how the components relate PCA 1,792 → 116 dimension reduction FRAME embedding

Evaluation

Aurora
21 categories
1,370 screens
Avgust
25 labels
2,475 screens
Enrico
20 labels
1,460 screens

all three are subsets of RICO

Baselines
  • Screen2Vec — UI embedding
  • BERT — text embedding
  • BLIP / CLIP — image embeddings
Metrics
  • HR@K — similar labels in the top K
  • MRR — where the first correct hit lands
  • Clustering — accuracy, FMI, V-measure, ARI, DBI, SSE

RQ1 — Retrieval · Aurora

1,370 screens · 21 categories · the sparse corpus
HR@1Screen2Vec0.3750BERT0.3194BLIP0.4097CLIP0.4722FRAME0.5625* HR@5Screen2Vec0.3027BERT0.2486BLIP0.3944CLIP0.4013FRAME0.4652* HR@10Screen2Vec0.2604BERT0.2187BLIP0.3639CLIP0.3590FRAME0.4166* MRRScreen2Vec0.4853BERT0.4363BLIP0.5479CLIP0.5963FRAME0.6729* bars share one 0–1 scale · * p < 0.05 The sparse corpus, and the widest margin: +12.8% MRR over CLIP.

RQ1 — Retrieval · Avgust

2,475 screens · 25 labels · the easiest corpus
HR@1Screen2Vec0.3922BERT0.8270BLIP0.8509CLIP0.8549FRAME0.8823* HR@5Screen2Vec0.3185BERT0.7788BLIP0.7937CLIP0.7788FRAME0.8329* HR@10Screen2Vec0.2739BERT0.7141BLIP0.7522CLIP0.7364FRAME0.7827* MRRScreen2Vec0.5116BERT0.8908BLIP0.8861CLIP0.8913FRAME0.9184* bars share one 0–1 scale · * p < 0.05 The easiest corpus — every model does well, and FRAME still leads each metric.

RQ1 — Retrieval · Enrico

1,460 screens · 20 labels · class-imbalanced
HR@1Screen2Vec0.2895BERT0.2212BLIP0.2837CLIP0.3365FRAME0.3894* HR@5Screen2Vec0.2389BERT0.2019BLIP0.2712CLIP0.3048FRAME0.3096 HR@10Screen2Vec0.2063BERT0.1822BLIP0.2577CLIP0.2740FRAME0.2903 MRRScreen2Vec0.4271BERT0.3849BLIP0.4534CLIP0.5051FRAME0.5391 bars share one 0–1 scale · * p < 0.05 Class-imbalanced and small: FRAME still leads, but only HR@1 reaches significance.

RQ2 — Clustering · Avgust

K-means, k = the number of labels · CLIP vs FRAME
0 0.25 0.50 Clustering accuracy 0.0072 CLIP 0.1669 FRAME FMI 0.1394 CLIP 0.4170 FRAME Homogeneity 0.0112 CLIP 0.4742 FRAME Completeness 0.0107 CLIP 0.4668 FRAME V-measure · NMI 0.0109 CLIP 0.4705 FRAME ARI 0.0032 CLIP 0.3183 FRAME LOWER IS BETTER DBI FRAME wins CLIP 7.2990 FRAME 2.1801 SSE CLIP wins CLIP 17,044.7 FRAME 54,236.0 Eight of nine. CLIP’s lower SSE is tightness, not accuracy.

RQ2 — Clustering · Aurora

K-means, k = the number of labels · CLIP vs FRAME
0 0.25 0.50 Clustering accuracy 0.0017 CLIP 0.0217 FRAME FMI 0.1403 CLIP 0.2799 FRAME Homogeneity 0.0192 CLIP 0.2530 FRAME Completeness 0.0208 CLIP 0.2702 FRAME V-measure · NMI 0.0199 CLIP 0.2613 FRAME ARI -0.0015 CLIP 0.1637 FRAME LOWER IS BETTER DBI FRAME wins CLIP 11.5100 FRAME 3.5316 SSE FRAME wins CLIP 86,165.5 FRAME 47,169.3 FRAME takes all nine metrics, SSE included.

RQ2 — Clustering · Enrico

K-means, k = the number of labels · CLIP vs FRAME
0 0.25 0.50 Clustering accuracy 0.0074 CLIP 0.0543 FRAME FMI 0.1372 CLIP 0.2314 FRAME Homogeneity 0.0083 CLIP 0.1364 FRAME Completeness 0.0081 CLIP 0.1362 FRAME V-measure · NMI 0.0082 CLIP 0.1363 FRAME ARI -0.0025 CLIP 0.1012 FRAME LOWER IS BETTER DBI FRAME wins CLIP 6.8759 FRAME 2.7673 SSE CLIP wins CLIP 9,832.1 FRAME 68,604.9 Eight of nine, on the hardest corpus in the study.

RQ3 — Ablation

Every configuration is a set of switches, so it is drawn as switches.
COMPONENT ABLATION · AURORA MRRfilled = present · struck through = removedwhole screencomponent textcomponent parts PRE-PROCESSING · AURORA MRRsame reading · bars start at 0.60greyscalecontrast ONE COMPONENT 0.5963 0.5026 0.3880 TWO COMPONENTS 0.6448 0.6077 0.5136 ALL THREE 0.6729FRAME 0.6484 0.6492 0.6171 0.6729 Contrast alone hurts. Raisingcontrast on a full-color screenshot hinderscomponent detection. Greyscale is theprerequisite that makes it pay. The whole screen matters most — and of the two component channels, text beats image.

RQ4 — What it actually retrieves

A query screen and the top three most similar screens retrieved by FRAME and by CLIP

Query: a disclaimer modal sitting over a base screen.

FRAME returns three more modals — same component relationship: an overlay panel, dense body text, an accept action in the corner.

CLIP returns list menus and legal pages. Same general look, different feature.

Cosine similarity over each embedding. Fig. 8 in the paper.

Where FRAME loses

SignIn
SignUp
Menu
Account
4 of Avgust’s 25 screen types

Near-identical input-field layouts. The difference lives in the text, where CLIP’s multimodal pre-training still has the edge over FRAME’s structural prior.

Nine hundred years later,
we still have cases of mistaken identity.

FRAME

  • A neuro-symbolic screen embedding that infuses UI component structure into a vision-language backbone — no UI-specific training required.
  • Beats Screen2Vec, BERT, BLIP and CLIP on retrieval across three datasets, and CLIP on 25 of 27 clustering metrics.
  • Ablation illustrates the importance of FRAME’s components: embedding propagation and greyscale + contrast pre-processing.
  • Don’t judge the screen by its surface. Ask what the components are doing with each other.
Thank you
Arun Krishna Vajjala  ·  Yanfu Yan  ·  Ajay Krishna Vajjala  · 
Shrunal Pothagoni  ·  Denys Poshyvanyk  ·  Kevin Moran
arXiv:2608.08880   ·   kpmoran@ucf.edu
github.com/SageSELab/FRAME
sagelab.io/talks/frame-icsme26.html
← → or space to advance  ·  F for fullscreen  ·  ↓ to skip builds