Home
Search

Discussion

Discussion categories

General
Tech

Library

Library pages

Design
Resources
Websites
Articles
Web development
Cool Libraries
Tools
Resources
Papers and Studies
Articles
Language Models
Tech and Systems
Computers
Windows Tools and Modding
Windhawk
Raycast for Windows
Rainmeter
Vintage Computing Festival Berlin 2026
Plasma 6.7
Haiku: BeOS-Inspired Open-Source OS
Random fun stuff
Esoteric File Systems
Cool websites
Friends
Unity - Cheaterman's Bar
ChatGPT
Gentoo is Rice - Humor about Linux Obsession
Storyden
Selfh.st
OpenAlternative
Microlaunch
Peerlist
Glama.ai
AlternativeTo
Brandfetch
Dokploy
PitchHut
Piefed Social
Collections
Links
Members
Roles
RegisterLogin
Login
Library
ui-ug-a-unified-mllm-for-ui-understanding-and-generation

No versions or drafts yet.

UI-UG: A Unified MLLM for UI Understanding and Generation

UI-UG: A Unified MLLM for UI Understanding and Generation

Although Multimodal Large Language Models (MLLMs) have been widely applied across domains, they are still facing challenges in domain-specific tasks, such as User Interface (UI) understanding accuracy and UI generation quality. In this paper, we introduce UI-UG (a unified MLLM for UI Understanding and Generation), integrating both capabilities. For understanding tasks, we employ Supervised Fine-tuning (SFT) combined with Group Relative Policy Optimization (GRPO) to enhance fine-grained understanding on the modern complex UI data. For generation tasks, we further use Direct Preference Optimization (DPO) to make our model generate human-preferred UIs. In addition, we propose an industrially effective workflow, including the design of an LLM-friendly domain-specific language (DSL), training strategies, rendering processes, and evaluation metrics. In experiments, our model achieves state-of-the-art (SOTA) performance on understanding tasks, outperforming both larger general-purpose MLLMs and similarly-sized UI-specialized models. Our model is also on par with these larger MLLMs in UI generation performance at a fraction of the computational cost. We also demonstrate that integrating understanding and generation tasks can improve accuracy and quality for both tasks. Code and Model: https://github.com/neovateai/UI-UG

arxiv.org