UI layers merger: merging UI layers via visual learning and boundary prior

Yunnong CHEN; Yankun ZHEN; Chuning SHI; Jiazhi LI; Liuqing CHEN; Zejian LI; Lingyun SUN; Tingting ZHOU; Yanfang CHANG

doi:10.1631/FITEE.2200099

Your Location：

Home >

Browse articles >

UI layers merger: merging UI layers via visual learning and boundary prior

Regular Papers | Updated：2023-03-24

- UI layers merger: merging UI layers via visual learning and boundary prior
- UI图层合并器：基于计算机视觉与边界先验的UI图层合并方法
- Frontiers of Information Technology & Electronic Engineering Vol. 24, Issue 3, Pages: 373-387(2023)
- Affiliations：
  
  1.School of Software Technology, Zhejiang University, Hangzhou 310027, China
  2.College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
  3.Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies, Hangzhou 310027, China
  4.Alibaba Group, Hangzhou 311121, China
  5.Zhejiang-Singapore Innovation and AI Joint Research Lab, Hangzhou 310027, China
- Author bio：
  
  ‡ Corresponding author
- Funds：
- DOI：10.1631/FITEE.2200099
  CLC： TP39
- Received：15 March 2022，
  
  Accepted：31 August 2022，
  
  Published：0 March 2023
- Accepted：
Scan QR Code
Yunnong CHEN, Yankun ZHEN, Chuning SHI, et al. UI layers merger: merging UI layers via visual learning and boundary prior[J]. Frontiers of Information Technology & Electronic Engineering, 2023, 24(3): 373-387.
DOI：

Yunnong CHEN, Yankun ZHEN, Chuning SHI, et al. UI layers merger: merging UI layers via visual learning and boundary prior[J]. Frontiers of Information Technology & Electronic Engineering, 2023, 24(3): 373-387. DOI： 10.1631/FITEE.2200099.

摘要

随着互联网行业图形用户界面（GUI）开发工作量的快速增长，国内外学者试图从GUI图片生成可维护的前端代码。该任务更适合从包含有用户界面元数据的UI设计稿来实现代码生成。然而，碎片图层不可避免地会出现在UI设计稿中，这大大降低了生成代码的质量。目前尚未有自动化GUI技术来检测并合并碎片图层以提高生成代码的可维护性。本文提出UI图层合并器（UILM），一种基于计算机视觉的方法，可以自动检测并合并碎片图层为完整的UI组件。本文提出的UILM包含合并区域检测器和碎片图层合并算法。合并区域检测器结合边界先验知识来准确地检测出UI组件的边界。基于此，碎片图层合并算法可以搜索出边界内的相关碎片层并将其合并为整体。此外，我们提出一种动态数据增强算法来提高合并区域检测器的性能。我们还构建了用于训练合并区域检测器和测试UILM的大规模UI数据集。实验结果表明，本文提出的方法在合并区域检测方面优于最佳基线，并达到了有效的图层合并准确率。最后，对真实应用程序的用户研究也证实了本文方法的有效性。

Abstract

With the fast-growing graphical user interface (GUI) development workload in the Internet industry

some work attempted to generate maintainable front-end code from GUI screenshots. It can be more suitable for using user interface (UI) design drafts that contain UI metadata. However

fragmented layers inevitably appear in the UI design drafts

which greatly reduces the quality of the generated code. None of the existing automated GUI techniques detects and merges the fragmented layers to improve the accessibility of generated code. In this paper

we propose UI layers merger (UILM)

a vision-based method that can automatically detect and merge fragmented layers into UI components. Our UILM contains the merging area detector (MAD) and a layer merging algorithm. The MAD incorporates the boundary prior knowledge to accurately detect the boundaries of UI components. Then

the layer merging algorithm can search for the associated layers within the components' boundaries and merge them into a whole. We present a dynamic data augmentation approach to boost the performance of MAD. We also construct a large-scale UI dataset for training the MAD and testing the performance of UILM. Experimental results show that the proposed method outperforms the best baseline regarding merging area detection and achieves decent layer merging accuracy. A user study on a real application also confirms the effectiveness of our UILM.

关键词

Keywords

references

Așıroǧlu B , Mete BR , Yıldız E , et al. , 2019 . Automatic HTML code generation from mock-up images using machine learning techniques . Scientific Meeting on Electrical-Electronics & Biomedical Engineering and Computer Science , p. 1 - 4 . doi: 10.1109/EBBT.2019.8741736 http://doi.org/10.1109/EBBT.2019.8741736

Behrang F , Reiss SP , Orso A , 2018 . GUIFetch: supporting app design and development through GUI search . Proc 5 th Int Conf on Mobile Software Engineering and Systems , p. 236 - 246 . doi: 10.1145/3197231.3197244 http://doi.org/10.1145/3197231.3197244

Beltramelli T , 2018 . pix2code: generating code from a graphical user interface screenshot . ACM SIGCHI Symp on Engineering Interactive Computing Systems , Article 3 . doi: 10.1145/3220134.3220135 http://doi.org/10.1145/3220134.3220135

Bunian S , Li K , Jemmali C , et al. , 2021 . VINS: visual search for mobile user interface design . CHI Conf on Human Factors in Computing Systems , Article 423 . doi: 10.1145/3411764.3445762 http://doi.org/10.1145/3411764.3445762

Cai ZW , Vasconcelos N , 2018 . Cascade R-CNN: delving into high quality object detection . IEEE/CVF Conf on Computer Vision and Pattern Recognition , p. 6154 - 6162 . doi: 10.1109/CVPR.2018.00644 http://doi.org/10.1109/CVPR.2018.00644

Chen CY , Su T , Meng GZ , et al. , 2018 . From UI design image to GUI skeleton: a neural machine translator to bootstrap mobile GUI implementation . Proc 40 th Int Conf on Software Engineering , p. 665 - 676 . doi: 10.1145/3180155.3180240 http://doi.org/10.1145/3180155.3180240

Chen CY , Feng SD , Xing ZC , et al. , 2019 . Gallery D.C.: design search and knowledge discovery through auto-created GUI component gallery . Proc ACM on Human-Computer Interaction , Article 180 . doi: 10.1145/3359282 http://doi.org/10.1145/3359282

Chen JS , Xie ML , Xing ZC , et al. , 2020 . Object detection for graphical user interface: old fashioned or deep learning or a combination? Proc 28 th ACM Joint Meeting on European Software Engineering Conf and Symp on the Foundations of Software Engineering , p. 1202 - 1214 . doi: 10.1145/3368089.3409691 http://doi.org/10.1145/3368089.3409691

Chen K , Wang JQ , Pang JM , et al. , 2019 . MMDetection: open MMLab detection toolbox and benchmark . https://arxiv.org/abs/1906.07155v1 https://arxiv.org/abs/1906.07155v1

Chen S , Fan LL , Su T , et al. , 2019 . Automated cross-platform GUI code generation for mobile apps . Proc IEEE 1 st Int Workshop on Artificial Intelligence for Mobile , p. 13 - 16 . doi: 10.1109/AI4Mobile.2019.8672718 http://doi.org/10.1109/AI4Mobile.2019.8672718

Deka B , Huang ZF , Franzen C , et al. , 2017 . Rico: a mobile app dataset for building data-driven design applications . Proc 30 th Annual ACM Symp on User Interface Software and Technology , p. 845 - 854 . doi: 10.1145/3126594.3126651 http://doi.org/10.1145/3126594.3126651

Fay MP , Proschan MA , 2010 . Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules . Stat Surv , 4 : 1 - 39 . doi: 10.1214/09-SS051 http://doi.org/10.1214/09-SS051

Feng SD , Ma SY , Yu JZ , et al. , 2021 . Auto-icon: an automated code generation tool for icon designs assisting in UI development . Proc 26 th Int Conf on Intelligent User Interfaces , p. 59 - 69 . doi: 10.1145/3397481.3450671 http://doi.org/10.1145/3397481.3450671

Ge XF , 2019 . Android GUI search using hand-drawn sketches . Proc IEEE/ACM 41 st Int Conf on Software Engineering: Companion Proc , p. 141 - 143 . doi: 10.1109/ICSE-Companion.2019.00060 http://doi.org/10.1109/ICSE-Companion.2019.00060

Halbe A , Joshi AR , 2015 . A novel approach to HTML page creation using neural network . Proc Comput Sci , 45 : 197 - 204 . doi: 10.1016/j.procs.2015.03.122 http://doi.org/10.1016/j.procs.2015.03.122

He KM , Zhang XY , Ren SQ , et al. , 2016 . Deep residual learning for image recognition . IEEE Conf on Computer Vision and Pattern Recognition , p. 770 - 778 . doi: 10.1109/CVPR.2016.90 http://doi.org/10.1109/CVPR.2016.90

Jain V , Agrawal P , Banga S , et al. , 2019 . Sketch2Code: transformation of sketches to UI in real-time using deep neural network . https://arxiv.org/abs/1910.08930 https://arxiv.org/abs/1910.08930

Li G , Baechler G , Tragut M , et al. , 2022 . Learning to denoise raw mobile UI layouts for improving datasets at scale . CHI Conf on Human Factors in Computing Systems , Article 67 . doi: 10.1145/3491102.3502042 http://doi.org/10.1145/3491102.3502042

Lin TY , Maire M , Belongie S , et al. , 2014 . Microsoft COCO: common objects in context . Proc 13 th European Conf on Computer Vision , p. 740 - 755 . doi: 10.1007/978-3-319-10602-1_48 http://doi.org/10.1007/978-3-319-10602-1_48

Lin TY , Dollár P , Girshick R , et al. , 2017 . Feature pyramid networks for object detection . IEEE Conf on Computer Vision and Pattern Recognition , p. 936 - 944 . doi: 10.1109/CVPR.2017.106 http://doi.org/10.1109/CVPR.2017.106

Liu Z , Chen CY , Wang JJ , et al. , 2020 . Owl eyes: spotting UI display issues via visual understanding . Proc 35 th IEEE/ACM Int Conf on Automated Software Engineering , p. 398 - 409 . doi: 10.1145/3324884.3416547 http://doi.org/10.1145/3324884.3416547

Liu Z , Chen CY , Wang JJ , et al. , 2023 . Nighthawk: fully automated localizing UI display issues via visual understanding . IEEE Trans Soft Eng , 49 ( 1 ): 403 - 418 . doi: 10.1109/TSE.2022.3150876 http://doi.org/10.1109/TSE.2022.3150876

Moran K , Bernal-Cárdenas C , Curcio M , et al. , 2020 . Machine learning-based prototyping of graphical user interfaces for mobile apps . IEEE Trans Soft Eng , 46 ( 2 ): 196 - 221 . doi: 10.1109/TSE.2018.2844788 http://doi.org/10.1109/TSE.2018.2844788

Nguyen TA , Csallner C , 2015 . Reverse engineering mobile application user interfaces with REMAUI . Proc 30 th IEEE/ACM Int Conf on Automated Software Engineering , p. 248 - 259 . doi: 10.1109/ASE.2015.32 http://doi.org/10.1109/ASE.2015.32

Ren SQ , He KM , Girshick RB , et al. , 2017 . Faster R-CNN: towards real-time object detection with region proposal networks . IEEE Trans Patt Anal Mach Intell , 39 ( 6 ): 1137 - 1149 . doi: 10.1109/TPAMI.2016.2577031 http://doi.org/10.1109/TPAMI.2016.2577031

Suleri S , Pandian VPS , Shishkovets S , et al. , 2019 . Eve: a sketch-based software prototyping workbench . CHI Conf on Human Factors in Computing Systems , Article LBW1410 . doi: 10.1145/3290607.3312994 http://doi.org/10.1145/3290607.3312994

Vu T , Jang H , Pham TX , et al. , 2019 . Cascade RPN: delving into high-quality region proposal network with adaptive convolution . Proc 33 rd Conf on Neural Information Processing Systems , p. 1430 - 1440 .

White TD , Fraser G , Brown GJ , 2019 . Improving random GUI testing with image-based widget detection . Proc 28 th ACM SIGSOFT Int Symp on Software Testing and Analysis , p. 307 - 317 . doi: 10.1145/3293882.3330551 http://doi.org/10.1145/3293882.3330551

Xu N , Price B , Cohen S , et al. , 2017 . Deep image matting . IEEE Conf on Computer Vision and Pattern Recognition , p. 311 - 320 . doi: 10.1109/CVPR.2017.41 http://doi.org/10.1109/CVPR.2017.41

Zhang XY , de Greef L , Swearngin A , et al. , 2021 . Screen recognition: creating accessibility metadata for mobile applications from pixels . CHI Conf on Human Factors in Computing Systems , Article 275 . doi: 10.1145/3411764.3445186 http://doi.org/10.1145/3411764.3445186

Zhao TM , Chen CY , Liu YM , et al. , 2021 . GUIGAN: learning to generate GUI designs using generative adversarial networks . Proc 43 rd IEEE/ACM Int Conf on Software Engineering , p. 748 - 760 . doi: 10.1109/ICSE43902.2021.00074 http://doi.org/10.1109/ICSE43902.2021.00074

Views

116

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

S3Det: a fast object detector for remote sensing images based on artificial to spiking neural network conversion

A modified YOLOv4 detection method for a vision-based underwater garbage cleaning robot^&

Automatic analysis of deep-water remotely operated vehicle footage for estimation of Norway lobster abundance

Detection of engineering vehicles in high-resolution monitoring images

Related Author

Mingqian SUN

Xiaofeng QI

Yanzhao GAO

Guangwei XIE

Fan ZHANG

Li CHEN

Junzhi YU

Licheng WU

Related Institution

National Digital Switching System Engineering & Technological R&D Center

Computation and Artificial Intelligence Innovative College, Fudan University

School of Cyber Science and Engineering, Southeast University

First Research Institute of the Ministry of Public Security of PRC

School of Information Engineering, Minzu University of China

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰