pix2code: Generating Code from a Graphical User Interface Screenshot

Abstract

Transforming a graphical user interface screenshot created by a designer into computer code is a typical task conducted by a developer in order to build customized software, websites, and mobile applications. In this paper, we show that deep learning methods can be leveraged to train a model end-to-end to automatically generate code from a single input image with over 77% of accuracy for three different platforms (i.e. iOS, Android and web-based technologies).

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Recurrent Neural Network Regularization
• Towards Diverse and Natural Image Descriptions via a Conditional GAN
• TerpreT: A Probabilistic Programming Language for Program Induction
• Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training

Cited by in corpus (6)

• Neural Program Synthesis with Priority Queue Training
• Auto Completion of User Interface Layout Design Using Transformer-Based Tree Decoders
• Sketch2code: Generating a website from a paper mockup
• Overcoming Language Dichotomies: Toward Effective Program Comprehension for Mobile App Development
• Automatically Generating Codes from Graphical Screenshots Based on Deep Autocoder
• CloudifierNet -- Deep Vision Models for Artificial Image Processing