Visual Intelligence For GUI and Beyond

The presentation introduce the major part of the research I have been doing in the duration of my PhD study.

The title of the talk is Visual Intelligence For GUI and Beyond, which aims to build intelligent approaches to understand the graphical user interface (GUI) from visual perspective. Topics involved includes GUI element detection, GUI layout understanding and GUI automated testing

The works introduced in the presentation.

Besides, the visual intelligence we build in this process can also be leveraged to solve many practical problems beyond GUI and software engineering.

Watch out the presentation for more details.

Visual UI Semantic Understanding

When we human beings viewing a UI, we would intuitively realize where the UI elements and texts are and what contents in them. We may also guess the possible functionality of the element (e.g., profile picture, setting button) and segment & summarize the UI element blocks (e.g, list of user, menu multi-tub).

With this visual understanding, we can quickly focus on our interested parts of the UI and execute operations. This project is meant to empower machines with such level of visual UI semantic understanding to accomplish various tasks such as auto testing and auto app operation.

UI Semantic Meaning for Elements and Blocks

Understanding the UI semantics can be converted to two intuitive questions: What does it contain? and What do the components do? To answer the first question, we utilize the advanced UI-specific UI element detector UIED to recognize the basic UI elements from image and the Perceptual Grouping to segment the layout block as human beings do.

UI Element Detection and Perceptual Grouping

This step produces the basic visual properties of the UI components. Based on the results, the approach utilizes more sophisticated algorithms and models to analyse the role, functionality and description of the UI components.

The project is still ongoing, and more details will be revealed in future.

NiCro: Non-intrusive Cross-device and Cross-platform UI Action Record & Replay

Due to the huge diversity of the hardware and software environment of the end-device, an APP is necessited to be tested on diverse devices to ensure the compatibility and user experience.

An app displayed on various devices. “P“ and “V“ indicate if the device is “Physical“ or “Virtual“; “A“ and “I“ represent “Android“ and “iOS“ platforms; the last number shows the screen size in inches.

This cross-device and cross-platform testing is highly laborious and involves plenty of repetitive work to run and revise the test script on different devices.

Therefore, we propose the automated UI testing system NiCro to automatically replay the actions on various devices with different hardware (e.g., screen size) and software (e.g., operating system) environments.

The summarized workflow of NiCro.

NiCro is a purely non-intrusive system. Its approach only requires GUI images (e.g., screenshot or photo) as the input without any need of the app metadata. This enables it to be highly easy to use and generalize to any types of devices, as it does not demand any specific APIs to hack into the underlying system.

NiCro contains three parts:

1. Virtual Device Farm: It supports various Android device emulators with different Android versions and screen sizes, which can be easily added or removed.
2. Robotic System: It contains one or multiple robot arms and cameras to interact with any type of physical devices.
3. Host Computer: It collects the GUI images (e.g., screenshot or photo) from the above two parts and run the core visual approach:
   1. Detect the UI widgets using an accurate computer vision based detector UIED
   2. Find the widget on the replaying device that is matched to the target widget on the recording device.
   3. Match the widget-independent actions (scroll or swipe) from the recording device to the replaying devices.
   4. Output the converted action to the Virtual Device Farm and Robotic System to replay it on diverse devices.

NiCro System.

Augment Reality x Object Detection

The current Virtual Reality & Augment Reality are majorly focusing on creating virtual objects and environment to interact with the user, while the visual understanding to the real surrounding world through the VR or AR has not been developed thoroughly.

AR with artificial objects and world

However, the wearable AR glasses enjoy the intrinsic advantage to facilitate the user through visual understanding and analysis upon the real world and surrounding environment, and the most fundamental technique of such visual understanding is Object Detection.

Object Detection

Therefore, combining the AR with the increasingly advanced object detection model would produce many pragmatic applications, such as real-time text translation and related goods searching.

Real-time Translating

Related Goods Searching

We hence conducted one of the pioneering project to integrate the object detection technique with the AR glasses as the foundation for various Wowing applications.

Wearable AR Glasses

I developed tow demo applications, including a basic object detection and a hand recognition, to present the idea and potential.

Demo applications: object detection and a hand recognition

Palm Recognition & Feature Extraction

This project is meant to recognize palm region and extract the feature for further matching and analysis in real-time.

Palm Recognition

Instead of machine learning models that demand massive annotated data to train, the approach is based on traditional image processing algorithms that can robustly recognize and correct the direction of palm region without any laborious and time-consuming training process.

Image Processing-based Algorithm

It first gains the binary map of the hand image and then uses my algorithms to identify the finger-tips and the concave points between points. Then it extracts the palm region based on these points.

Next, the approach produces the features of the palm region for matching and identification. A vertical-rotation process is also involved to keep the consistency of the feature extraction.

Palm Feature Extraction

My approach is proven to be robust and effective that is able to process 18 frames per second in average. The demo of the approach can be viewed in the video.

APP Voice Control x Robot Arm

Automatically identify the target icon/UI element according the input natural language command (credit to Ms. Mengyu) and execute the APP operation through the robot arm.

APP Voice Control x Robot Arm

The Robotic System is a general platform that can be integrated with various applications to perform repetitive and laborious operations automatically. It contains three parts: (1) Robot arm (2) Camera and (3) a Plan Surface to put devices.

Robotic System

This system is easy to set up and control (only needs 3-d coordinates). It mimics the human interaction with the devices that uses the eye (camera) to see and the finger (robot arm) to operate, which has huge potential to be applied in scalable APP development and testing project.

UI Component Perceptual Grouping and Layout Block Segmentation

On GUIs, we do not just see a collection of separated texts, images, buttons, etc. Instead,we see perceptual groups of GUI widgets, such as card, list, tab and menu.

UI Grouping Block

Although humans can intuitively see perceptual groups of GUI widgets, current computational approaches are rather limited in partitionnig a GUI into meaningful groups of widget elements. Thus, the GUI Perceptual Grouping is proposed to mimic how human beings perceive the GUI to recognize and gather the atomic GUI elements into structured groups.

Gestalt Principles

The approach's design uses a well-established psychological theory, the Gestalt principles of perception, for reference to develop its core algorithms. Gestalt theory systematically explains how humans see the whole rather than individual and unrelated parts. It includes a set of principles of grouping, among which connectedness , similarity, proximity and continuity are the most essential ones.

Connectedness - We perceive elements connected by uniform visual properties as being more related than those not connected

Similarity - Elements are perceptually grouped together if they are similar to each other

Proximity - When people see an assortment of objects, they tend to perceive objects that are close (proximate) to each other as a group

Contiuity - Elements arranged in a line or curve are perceived to be more related

Gestalt in GUI

Gestalt principles also greatly influence UI design, but they may be represented in a distinct way.

Connectedness - Box container that contains multiple widgets within it, and all the enclosed widgets are perceived as in the same group

Similarity - Similarity can be observed in aspects of various visual cues, such as size, color, shape or position

Proximity - Some groups are close to each other and similar in terms of the number and layout of the contained widgets

Contiuity - Some detection errors are likely to be spotted if a GUI area or a widget aligns with all the widgets in a perceptual group in a line but is not gathered into that group

The approach is proven to be effective, and some of the results are shown below:

Grouping Result

ezForm: Easing Form Filling with Intelligent Form Analysis

Form is a common carrier to collect and convey information between individuals and organizations for various purposes. An effective form-filling design benefits the filler and the issuer by easing the filling process and gathering data efficiently.

Traditional filling methods, such as handwriting paper forms and editing the PDF form, involve many laborious actions and lack usability support.

Traditional ways of form filling suffer from many troubles

One the other hand, the web-based or mobile-based online form that enables many practical human-computer interaction functions emerges with the advancement of information technology.

We propose a novel web-based tool ezForm that automatically converts the PDF form or form image into an online form with rich usability enhancement.

ezForm Workflow

Our approach first recognizes all the form components and the structure layout using computer vision (CV) techniques and then generates an online form accordingly.

ezForm Approach

Our quantitative evaluation proves that ezForm achieve a promising result and has huge potential to be developed further.

ezForm Result

EasyD2C: Interactive Demonstration of Code Generation from Image

A interactive demonstration website to show the structural front-end code generation based on GUI image. Play with it on theWebsite.

EasyD2C Demo Page

Approach of UI2Code Generation

UIED: An Accurate Unsupervised UI Element Detector

Detecting Graphical User Interface (GUI) elements in GUI images is a domain-specific object detection task. It supports many software engineering tasks, such as GUI automation and testing, GUI search and code generation.

Existing studies for GUI element detection directly borrow the mature methods from computer vision (CV) domain, including old fashioned ones that rely on traditional processing features (e.g., canny edge, contours), and deep learning models that learn to detect from large-scale GUI data.

Detection Result Comparison of Different CV Methods (the green boxes are the detection bounding boxes for UI elements)

Unfortunately, these CV methods are not originally designed with the awareness of the unique characteristics of GUIs and GUI elements and the high localization accuracy of the GUI element detection task.

We design a new GUI-specific old-fashioned method for GUI element detection which considers the particular properties of GUI elements and achieves a promising performance.

UIED Approach

UIED comprises two parts to detect UI text and graphic elements: (1) For text , it leverages Google OCR to perfrom detection. For non-text elements, it uses old-fashioned CV approaches to locate the elements and a CNN classifier to achieve classification.

UIED Text and Non-Text Detection

UIED Interactive Website

We also developed an interactive website for the user to easily use UIED

UIED Website

The website also provides an interactive dashboard where the user can adjust the detection result and export it as structure file.

UI2CODE: Computer Vision Based Reverse Engineering of User Interface Design (Exploratory)

In practice of UI development, it usually involves two roles to complete the work, (1) a designer and (2) a developer.

UI2Code is HELPFUL!

The developers must manually convert GUI design produced by the designer into working code and laboriously revise the implementation whenever the designs undergo some (even trivial) changes.

Most existing GUI development tools focus attention on providing fancy design images and convenient drag and drop editor, but they do not directly offer the maintainable and usable source code, which is not suitable for the real industrial GUI development project.

To bridge the gap between GUI design and implementation, we propose a novel computer vision based GUI reverse engineering system, UI2CODE.

UI2Code Overview

Therefore, the most fundamental task for image-driven UI analysis - UI element detection should be addressed. And there from, the study of visual understanding of the graphical user interface just commenced.

Detect changes (e.g., plants, constructions) on a given region or block at some time periods by contrasting satellite images using computer vision techniques.

The architecture of the search engine

It then generates a comprehensive report for the local change detected, which including the region change overview and the detailed change type for each block.

Change type in each block

Search related unstructured log files in various formats (txt, pdf, word) from massive database by given some keywords. The engine is built upon ElasticSearch and deployed with a user-friendly webpage.

The architecture of the search engine

Detect targets in natural environment and recognize the digital numbers on them to instruct the unmanned Aerial Vehicle to complete certain action accordingly.

The digital detection result under complicated environment

Detect colored target regions in the dynamic natural environment to instruct the unmanned Aerial Vehicle to complete certain action accordingly.

The UAV and color targets

Detection results