Loading...
Please wait, while we are loading the content...
Similar Documents
Correlating the visual representation of user interfaces with their internal structures and metadata.
| Content Provider | CiteSeerX |
|---|---|
| Author | Chang, Tsung-Hsiang Yeh, Tom Miller, Robert C. |
| Abstract | Pixel-based methods emerge to be a new and promising way for developing new interaction techniques on top of existing user interfaces. However, in order to maintain platform independence, other available low-level information about GUI widgets, such as accessibility metadata, was neglected intentionally. In this paper, we present a hybrid framework, PAX, which correlates the visual representation of user interfaces (i.e. the pixels) and their internal hierarchical metadata (i.e. the content, role, and value). We identify challenges to build such a framework. We also develop and evaluate two new algorithms for detecting text at arbitrary places on the screen, and for segmenting a text image into individual word blobs. Finally, we validate our framework in implementations of three applications. We enhance an existing pixel-based system, Sikuli Script, and preserve the readability of its script code at the same time. Further, we create two novel applications, Screen Search and Screen Copy, to demonstrate how PAX can be applied to developing desktop-level interactive systems. ACM Classification: H5.2. Information interfaces and presentation: User Interfaces. |
| File Format | |
| Access Restriction | Open |
| Content Type | Text |
