Automating User Actions on GUI: Defining a GUI Domain-Specific Language

Transcription

1 Automating User Actions on GUI: Defining a GUI Domain-Specific Language Michaela Kreutzová 1 and Jaroslav Porubän 1 1 Department of Computers and Informatics, Faculty of Electrical Engineering and Informatics, Technical University of Košice, Slovak Republic {Michaela.Kreutzova@gmail.com, Jaroslav.Poruban@tuke.sk} Abstract. GUI (Graphical User Interface) is the most important part of a software system allowing user interaction. Its code takes more than 50% of the whole application code [13]. Research is made mainly in the area of GUI testing, which is in general practice often omitted. Main cause is a need for creating new specific testing tool for each application. Our goal is to prove a possibility of defining computer languages via user interfaces in a way of designing a universal software system allowing GUI components identification, recording and automatic performing of user actions on them. The recorded files use a domain-specific language of the interface it is therefore easily creatable and readable even for domain experts and non-programmers. Formalized terms of the GUI can be used in further processing. Keywords: domain-specific language, graphical user interface, component, Java. 1 Introduction The primary reason why the computers exist and why we use them is to ease the work for people. We try to replace manual work, writing and calculation with semiautomatic work (a user working with the computer), or with fully automated computer processing (a computer working alone). The primary point where the user meets the application is the GUI, therefore we tried to apply the same principle of automation in this area. 1.1 Motivation The starting point enabling further processing is the GUI domain terms formalization. Based on the GUI containing components we can find these components, identify them and extract the terms from them. The extracted terms are known to the user as he sees them in the interface and we can formalize them in shape of a DSL (Domain Specific Language). A DSL is a computer programming language of limited expressiveness focused on a particular domain (for instance medicine, banking, building, etc.) [3],[4],[5].

2 The extracted information can be further processed with the computer in different ways: Automatic product usability evaluation If a developer wants to verify his application s usability and how close it is to users, probably he d done it hiring some common users to test it manually. Each user gets a list of tasks to execute an application s GUI. While he s working he is monitored by programming experts or with camera to find out, how did he performs each of the tasks, which task did he wrong and which task he didn t manage to finish at all. The collected data are evaluated and the result tells the developers what they should change on the application so it would respond to the customer requirements. Based on formalized domain terms it s possible to programmatically evaluate the application s usability. Product usage teaching When the components are identified successfully based on domain terms it s possible to programmatically execute user actions on them, even to create some sort of a macro to be played on the real application. It sure is more understandable than some long manuals full of text and static screenshots of application. Automation and testing Current trends in research head just the way of automated GUI testing. Most approaches are based on an event flow ([7][8][9][11][12][13]). A test case is usually a sequence of events that can be executed on GUI. The starting point is to locate the GUI components [6]. Each testing tool identifies the components differently. Mainly, identification is performed using coordinates in main window and listening to events (mouse clicks) on these coordinates. Brooks and Memon in their work [2] are proposing a solution based on Event Flow Graphs and creating Usage Profiles to generate test cases and test packages. They however create output files as matrices of events that were conducted on GUI. These files are not so simply creatable and readable. An interesting solution is presented in [10]. It s simple for testers, it s however based on actual look of the graphical components icons, buttons etc. Some testing tools use listening to exact keystrokes, mouse movements or commands. Mouse coordinates and movements however depend on operating system, on which the application runs and on actual look of the application interface. Keystrokes and commands can be easily changed during application evolution. If these things change the generated test cases are not usable anymore. In practice GUI testing is usually omitted because of the need to create a new specific testing tool for every application and creating test cases with each change in the application interface or code. It s however possible to automate user actions on application based on GUI domain terms component identification. The domain terms are relatively changeless and that enables us to develop such a tool, that could use the same generated test cases even if the whole application GUI is replaced to another. It s however important not to change the terms in the application interface. These three areas represent a good subject for further research.

3 1.2 Problems and Objectives Problems discussed in previous sections can be formulated as follows: 1. A lack of research in the area of GUI. 2. The research is made mainly in the area of GUI testing, but it s not based on GUI domain terms. Generated test cases are not easily creatable and readable by non-programmers and depend on actual GUI graphical form and components layout. Considering the stated problems, we have defined the objectives which need to be fulfilled: 1. Examine the possibilities of GUI domain terms formalization. 2. Design a tool that proves a possibility to use a defined domain specific language to identify components and programmatically execute user actions on them. 2 GUI as a Source of Domain Terms How we already know, the GUI consists of components. A basis is always some window or a dialog and we put every component in it. For better prospect we can put the components into unlimited amount of containers (JPanel). All the components and containers in the interface represent terms in the user interface. For these terms there exist some specific rules and relationships between them and their hierarchy is exactly determined. For instance, JButton and JPanel represent terms. Their relationship is that JPanel component can contain JButton components. A hierarchy builds up using inheritance, which defines a membership of the component in a logically related group. For instance if JButton inherits from AbstractButton, this relationship automatically puts it into a group of buttons, assigning correct attributes and functions to it and determining a correct interface for its class. We can identify these terms, extract information from them and reflect them into a component tree (Fig. 1). Name of the window/dialog can tell us, to which domain (domain part) its components belong (for instance on Fig. 1 the components represent information about a Person). Fig. 1 Person form and mapping its components into a component tree

4 2.1 Component as a domain-specific unit The following list presents the most frequently used graphical components of Java Swing library in alphabetical order and their meaning as terms of GUI language: AbstractButton abstract class representing all functional components of button and menu type. Buttons and menus are basic functional components of a GUI and they relate to a window (dialog), in which they are located. List of all buttons and menus in a window (dialog) represents basic functionality of the window (dialog). Excluding general terms like OK, Cancel, Edit, Add, Delete, Save etc.) we are getting domain-specific application functionality. Container represents a grouping of components. About domain-specific components located in a container we can tell that they are related logically or graphically. JComboBox allows choosing one item of a list. Items can represent domainspecific units and we can tell that they are logically related and mutually exclusive. JFileChooser allows choosing a file or directory. It s a functional GUI item enabling using a local file or directory to read or process. JList allows choosing one or more items of a list. Items can represent domainspecific units and we can tell that they are logically related and not mutually exclusive it s possible to choose more items at once depending on selection model. JMenu a specific case of a container grouping together menu items (or functional elements). We can tell that items of its group are logically related in functionality. For instance a menu File groups together functional elements allowing file manipulation like: Open, Save, Save As, Close etc. A menu can represent a group of logically related functional domain-specific units. JSpinner represents a functional item displaying a specific type of data (number, date etc.). It allows incrementation and decrementation of these data using its functional buttons (arrows). About domain-specific units of this type we can tell what type they are (according to display) and what their constraints are. JTabbedPane a specific type of container allowing displaying its components on one or more switchable pages (tabs). These pages group together logically related components. Thus a JTabbedPane component can represent a list of groups of domain specific units and membership of a unit is determined with a tab title. JTextComponent represents all basic text components. About this item type we can tell that it serves for acquiring and/or saving data of text type given by user. We can get information about type of data (from label, type classic text field, password text field etc.), their maximal length, about their content and constraints for it; and in case of existence of a label also about their purpose. JTree represents a tree structure. The tree label tells us, to which domain do its items belong. Items of the tree represent units of this domain (domain part) with a precise hierarchical order, which indicates relations between them. Own components when creating new components a programmer inserts new terms into the general language, he enlarges the general language with new domainspecific terms. A domain-specific content depends on a concrete component implementation. If a programmer creates a new component, it is like an injection of new words into natural interface vocabulary. However, if a programmer creates these components

5 incorrectly (he defines terms, relationships and rules incorrectly or not at all) it s like a violation of a natural language, inserting unknown terms in it making impossible further communication in this language. [16] The components therefore should be created according to a component definition so that they d be fully open, reusable [14][15] and so they would not interfere with the existing GUI language. 2.2 Identification of a component Different types of components are identifiable differently and we can extract different information from them. Components in the component tree can be identifiable with these basic properties: 1. Component type a class of the component. 2. Component name the name attribute of the component. 3. Component title the title attribute of the component. 4. Component tool tip text the tooltiptext attribute of the component. 5. Action command the actioncommand attribute of a functional component. 6. A labelfor attribute of a JLabel component when using a component, that doesn t have its own visible identifier we can use a JLabel and set its labelfor attribute to refer to that component. 7. Component path a path in the component tree. It is a representation in memory and that always exists. Therefore a computer path is a universal identifier. 8. Component coordinates not stable, depends on interface shape, component layouts and many other conditions. 9. Own components specific component identification capability depends on correct component implementation. Points 1 to 6 can represent domain-specific terms. Based on terms presented in section 2.1 and identification properties presented in this section an application domain analysis can be conducted. 3 Implemented Software Solution Implemented software system uses a principle of capture/replay design pattern. In the first phase user actions on an application are recorded and in the second phase the recorded commands can be automatically performed on the application. Components identification signs used to save commands into output file are based on domain terms visible in GUI, therefore the file is easily readable and creatable. The identifiers as domain terms do not change so rapidly as graphical positions of components, therefore the generated files are reusable even if the whole interface is replaced with another. A conceptual design of the software solution is on Fig. 2.

6 Fig. 2 Conceptual design of the software solution The ParserGenerator (a language parser generator) generates a language parser, which parses the input file and creates a list of commands (program). Executor executes these commands step by step on an application (blue). User communicates with the application and with the ComponentTreeUI interface, where he can start recording, see the application component tree and components information. Recorder is responsible for physical saving of commands into the user chosen output file. 3.1 Graphical User Interface Interaction Language (GUIIL) Input and output files are recorded in a simple domain-specific language - GUIIL. A sentence in this language consists of a command sequence that can be executed on GUI [1]. Language model is based on these basic terms [1]: Command abstract class representing one command that has to be executed on GUI. Defines one operation execute (T component) which executes a command on the given component. DelegateCommand a concrete implementation of the command. Every command has its name (Name class) and a list of its parameters (a field of Parameter class instances). Parameters are separated by comma and they are stated in round brackets. Parameter represents a parameter of a command. Every parameter has its name (Name class) and a value subsistent to this name. Program consists of a Command sequence that can be executed with or without sequencing. Name represents a name of command or parameter. Contains only one string attribute: name. Concrete syntax of the GUIIL generated of the annotated domain model using YAJCo tool has the following shape [1]: Program ::= (Command)* Command ::= DelegateCommand

7 DelegateCommand ::= Name ('(' (Parameter (',' Parameter)*)? ')')? Parameter ::= Name '=' <VALUE> Name ::= <NAME> As an example of a sentence of this language we can use a sequence of commands for a calculator application: 1 2 [-] 3 4 [+] 5 [=] [About Calculator] OK Exit We can see that this notation is easily understandable for normal users and nonprogrammers. 3.2 Component Handlers The software solution has build-in handlers for recording and executing for these basic Java Swing components: AbstractButton, Container, JComboBox, JFileChooser, JMenu, JList, JSpinner, JTabbedPane, JTextComponent and JTree. But we do not assume a closed group of components, therefore the system is open to add handlers for custom components in a simple way extending and implementing build-in handler interfaces and abstract classes: CommandHandler<T> for handling components, Composite<T> for handling containers and RecordSupport<T> for recording. 4 Conclusion In this work a software solution is presented that defines a domain-specific language via user interface. This way it enables component identification based on domain terms that they represent. Based on successful identification of these components it s possible to programmatically automate user interaction with the interface independent on component layout in it. Generated files are easily readable and writable even for non-programmers. In process of identifying these components there are some problems caused by incorrect implementation of components. Therefore we defined basic guidelines for GUI programming. According to the obtained results of GUI domain terms formalization, possible further research areas are computer domain analysis, automatic application usability evaluation, methodical learning process of software using and automatic generation of software manuals. These areas present suitable subjects for a further research. References 1. Porubän, J.: Design and implementation of computer languages. Technical University of Košice, Faculty of Electrical Engineering and Informatics (2008)

8 2. Brooks, P.A., Memon, A.M.: Automated Gui Testing Guided By Usage Profiles. University of Maryland, Department of Computer Science, Coolege Park, Maryland, USA (2007) 3. Fowler, M.: Domain Specific Languages. (2009) 4. Hudak, P.: Modular Demain Specific Languages and Tools. Yale University, Department of Computer Science, New Haven, CT (1998) 5. Mernik, M., Heering, J., Sloane, A.M.: When and how to develop domain-specific languages. ACM Computing Surveys, Macquarie University. 34(4): (2005) 6. Li, K., Wu, M.: Effective GUI Test Automation: Developing an Automated GUI Testing Tool. SYBEX Inc., pp (2005) 7. Memon, A., Pollack, M.E., Soffa, M.L.: Hierarchical GUI Test Case Generation Using Automated Planning. IEEE Transactions on Software Engineering (Vol. 27, No. 2), /01 (2001) 8. Umar, M.: Generating Automated Test Scripts for AltioLive using QF-Test. IntegraSP Ltd., London (2008) 9. Last, M., Kendel, A., Bunke, H.: Artificial Intelligence Methods in Software Testing. Series in Machine Perception Artificial Intelligence (Vol. 56), World Scientific Publishing Co. Pte. Ltd. pp (2004) 10. Chang, T-H., Yeh, T., Miller, R.C.: GUI Testing Using Computer Vision. MIT CSAIL, UMIACS & HCIL, University of Maryland, Atlanta, Georgia, USA (2010) 11. Memon, A.M., Pollack, M.E., Soffa, M.L.: Plan Generation for GUI Testing. Department of Computer Science, University of Pittsburgh, USA (1999) 12. McMaster, S., Memon, A.: Call Stack Coverage for GUI Test-Suite Reduction. Department of Computer Science, University of Maryland, College Park, Maryland, USA (2008) 13. Memon, A., Soffa, M.L., Pollack, M.E.: Coverage Criteria for GUI Testing. Department of Computer Science, University of Maryland, USA (2001) 14. Krueger, C.W.: Software Reuse: Methods, Techniques and Tools. ACM Computing Surveys, pp (1992) 15. Slašťan, M., Mederly, P.: Software Reuse. Lecture for subject Software engineering, University of Komenský in Bratislava. (2004) 16. Nielsen, J.: Top-10 Application-Design Mistakes. Jacob Nielsen s Alertbox, (2008) Appendix: Springer-Author Discount LNCS authors are entitled to a 33.3% discount off all Springer publications. Before placing an order, they should send an to orders-hd-individuals@springer.com, giving full details of their Springer publication, to obtain a so-called token. This token is a number, which must be entered when placing an order via the Internet, in order to obtain the discount. Checklist of Items to be Sent to Volume Editors Komentár [m1]: Toto treba vyhodit c nechat? Komentár [m2]: Toto treba vyhodit c nechat? 1. A final Word or RTF file 2. A final PDF file 3. A copyright form, signed by one author on behalf of all of the authors of the paper 4. A readme giving the name and address of the corresponding author