The ioxp worker assistance system an overview

Transcription

1 The ioxp worker assistance system an overview 1

2 Cognitive Augmented Reality The software developed by ioxp gives industrial workers a step-by-step instruction for the execution of manual tasks and processes. It uses an example video of the correctly executed procedure as a ground truth. The software extracts the workflow and process knowledge directly from the video material by analyzing and interpreting the recorded sequence in keeping with the motto content creation through observation. The workflow knowledge alongside the video footage enables the system to create documentation content and step-by-step instructions automatically. The content can be shown on several mobile devices including smartphones, tablets and smart glasses. Depending on the requirements and the hardware setup the information can be deployed and viewed as a picture-based, printable document, voicecontrolled tutorial videos or a full feedback augmented reality assistance system. For almost all situations when a worker needs guidance in a specific task, the ioxp system provides a suitable assistance system based on the single reference video. The fundamental technology behind the ioxp software is the algorithmic backend for the recognition and extraction of actions from video streams. This technology has been developed at the German Research Center for Artificial Intelligence (DFKI) and ioxp. It allows the creation of documentation content and explanatory instructions from a sample video. Together with ergonomic hardware like smart glasses or head mounted displays with a camera, the content creation becomes a side-product of every days work. The expert just pushes record and the system does the rest. After a short processing time an individual, easy-to-use augmented reality instruction has been generated automatically. Figure 1: Automatic content creation pipeline used by the ioxp system Another user can now easily display the information on another device to be guided through the process. Further the content can be used for training, to support technical authors or even for process management. In most scenarios the system can also be used to actively guide the user through the work process. A feedback system shows potentials errors or misuse. That way the error rate as well as quality assurance and work safety can be improved by using the ioxp system 2

3 The ioxp system The central element of the ioxp system is the knowledge base. It is a database that comprises of the reference videos and all additional information that has been extracted or has been added manually. This contains, among others, the structural information, temporal succession, descriptive illustrations of the actions, supplementary annotations such as text labels or animations and all information the tracking system needs to superimpose the camera image with virtual information. Based on this, different output formats can be deployed and used as documentation or assistance system. Figure 2: The ioxp system input / output / data management iopages The automatically extracted workflow knowledge can be deployed as an easy to understand, picture-based step-by-step manual. This view can be printed or used on mobile devices like tablets or smartphones. The individual tasks within the sequence of actions are shown in a picture sequence. This results in a very intuitive and almost language independent documentation. Figure 1: Printable step-by-step guides 3

4 iomanual The software extracts a representative and explanatory video sequence that is stored in the knowledge base. At runtime, the worker can view these sequences in the form of a stepwise video tutorial to watch on a mobile device. Each step is repeated until the worker tells the system to move on. Using smartphones or tablets the user can interact via voice commands or touch input. Figure 3: Service technician s app with utility films. This form of user interaction allows the worker to concentrate on the current task, even when performing two-handed tasks. Alongside the video content the assistance system can contain additional material like sketches, plans or any kind of readable document like technical drafts. Such content can be easily accessed by the user as it is seamlessly integrated into the ioxp system. If the displaying device is equipped with a camera it can be used to recognize the context of interest. Whether it is the location, the specific machine or single parts, the ioxp system can recognize it using the camera and the system delivers only the content in the current context. This allows a very intuitive and effective way to deal with large amounts of information and with knowledge bases covering the documentation of the complete shop floor. ioassist When using a display device with an attached camera, like AR glasses, a tablet or smartphone, a live recognition of the hand movement and the tasks can be made which is analog to the reference recording. This allows the system, besides the correctly placed, superimposed content, to give a visual feedback about the user behavior. Figure 4: Visual Feedback during runtime The user gets a visual feedback whether the action was performed according to the reference. The hands of the user are digitally colored when residing in the field of view. When they are at the correct location of the activity, they are colored green, otherwise red. A variance analysis is done per action to validate the result. The user gets an explicit visual feedback and is Figure 5: Feedback on performed actions automatically guided to the next step in case of a successful verification. This allows self-monitoring while performing the sequence of actions which drastically reduces the source of errors for perfunctory tasks. Moreover, depending on the situation, it can increase the industrial safety when working in danger zones by warning before performing hazardous actions. 4

5 Backend The basis of ioxp s step-by-step manuals is its database and service backend. The recorded data along with all additional information is structured in the filesystem of the database system. The backend provides interfaces to expand the database content, retrieve content and allows full customization of any aspect of the AR system. Ordinarily, the video processing pipeline of the backend will be used to provide new content to the database. The cognitive AR system allows to fully automatically create new content by providing a reference recording of the sequence of actions. It utilizes computer vision algorithms and machine learning techniques to gather all required information to create descriptive illustrations of each action as well as tracking information for the AR system. A short overview of the automatic steps can be found in Figure. The backend allows administration through a web interface. Cognitive Creator The cognitive creator allows to view the stored data in a structured manner and allows post-processing of the structure itself and postprocessing for each step. The process view shows the relationship between actions similar to a business process modeling language or an expert system. Processes can be restructured easily by altering the connections between the actions Figure 6: Sequence of actions in the process view the AR system will immediately know how the transition to the new action has to be handled. This allows to quickly adapt the process to the current situation without the need to re-record any video material. In case a new workflow follows an already recorded part, it can be appended right there. Complex decision trees can be modeled equally. Besides, the cognitive creator gives the opportunity to add supplementary information per chapter. For example, helpful animations can be added to highlight certain aspects - like superimposing a virtual screwdriver on top of a screw in the camera view. All virtual information can be placed without having an insight in 3D content creation or the underlying tracking system needed to superimpose real world objects with virtual information. 5