Framework based on Mobile Augmented Reality for Translating Food Menu in Thai Language to Malay Language

Transcription

1 Vol.7 (2017) No. 1 ISSN: Framework based on Mobile Augmented Reality for Translating Food Menu in Thai Language to Malay Language Muhammad Pu, Nazatul Aini Abd Majid, Bahari Idrus * # Center for Artificial Intelligence Technology (CAIT), Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia (UKM), Bangi, Selangor, Malaysia * bahari@ukm.edu.my Abstract Augmented reality (AR) technology is a technique that combines the real world and the virtual world digitally using mobile devices. Mobile AR technology is expected to help Malaysian tourists who have difficulties to understand the Thai language when visiting the country. Hence, a prototype called ARThaiMalay translator was developed to printed Thai food menu to Malay language. The objective of this study is to design a food menu translation framework from Thai to Malay language based on mobile AR, develop a translator application and to test the effectiveness of the translator application. The prototype consists of three main components which are translation based on optical character recognition (OCR) technology, dictionary development using SQLite database and display data from the local database. Evaluation of the developed application shows its effectiveness to perform translation of Thai text with certain features to Malay language. Keywords malay language; mobile augmented reality; optical character recognition; thai language; translation. I. INTRODUCTION Translating foreign language for a tourist using a mobile phone can prevent undesirable difficulties and allow the tourist to explore the visited country with ease. In visiting Thailand, for example, the use of mobile phone for translating the Thai language to Malay language can help Malaysian tourists to experience the Thai culture effectively. However, developing a real-time translator for Thai language is difficult since the language is complex with 44 voiced consonants, 32 vocals, four stage tones in one paragraph, 10 numbers and five special symbols. The problems in translating Thai language have motivated this research to use mobile AR and OCR to develop a new Thai- Malay translator. AR technology is a technology that allows a virtual image to combine with physical objects on a computer screen or a smartphone in real time [1]. Wagner et al [2] reported the camera phone allows the use of magic lens in carrying out technology-based AR application. For example, margraphy application that uses a mobile phone camera to detect a traditional map marker in identifying and displaying objects in 3D so that the user can clearly understand the information on the map [3]. In addition, OCR technology is a powerful tool to bring information from analog to digital [4]. OCR technology is capable to detect, recognize and extract either printed or written text to understand the writing so that it will help users to scan documents that need further analysis.both technology, AR and OCR has been used for a number of proposed translators including TranslatAR: a mobile augmented reality translator on the Nokia N900 [5], android real time translation for translating Japan language to Indonesia language [6], and application of augmented reality: mobile camera based Bangla text detection and translation [7]. These works have shown the potential of a mobile AR and OCR based apps to complex languages such as non-roman languages. Therefore, this paper presents a model to develop a mobile AR and OCR based app that can be used to complex Thai language to Malay language. This paper is organized as follows: Section 2 presents related work about Thai language and translation and also described about AR for translation. Section 3 discussed the model of ARThai Malay translator. Finally, section 4 presents the conclusion. A. Thai Language and Translation Thai text detection is important for the procedures to be used as language translation application using mobile smartphones in assisting foreign tourists [8]. Research of Tanprasert and Sae-Tang [9], type-written character recognition Thai language is complex and difficult. The letters in the Thai language has arches, zigzag, notched and some head and circle. There are also some letters written Thai is almost similar to each other as shown in Fig

2 Fig. 1. Examples of some Thai characters that are almost the same According Tanprasert and Koanantakool [10], writing Thai language is starting from left to right with the line of the page from top to bottom. Every word in Thai language has a four-level tones of voice. The meaning of the word depends on the tone of those words. Fig. 2 shows that there are four stages in the writing of Thai language. This stages cause Thai letter recognition process difficult. Although there are online dictionary for translating Thai Language to Malay language such as Bing and Google Translate, it is difficult for tourists to type in the words in Thai Language. Furthermore, some words were not accurately d by Bing and Google. Therefore, there is a need to develop a new translator that can directly capture Thai printed texts without typing and it into Malay language. Mobile AR with OCR technology is the viable option to develop the translator. C. Automatic text detection for mobile augmented reality translation Fragoso et al. [5] developed a mobile AR translation system, using a smartphone's camera and touchscreen that can automatically detect, extract, track and a text that has been tapped earlier, and finally present the translation as a live AR overlay. The application starts by scanning a marker text and the translation results will be displayed in the form of AR display. An algorithm was developed to locate and orient the text precisely on streaming directly. The ICDAR 2003 detection dataset was used to evaluate the system. In the system, OCR technique was used to recognize the patterns of the text. Tesseract OCR is easy to use because it can be compiled on any target platform. The input language was detected automatically by Google Translate, and the desired output language can be selected by the user in the GUI of the system. Fig. 3 shows the main steps in the system. Petter et al. [14] developed a detection algorithm for text translation system that was designed for AR based application. The aim of their study was to improve the capabilities of translation application that was built by Fragoso et al. [5] where the proposed application can automatically find the text for translation without tapping on the mobile phone screen. The algorithm consists of three main steps as shown in Fig. 4. Fig. 2. Four stages in the writing of Thai words B. Models of Translation Application Du et al. [11] explained that the two examples of very famous translation app is Pleco and Word Lens. Pleco application [12], is the Chinese dictionary application that designed to Chinese into English without requiring an connection. Pleco application can be used on ios and Android platforms. While Word Lens [13] is a smartphone app that can signboards in English to Spanish using camera phone with the results of translations will be directly displayed on the phone screen. However, both of these applications work properly if the translation involve the words one by one. Word Lens uses the camera of a smartphone to scan the text and identify the text. The text will be d and displayed on mobile devices according to the selected language. The original text is displayed and the translation is done in real time without an connection. After the text translation, the d word is displayed as the original image was captured. Word Lens is an AR application that can read printed text using OCR technology. Translation applications based mobile AR technology have also been studied by several researchers, including [5], [7], [6] and [14]. Fig. 3. Main steps in TranslatAR [5] 154

3 TABLE I RESEARCH ON USING AR WITH OCR FOR TRANSLATION Fig. 4. Three steps in text detection algorithm [14] D. Mobile augmented reality translation for Japan- Indonesia and Bangla-English Application of mobile augmented reality for Japan to English was developed by Nugroho et al. [6] by using engine (Tesseract) and Google API for Android operating system. Meanwhile, an application for translating Bangla to English was developed by Mahbub-Uz-Zaman & Islam [7] that also used similar techniques which are engine (Tesseract) and Google API for Android operating system. The framework for both applications can be visualized in several steps as shown Fig. 5. This framework put less focus in text detection technique as compared to the framework in Fig. text detection technique as compared to the framework in Fig. 3 by Fragoso et al. [5]. As shown in Table I, mobile AR has been used to languages using OCR but none of the apps focus on translating Thai language to Malay language. Most of the apps in Table 1 must use connection and Google API for translation. Since some texts cannot be accurately d using Google translation, especially three words, this research proposed a translator based on mobile AR with OCR, and used a local database, SQLite, as part of the translation. Research Implementation Technique TranslatAR: A Mobile AR Translator on the Nokia N900 [5] Automatic text detection for mobile AR translation [14] Realtime Android applications Translation to Japanese to Indonesian [6] Application of AR Mobile camera based Bangla text detection and translation [7] The text area detection, video frames orientation and works with the press of a button. Displays results directly on phone screen. Using a mobile phone camera and the touch screen. User simply tap on the words that need to be d, and the results appear in the form of AR Design applications for translation using a pre-identified programming language. The results appear in the form of AR Users do not need to type a word to. The text is captured by a mobile phone camera. The test is d and displayed on the mobile phone screen. - Using standard off the shelf as a library to support multiple language - OCR and AR - Use ICDAR 2003 dataset - Using a mobile phone camera and touchscreen - OCR - OCR - Google API - Mobile phone camera and touchscreen - Tesseract OCR - Google API - Open source application based on Android and OCR Advantages / disadvantages Using the words - Cannot Thai - Using the words - Translate Japanese to Indonesia only - Using the words - Translating from English to Bangla language only. - Using the words Fig. 5. Main steps for mobile augmented reality translation for Japan- Indonesia and Bangla-English [6] II. MATERIAL AND METHOD Development architecture for ARThaiMalay translator involves four main components, which are users, mobile phone, markers and displays. Fig. 6 shows an overview of the system architecture in application development Translator ARThaiMalay. Users have two important roles, which are scan and verify the results of the scan. The use of Android's camera is to scan and recognize text contained in the marker. Identification of the text is done by scanning the markers used mobile phone camera. The marker used is the text of Thai with font size 16 Times New Roman and the contrast or brightness of the image is good. The results of Malay translation will be displayed on the mobile phone screen. The ARThaiMalay Translator Application Framework consists of three main modules: 1) translation, 2) dictionary development and 3) display. An overview of the framework shown in Fig. 7. Framework for translation is divided into three components, namely capturing, tracking and rendering. The capturing component captures markers which are printed food menu in Thai language using mobile phone camera. The results of the scanning can be seen in Fig

4 Fig. 6. Development architecture for ARThaiMalay translator The tracking component processes the captured images with two main steps. Firstly, the captured image is decoded using an OCR engine which is tesseract. The result of the recognition of the Thai language using the OCR technique is passed to the translation module. Secondly, the result which is the Thai text is d to Malay words by using local database, SQLite. The identified Thai texts are matched with Thai word contained in the database and d into Malay words. This study only uses a tracking technique that is based on the vision camera. If the translation for the Thai word has been found, the result will be sent to the rendering component. The rendering component will overlay the AR output which is the translation of the Thai words in Malay language. Framework for dictionary development provides functions to add menu in Thai language and their translation in Malay Language and save them into the database of the application (Fig. 9). All the menu lists are stored in a SQLite database. An example of data for the database listed in Table II. Framework for display provides function to display all the lists of Thai and Malay language menu on the screen of the mobile phone. Users will be able to view the food menu in Thai and the meanings in Malay on the mobile screen (Fig. 10). Fig. 7. Framework for ARThaiMalay Translator Fig. 9. Interface for developing a dictionary TABLE II EXAMPLE OF INPUT FOR DICTIONARY DEVELOPMENT No Menu in Thai Menu in Malay 1 ปลาทอด Ikan Goreng 2 ข าวเปล า Nasi Putih 3 ต มย า Tom yam Fig. 8. Results of the scanning 156

5 Fig. 12. Translation for three words of Thai Language TABLE III LIST OF MENU THAT CAN BE ACCURATELY TRANSLATED USING ARTHAIMALAY TRANSLATOR Fig. 10. Interface for displaying information III. RESULTS AND DISCUSSION One of a popular Thai menu is fried sea bass or in Thai is ปลา กระพง ทอด. This menu is not accurately d by Google or Bing (Fig. 11) because the correct translation is ikan siakap goreng. Furthermore, the tourists need to type in the words in Thai Language by using this online dictionary. A prototype of ARThaiMalay translator developed for android phone can directly capture printed Thai words and it to Malay language without typing the Thai words. The result is shown in Fig. 12. This shows the use of the prototype based on the framework in Fig. 7 is a viable option to develop an effective translator for Thai to Malay language. As shown in Table III, the ARThaiMalay translator can Thai food menu to Malay language effectively although with different light setting (e.g. Fig. 12, 13 and 14). The translation accuracy for the listed words in Table 3 is 100%. One word ปลา Ikan Two words ปลา ทอด Ikan Goreng Three words ปลา ทอด กรอบ Ikan Goreng Cryspy ปลา Ikan ปลา เผา Ikan Bakar ปลา เผา แซบ Ikan Bakar Pedas หอย Kerang หอย ทอด Kerang Goreng หอย ทอด กระเท ยม Kerang Goreng Bawang สต ม Stim สาม รส Tiga Rasa สต ม สาม รส Stim Tiga Rasa กระพง Siakap กระพง ทอด Siakap Goreng ปลา กระพง ทอด Ikan Siakap Goreng สาม Tiga สาม รส Tiga Rasa ปลา สาม รส Ikan Tiga Rasa ป ก Sayap ป ก ไก Sayap Ayam ป ก ไก ทอด Sayap Ayam Goreng Fig. 11. Translation from Bing translator Fig. 13. Translation for one word of Thai Language 157

6 Fig. 14. Translation for two words of Thai Language Fig. 17. Unsuccessful translation for three words of Thai Language However, there is also a list of food menu that cannot be d accurately using ARThaiMalay translator (e.g Fig 15, 16 and 17) as listed in Table IV. This might be related to the words that involve the fourth stage in Thai language (Fig. 2). Future work should focus on text detection so the every stage in Thai language can be detected. TABLE III LIST OF MENU THAT CANNOT BE TRANSLATED ACCURATELY USING ARTHAIMALAY TRANSLATOR One word Two words Three words ผ ด goreng ข าว ผ ด Nasi goreng ข าว ผ ด ป Nasi goring ketam ข าว nasi ไข เจ ยว Telur dada ข าว ไข เจ ยว Nasi telur dada ย า kerabu ย า ปลา Kerabu ikan ย า ปลา ด ก Kerabu ikan keli สล ด salad ท น า tuna สล ด ท น า Salad ikan tuna เน อ daging เน อ แดด Daging kering เน อ แดด เด ยว Daging kering lada hitam ต ม rebus ต ม ย า tomyam ต ม ย า ปลา Tomyam ikan ลาป Kerabu daging ลาป เน อ Kerabu daging ลาป เน อ เผา Kerabu daging bakar Fig. 15. Unsuccessful translation for one word of Thai Language IV. CONCLUSIONS A framework based on mobile AR and OCR was developed for translating Thai language to Malay language. This translation could play important role in assisting Malaysian tourists to explore Thailand with ease. The difficulties in typing a complex non-roman language like Thai could be solved by using AR so that the input for the translation process can be captured using camera. The important elements in this ARThaiMalay translator are as follows: Building local dictionary in the phone for food menu so that it can be extended to cover various type of food menu; Translating the Thai words without the needs of connection because the application using local dictionary via SQLite. This porotype of ARThaiMalay translator has shown promising results using a combination Fig. 16. Unsuccessful translation for two words of Thai Language 158

7 of mobile AR, OCR and local database. It is expected to improve the process of translation of non-roman language using mobile phone without connection. ACKNOWLEDGMENT We acknowledge our AP grants for funding this research. REFERENCES [1] Z. Feng, et al., "Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR," in Mixed and Augmented Reality, ISMAR th IEEE/ACM International Symposium on, 2008, pp [2] D. Wagner, et al., "Multiple target detection and tracking with guaranteed framerates on mobile phones," in Mixed and Augmented Reality, ISMAR th IEEE International Symposium on, 2009, pp [3] A. Choi, et al., "margraphy: Mobile AR-based Dynamic Information Visualization," in Ubiquitous Virtual Reality (ISUVR), 2011 International Symposium on, 2011, pp [4] M. Elmore and M. Martonosi. (2008, A Morphological Image Preprocessing Suite for Ocr on Natural Scene Images. Available: more/melmore_dmp.pdf [5] V. Fragoso, et al., "TranslatAR: A mobile augmented reality translator," in Applications of Computer Vision (WACV), 2011 IEEE Workshop on, 2011, pp [6] O. Nugroho Kusumo, et al., "Pengembangan Aplikasi Android Realtime Translation Untuk Menerjemahkan Bahasa Jepang Ke Bahasa Indonesia," BINUS., [7] S. Mahbub-Uz-Zaman and T. Islam, "Application of Augmented Reality: Mobile Camera Based Bangla Text Detection and Translation.," BRAC University., [8] K. Woraratpanya, et al., "Improved Thai text detection from natural scenes," in Information Technology and Electrical Engineering (ICITEE), 2013 International Conference on, 2013, pp [9] C. Tanprasert and S. Sae-Tang, "Thai type style recognition," in Circuits and Systems, ISCAS '99. Proceedings of the 1999 IEEE International Symposium on, 1999, pp vol.4. [10] C. Tanprasert and T. Koanantakool, "Thai OCR: a neural network application," in TENCON '96. Proceedings., 1996 IEEE TENCON. Digital Signal Processing Applications, 1996, pp vol.1. [11] J. Du, et al., "Snap and Translate Using Windows Phone," in 2011 International Conference on Document Analysis and Recognition, 2011, pp [12] Pleco. (2015). Available: [13] Word Lens. (2016). Available: [14] M. Petter, et al., "Automatic text detection for mobile augmented reality translation," in Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on, 2011, pp