Using Apache FreeMarker Template Language in BotML

Transcription

1 Oracle Intelligent Bots TechExchange Article. Using Apache FreeMarker Template Language in BotML Frank Nimphius, January 2018 As a bot designer, how do you deal with the problem of users providing a "Yes" where your data logic expects a "yes"? Or, how do you replace a null value with a default string? And do you know how to trim blank space from a user input? What about sorting arrays? Rounding numbers? Well, as an Oracle Intelligent Bots expert, custom component certainly came to mind. However, there is another solution for this, one that doesn't require a HTTP roundtrip to a remote component service, and it is also easier to implement. Oracle Intelligent Bots dialog flow uses Apache FreeMarker in BotML to access data values from variables. However, there is a lot more that bot designers can use Apache FreeMarker Template Language (FTL) ( for. This article explains how to use ApacheFreeMarker in BotML and shows examples for selected built-in FTL expressions. Change Log Change Date Change Novermber 2017 Initial January 2018 Added date handling 1 Oracle Intelligent Bots TechExchange

2 Apache FreeMarker in Oracle Intelligent Bots The Apache FreeMarker Template Language is used in BotML to access values stored in variables. The syntax for this is ${ }. For example, to read a value saved in a context variable, you use an expression like this ${variable_name.value} To use other FTL features like string and array operations, you use the similar expression followed by a question mark and the name of a function ${variable.value?ftl_function} It is also possible to concatenate FTL expressions, for example if you wanted to first capitalize and then trim a string value. ${variable.value?ftl_function1?ftl_function2} Note that Oracle Intelligents Bots works with Apache FreeMarker expressions that are appended to an expression with a question mark as well a tag style syntax like <#if> </#if>. However, tag style syntax should be used with care and the support of it may change over Oracle Intelligent Bots releases. For this reason, this article only covers expressions appended with a question mark. Also note that not all FreeMarker built-in expressions work in the context of BotML. If you experience problems with an expression and don't know if this is supposed to work, please post questions to the Oracle Developer Community forum for Oracle Mobile Cloud and Chatbots: Working with Strings Apache FreeMarker provides a large set of string operations including upper and lower case transformation, captitalization, string replacement, truncation, string detection and finding the index of a contained string. The expression in the examples in this section uses a variable "tester" of type string that has its value set to "hello world " (with three trailing blank characters). Here's the BotML code that defines the variable in Oracle Intelligent Bots dialog flow. context: variables: tester: "string" states: setvariable: 2

3 variable: "tester" value: "hello world " Note: The Apache FTL built-in expressions can be used with a variety of components in Oracle Intelligent Bots and can be used for setting the component value, prompt and labels. The System.SetVariable and System.Output componets were used for testing when writing this article. The examples in the table below don't cover all of the built-in string operations available in Apache FreeMarker. However, you will learn enough to be able to also work with those expressions that are not explicitly introduced in the following. See for the full list of FTL operations. Example printvariable: component: "System.Output" Description Prints the value of the variable tester ("hello world " in case of the above BotML) or "no string found" if the variable has no value is set. text: "${tester.value! 'no string found'}" transitions: {} ${tester.value?capitalize} Returns "Hello World " ${tester.value?last_index_of('orld')} Returns 7 ${tester.value?left_pad(3,'_')} Returns " hello world " ${tester.value?length} Returns 14 ${tester.value?upper_case} Returns "HELLO WORLD " ${tester.value?replace('world', 'friends')} Returns "hello friends " ${tester.value?remove_beginning('hello')} Returns "world " ${tester.value?trim} ${tester.value?ensure_starts_with('brave new ')} ${tester.value?ensure_ends_with(' my friend')} ${tester.value?contains('world')?string ('You said world', 'You did not say world')} Returns "hello world" Returns "brave new hello world" Returns "hello world my friend" Returns "You said world". The contains('world') expressions returns the boolean value true or false that then is replaced with a string using the string ('string1', 3

4 'string2') function ${tester.value?ends_with('world')?string ('Ends with world', 'Doesn't end with world')} ${tester.value?starts_with('world')?string ('Starts with world', 'Doesn't start with world')} ${tester.value?matches('^([^0-9]*)$') ${tester.value?matches('^([^0-9]*)$')?string Returns "Ends with world" Returns "Doesn't start with world" Regular expresion returns true or false dependent on whether the value contains a number (in which case the boolean value is returned as false). In our example the boolean value true is returned. Same as before, but this time "true" is returned as a string. The matches('regular expression') function returns true or false as boolean types. To print true or false in a System.Output component, you use?string to perform a to-string conversion. Table 1: String Operation Examples Note: Regular expressions cannot be used with expressions that return groups. You want to use expressions that return a single match or no match as in the example shown in the table. Use the matches( ) function with care if you are not an expert user of RegularExpressions. Advanced Examples Let s put some of the basic examples into practice and show how Apache FreeMarker built-in string functionality can help you solving real world bot problems. Modify User Input for Intent Recognition The language recognition models in Oracle Intelligent Bots to some degree are case sensitive. Dependent on the bot use, chances are that an intent can be resolved with a higher confidence level if user input provided all lower case input. Let s take "may" as an example. Using "may" with a capital "M" could be interpreted as "May", the month, during language processing. Unless May, the month, is really what you wanted, a false interpretation could negatively influence the confidence level of the intent recognition. Another example for when you want to change the case of the user provided input is for entities recognized in a sentence. A user ordering a pizza may write pizza as "pizza", "Pizza", "PIZZA", PiZZa". So instead of catching all possible case variations as synonyms in the entity definition it may be better to change the user input to all upper case or all lower case before processing the intent. A use case for FTL in those cases could be to transform the user entered string to lower case before passing it to the intent engine. Here's an example BotML code for this. 4

5 getintent: component: "System.Text" prompt: "Hi, I am a the Pizza Palace bot. How can I help?" variable: "userstring" transitions: {} tolowercase: variable: "userstring" value: "${userstring.value?lower_case}" transitions: {} intent: component: "System.Intent" variable: "iresult" confidencethreshold: 0.8 sourcevariable: "userstring" transitions: actions: orderpizza: "orderpizza" cancelorder: "cancelorder" unresolvedintent: "handleunresolved" To implement this you first ask the user for her input. The System.Text component saves the user input in the "userstring" variable for later use. In the following, the Sytem.SetVariable uses FTL to change the case of the user input string to lower case and saves the modified string to the same "userstring" variable. Finally, the "userstring" variable is referenced by the System.Intent component using the "sourcevariable" property to run the modified user string against the intent engine. Note: The success of the intent recognition primarily depends on the quality of utterances provided when training the intent models. Converting strings to a specific case is not a general optimization recommendation. Optimize User Input for Use with System.Switcher Another component use that can be simplified with FTL is the System.Switcher. switch: component: "System.Switch" variable: "choice" values: - "wine" - "beer" 5

6 transitions: actions: wine: "serverwine" beer: "servebeer" NONE: "servewater" The BotML code snippet above calls a state within a dialog flow dependent on the user input (saved in the "choice" variable") being "wine" or "beer". If the user input is through a System.Text component, then chances are that she may type "Wine", "WINE", "WiNe" instead, in which case she gets served water (you see how ugly this could get!). So how to solve this problem? Instead of adding all possible variations for wine and beer to the System.Switch BotML definitions, FTL can be used to change the user input to upper case. setvariable: variable: "choice" value: "${choice.value?upper_case}" switch: component: "System.Switch" variable: "choice" values: - "WINE" - "BEER" transitions: actions: WINE: "servewine" BEER: "serverbeer" NONE: "servewater" Concatenating FTL Expressions Lets construct a simple case for this sample. Flight numbers have an airline code, like 'LH' for Lufthansa or 'BA' for British Airways followed by a code identifying the flight. So let s assume the booking bot for United Airlines only needs the actual flight number and not the airline code. Since you don't know what the user enters as the flight number, you want to ensure the airline code is removed. normalizeflightnumber: variable: "flight" value: "${flight.value?trim?lower_case?remove_beginning('ua ')?remove_beginning('ua')}" In the code above, leading and trailing blank characters are removed from the user input before it is converted to lower case. Finally the strings 'ua ' or 'ua' are removed. If the user provided flight number was UA 1234 then the variable "flight" now holds "1234" as the flight number" 6

7 I agree that this example looks like a poor man's approach to regular expression. However, the message of this sample is that you can concatenate FTL expressions in BotML. Working with Numbers The number of FTL expressions that can be used with numeric values is small compared to expressions that work on strings and arrays. See for the list of number specific built-in expressions. However, let s have a look at these. As before, let s start defining the test data to use in the examples. context: variables: negativevalue: "float" positivevalue: "float" states: setnegativevalue: variable: "negativevalue" value: -2.5 setpositivevalue: variable: "positivevalue" value: With this, let s have a look at some samples for built-in expressions in FTL that operate on numeric values. Note: Expressions can be used with different bot components, including System.Output and System.SetVariable. Example ${negativevalue.value?abs} ${negativevalue.value?abs?string.percent} ${negativevalue.value?round} Description Returns 2.5, turning the negative numeric value into a positive value Returns 250%. First it changes the negative value to a positive. Then it converts it into percent, implicitly multiplying the value with 100 Returns -2. Rounds to the nearest whole number. If the number ends with.5, then it rounds upwards 7

8 ${positivevalue.value?round} ${positivevalue.value?floor} ${positivevalue.value?ceiling} ${negativevalue.value?abs?round?lower_abc} ${negativevalue.value?abs?round?upper_abc} ${positivevalue.value?is_infinite?string} Returns 1. Rounds to the nearest whole number. If the number ends with.5, then it rounds upwards Returns 0. Rounds downwards. Returns 1. Rounds upwards. Returns "c". The negative value is turned into a positive and then rounded to 3. As "c" is the 3 rd character of the alphabet, "c" is returned. Returns "C". The negative value is turned into a positive and then rounded to 3. As "C" is the 3 rd character of the alphabet, "C" is returned. Returns "false" as the float value is not infinite according to IEEE 754. ${positivevalue.value?string['###.##']} Prints 0.51 Note that without?string at the end, the returned value would be of type boolean. ${positivevalue.value?string['###.##%']} Prints 51%, adding a percentage character after multiplying the value with 100 ${positivevalue.value?string['##.###\u00a4']} Prints 0.51 $ ${positivevalue.value?string['##.###\u20ac']} Prints 0.51 ${positivevalue.value?string['##.###\u00a3']} Prints 0.51 Get more: Table 2: Numeric Operation Samples Note: Apache FreeMarker uses and extends the Java decimal format patterns documented here: You can read about the extensions in the Apache FreeMarker documentation Working with Arrays 8

9 In Apache FreeMarker, arrays are referred to as "sequences". The FreeMarker template language provides built-in expressions that determine the size of an array, to sort arrays, find content in an array and many more. Before explaining how to work with arrays in BotML, let s look at where in Oracle Intelligent Bots you find arrays. The bot natural language-processing returns entities extracted from a user input string and the list of recognized intents as arrays exposed on the "iresult" variable. For example, ${iresult.value.entitymatches['name of entity']} returns an array of entities found in a user string passed to the System.Intent component. Similarly, ${iresult.value.intentmatches.summary} returns an array of intents and the confidence for a given user input. On top of this, you can save arrays in context and user scope variables using FTL in BotML or a custom component. Note: An in depth discussion of creating arrays from custom components is beyond the scope of this article. Creating Arrays in BotML Creating arrays in BotML makes sense if you quickly need to create mockup data to test bot functionality or if you want to pre-define a data structure for objects to store beyond user sessions (using variables in user scope). No surprise that FTL can help with that. The following BotML code creates two arrays. The "person" variable holds an array of Oracle staff you may have met at conferences and trainings, or that you have seen in Oracle Intelligent Bots videos on Youtube. context: variables: person: "string" colors: "string" setperson: variable: "person" value: - firstname: "Frank" lastname: "Nimphius" - firstname: "Grant" lastname: "Ronald" - firstname: "Geoff" lastname: "Poremba" - firstname: "Marcelo" lastname: "Jabali" 9

10 The "colors" variable list holds color names in an array of of strings. setcolors: variable: "colors" value: - "yellow" - "blue" - "red" - "black" - "white" - "green" Both arrays: person and colors, are used in the following to explore Apache Framemarker array expressions in BotML. FTL Array Operation Array expressions can be used with various bot components, including System.Output and System.SetVariable. So let s have a quick look at some examples in the table below Example ${person.value?size?number} ${person.value[1].firstname} ${person.value[1].firstname!'unknown'} ${person.value?first.firstname} ${person.value?last.firstname} Description Returns the number 4, representing the size of the person array Returns "Grant" as the value of firstname property of the second entry in person array Same as above. Prints "unknown" if the firstname property is missing Returns "Frank" as the value of the firstname property of the first entry in the person array. This expression does not use the array index to find the person element. Same as above but returns the firstname property value of the last person element, which is "Marcelo" 10

11 ${person.value?sort_by('lastname') [0].firstName} ${person.value?sort_by('lastname')?reverse[0].firstname} ${colors.value?seq_index_of('red')} ${colors.value?seq_last_index_of('red')} ${colors.value?join(',')} ${colors.value?seq_contains('red')? string('yes','no')} ${colors.value?sort?join(',')} ${colors.value?sort?reverse?join(',')} Sorts the person array by the lastname property in ascending order. It then prints the value of the firstname property, which is "Marcelo" in the "person" example Note: Unless you save the sorted array in a variable using System.SetVariable, the data is sorted for the single request Same as above, but this times the sorting order for the lastname property is descending. The printed first name is "Grant" Returns 2, which is the array index of "red" Returns the index of the last occurrence of "red" in the array Returns the colors array as a string "yellow,blue,red,black,white,green" Returns "yes" because the string "red" is contained in the array. Note that?seq_contains returns a boolean value. This value is then replaced by a string using the?string(' ',' ') expression Returns colors as string in ascended sort order: black, blue,green,red,white, yellow Returns colors as string in descended order Table 3: Array Operation Examples Advanced Example: Iterating Arrays When working with entities detected in a user input string you may not know how many entities there are. Using Apache FreeMarker Template Language, you can determine the size of an array and then iterate over its elements. The BotML code below shows this for the array held in the "person" variable context: variables: person: "string" count: "int" runner: "int" states: #set the "count" variable to the size of the array setarraycount: 11

12 variable: "count" value: "${person.value?size?number}" #set the value of the "runner" variable to 0 initializerunner: variable: "runner" value: 0 #use the runner variable as the index of the person to print printpersonname: component: "System.Output" text: "Person ${runner.value +1} is ${person.value[runner.value].firstname} ${person.value[runner.value].lastname}" keepturn: true #increase the runner increaserunner: variable: "runner" value: ${runner.value + 1} #if the value of the runner variable is not equal the size of the #array size, repeat. If equal, go to the "exit" state conditionequals: component: "System.ConditionEquals" variable: "runner" value: "${count.value}" transitions: actions: equal: "exit" notequal: "printpersonname" As the array wasn't sorted, the code above prints 12

13 Working with Dates Handling dates is a common usecase when building bots. As before, you can use the date FTL expressions in output components as well as the "setvariable" component. The examples in the table below don't cover all of the built-in date operations available in Apache FreeMarker. See for the full list of FTL operations. Example PrintToday: component: "System.Output" Description Prints the current date in default format like "Jan 17, 2018" text: "${.now?date}" keepturn: false "${.now?time} "${.now?datetime} ${(.now?long )?number_to_date } ${.now?string.full} ${.now?string.long} ${.now?string.short} ${.now?string.medium} Prints time of the day, e.g. 5:35:09 PM Prints date of the day and time, e.g. Jan 17, :36:13 PM Adds 24 hours to the current date and prints "Jan 18, 2018" for when call was made on Jan 17, 2018 Converts date to string with the following formatted output: Wednesday, January 17, :35:12 PM UTC Converts date to string with the following formatted output: January 17, :36:47 PM UTC Converts date to string with the following formatted output: 1/17/18 6:37 PM Converts date to string with the following 13

14 formatted output: Jan 17, :38:35 PM ${date_variable?datetime?string.short} Converts date to string with the following formatted output: 1/17/18 6:37 PM. The "datetime" flag helps freemarker to tell that the variable holds a date that contains date and time information. Similar you can use "date" or "time" to indicate if the date only contains the date or time. Using this format avoids errors. ${datevar.value.date?long?number_to_date?date?string.short} Converts date from entity extraction to a string with the following formatted output: 1/17/18 The "date" flag helps freemarker to tell that the variable holds a date that contains no time information. Using this format avoids errors. ${.now?string.iso} Prints the date in the ISO 8601 standard: T18:54:01.129Z ${ datevar.value.date?long?number_to_date?string.medium} Converts date drived from entity extraction to string with the following formatted output: Jan 17, Note that all other formats like "full","short","long" and "iso" work the same with dates derived from entity extraction. ${.now?string['dd.mm.yyyy, HH:mm']} Prints the date in custom format, e.g , 18:58 ${ datevar.value.date?long?number_to_date?string['dd.mm.yyyy']} Prints date derived from entity in a custom format ${.now?string['yyyy']} Extracts the year from a date. You can do the same with day, month and time ${ datevar.value.date?long?number_to_date?string['yyyy']} Extracts the year from a date derived from enity extraction. You can do the same with day and month Example: Working with Dates Extracted from User Input 14

15 Usually you don't start from today in a bot conversation but look for date occurences in the user input. Lets take a bot that manages appointments. Lets assume a user asks ""Can you arrange a meeting with Mr. Higgs a day later than tomorrow?" The date reference in this sentence is tomorrow. To make it a day later than tomorrow, we would need to add 24 hours to the date finding. Here is how the use case looks in BotML. context: variables: iresult: "nlpresult" thedate : "DATE" states: intent: component: "System.Intent" variable: "iresult" confidencethreshold: 0.4 transitions: actions: unresolvedintent: "dunno" Appointment: "printtoday" printtoday: component: "System.Output" text: "Today is: ${.now}" keepturn: true startappointement: variable: "thedate" value: "${iresult.value.entitymatches['date'][0]}" printdatefound: component: "System.Output" text: "Date found is: ${thedate.value.date}" keepturn: true printdayafter: component: "System.Output" text: "A day later is ${(thedate.value.date?long )?number_to_date}" transistions: return: "done" 15

16 How-to Set a Default Date if a Date Could Not Be Extracted From a User Sentence Continuing with the sample in the previous section, what if the user asks "Can you arrange a meeting with Mr. Higgs?". One option to handle the missing date information is to ask the user for when she wants to meet that person. Another is to set a default date, for example, the current date. Good for you to know that both approaches follow the same implementation principle. For this article we will use the option of a default date set to the current date. context: variables: iresult: "nlpresult" thedate : "DATE" #need extra variable for default date input defaultdateinput: "string" states: #try to extract date information from user sentence startappointement: variable: "thedate" value: "${iresult.value.entitymatches['date'][0]}" #set default date if none found conditionequals: component: "System.ConditionEquals" variable: "thedate" value: null transitions: actions: equal: "setdefaultdate" notequal: "printdatefound" 16

17 setdefaultdate: variable: "defaultdateinput" value: "${.now?datetime?string.long}" matchentity: component: "System.MatchEntity" sourcevariable: "defaultdateinput" variable: "thedate" transitions: actions: match: "printdatefound" nomatch: "exit" printdatefound: component: "System.Output" text: "Date found is: ${thedate.value.date?long?number_to_date?date?string.medium}" keepturn: true To implement this use case you first need to check if the data variable has been set after the intent engine extracted known entities from the user input. If no date value has been set, you use a setvariable component to define a default value in a variable of type string. This variable is then used in a System.MatchEntity component to verify the provided value is a date and to set it to the DATE variable. Example: Printing Dates Derived from Entity in Short, Medium, Long, Iso and Full Format While ${.now?string.medium} works well for formatting the current date as a string, it seems to fail for dates derived from the DATE entity. To be able to format date as a string in short, medium, long and full format for Dates derived from the DATE entity, you use the following BotML. 17

18 startappointement: variable: "thedate" value: "${iresult.value.entitymatches['date'][0]}" printdatefound: component: "System.Output" text: "Date found in user sentence is: ${thedate.value.date?long?number_to_date?string.short}" keepturn: true Similar to the code above you can use string.medium, string.full, string.long and string.iso. Conclusion Apache FreeMarker Template Language built-in expressions are powerful and available within Oracle Intelligent Bot BotML. You can use these expressions for many use cases, including modification of user input, re-arranging of values through sorting or analyzing user input for patterns and key words. The examples provided in this article give you enough hints to get started with Apache FreeMarker Template Language expressions in your own bot development. 18