Adam Zieliński May 30, 2018
Contents 1 Introduction 1 1.1 Installation................................................ 1 1.2 Constructing the Translator....................................... 1 1.3 The Translation Process......................................... 3 1.4 Using Message Domains......................................... 3 1.5 Usage................................................... 4 2 Using the Translator 5 2.1 Message Placeholders.......................................... 5 2.2 Creating Translations........................................... 6 2.3 Pluralization............................................... 7 2.4 Forcing the Translator Locale...................................... 8 2.5 Retrieving the Message Catalogue.................................... 9 3 Adding Custom Format Support 11 3.1 Creating a Custom Loader........................................ 11 3.2 Creating a Custom Dumper....................................... 12 4 Using with django 13 5 License 15 i
ii
CHAPTER 1 Introduction python_translate provides tools to internationalize your application. 1.1 Installation You can install the component in 2 different ways: Install it via pip (pip install python_translate) Use the official Git repository (https://github.com/adamziel/python_translate). 1.2 Constructing the Translator The main access point of the Translation component is python_translate.translations.translator. Before you can use it, you need to configure it and load the messages to translate (called message catalogs). 1.2.1 Configuration The constructor of the Translator class needs one argument: The locale. from python_translate.translations import Translator translator = Translator('fr_FR') Note: The locale set here is the default locale to use. You can override this locale when translating strings. 1
Note: The term locale refers roughly to the user s language and country. It can be any string that your application uses to manage translations and other format differences (e.g. currency format). The ISO 639-1 language code, an underscore (_), then the ISO 3166-1 alpha-2 country code (e.g. fr_fr for French/France) is recommended. 1.2.2 Loading Message Catalogs The messages are stored in message catalogs inside the Translator class. A message catalog is like a dictionary of translations for a specific locale. python_translate uses Loader classes to load catalogs. You can load multiple resources for the same locale, which will then be combined into one catalog. The component comes with some default Loaders and you can create your own Loader too. The default loaders are: python_translate.loaders.dictloader - to load catalogs from Python dictionaries. python_translate.loaders.jsonfileloader - to load catalogs from JSON files. python_translate.loaders.yamlfileloader - to load catalogs from Yaml files (requires pyyaml). python_translate.loaders.mofileloader - to load catalogs from gettext files (requires polib). python_translate.loaders.pofileloader - to load catalogs from gettext files (requires polib). You can also create your own Loader, in case the format is not already supported by one of the default loaders. At first, you should add one or more loaders to the Translator: #... translator.add_loader('dict', DictLoader()) The first argument is the name to which you can refer the loader in the translator and the second argument is an instance of the loader itself. After this, you can add your resources using the correct loader. Loading Messages with the ArrayLoader Loading messages can be done by calling python_translate.translation.translator::add_resource. The first argument is the loader name (this was the first argument of the add_loader method), the second is the resource and the third argument is the locale: #... translator.add_resource('dict', { 'Hello World!': 'Bonjour', }, 'fr_fr') Loading Messages with the File Loaders If you use one of the file loaders, you should also use the add_resource method. The only difference is that you should put the file name to the resource file as the second argument, instead of an array: #... translator.add_loader('yaml', YamlFileLoader()) translator.add_resource('yaml', 'path/to/messages.fr.yml', 'fr_fr') 2 Chapter 1. Introduction
1.3 The Translation Process To actually translate the message, the Translator uses a simple process: A catalog of translated messages is loaded from translation resources defined for the locale (e.g. fr_fr). Messages from the Fallback Locales are also loaded and added to the catalog, if they don t already exist. The end result is a large dictionary of translations If the message is located in the catalog, the translation is returned. If not, the translator returns the original message. You start this process by calling python_translate.translation.translator::trans or python_translate.translation.translator::transchoice. Then, the Translator looks for the exact string inside the appropriate message catalog and returns it (if it exists). 1.3.1 Fallback Locales If the message is not located in the catalog of the specific locale, the translator will look into the catalog of one or more fallback locales. For example, assume you re trying to translate into the fr_fr locale: 1. First, the translator looks for the translation in the fr_fr locale; 2. If it wasn t found, the translator looks for the translation in the fr locale; 3. If the translation still isn t found, the translator uses the one or more fallback locales set explicitly on the translator. For (3), the fallback locales can be set by calling python_translate.translation.translator::set_fallback_locales: #... translator.set_fallback_locales(['en']) 1.4 Using Message Domains As you ve seen, message files are organized into the different locales that they translate. The message files can also be organized further into domains. The domain is specified in the fourth argument of the add_resource() method. The default domain is messages. For example, suppose that, for organization, translations were split into three different domains: messages, admin and navigation. The French translation would be loaded like this: //... translator.add_loader('yml', YamlLoader()) translator.add_resource('yml', 'messages.fr.yml', 'fr_fr') translator.add_resource('yml', 'admin.fr.yml', 'fr_fr', 'admin') translator.add_resource('yml', 'navigation.fr.yml', 'fr_fr', 'navigation') When translating strings that are not in the default domain (messages), you must specify the domain as the third argument of trans(): translator.trans('symfony is great', {}, 'admin') Symfony will now look for the message in the admin domain of the specified locale. 1.3. The Translation Process 3
1.5 Usage Read how to use the Translation component in Using the Translator. 4 Chapter 1. Introduction
CHAPTER 2 Using the Translator Imagine you want to translate the string Symfony into French: from python_translate.translations import Translator from python_translate.loader import DictLoader translator = new Translator('fr_FR') translator.add_loader('dict', DictLoader()) translator.add_resource('dict', { 'Symfony!': "J'aime python_translate!", }, 'fr_fr') print translator->trans('symfony!') In this example, the message Symfony! will be translated into the locale set in the constructor (fr_fr) if the message exists in one of the message catalogs. 2.1 Message Placeholders Sometimes, a message containing a variable needs to be translated: //... translated = translator.trans('hello ' + name) print translated However, creating a translation for this string is impossible since the translator will try to look up the exact message, including the variable portions (e.g. Hello Ryan or Hello Fabien ). Instead of writing a translation for every possible iteration of the name variable, you can replace the variable with a placeholder : translated = translator.trans( 'Hello {name}', {'name': name} (continues on next page) 5
) (continued from previous page) print translated python_translate will now look for a translation of the raw message (Hello {name}) and then replace the placeholders with their values. Creating a translation is done just as before: Note: The placeholders are simply python formating placeholders. As you ve seen, creating a translation is a two-step process: 1. Abstract the message that needs to be translated by processing it through the Translator. 2. Create a translation for the message in each locale that you choose to support. The second step is done by creating message catalogs that define the translations for any number of different locales. 2.2 Creating Translations Translation files consist of a series of id-translation pairs for the given domain and locale. The source is the identifier for the individual translation, and can be the message in the main locale (e.g. Symfony is great ) of your application or a unique identifier (e.g. symfony.great - see the sidebar below). Translation files can be created in several different formats, Yaml being the recommended format. These files are parsed by one of the loader classes. Using Real or Keyword Messages This example illustrates the two different philosophies when creating messages to be translated: translator.trans('symfony') translator.trans('symfony.great') In the first method, messages are written in the language of the default locale (English in this case). That message is then used as the id when creating translations. In the second method, messages are actually keywords that convey the idea of the message. The keyword message is then used as the id for any translations. In this case, translations must be made for the default locale (i.e. to translate symfony.great to Symfony is great). The second method is handy because the message key won t need to be changed in every translation file if you decide that the message should actually read Symfony is really great in the default locale. The choice of which method to use is entirely up to you, but the keyword format is often recommended. Additionally, the yaml file format supports nested ids to avoid repeating yourself if you use keywords instead of real text for your ids: The multiple levels are flattened into single id/translation pairs by adding a dot (.) between every level, therefore the above examples are equivalent to the following: 6 Chapter 2. Using the Translator
2.3 Pluralization Message pluralization is a tough topic as the rules can be quite complex. For instance, here is the mathematical representation of the Russian pluralization rules: ((number % 10 == 1) && (number % 100!= 11))? 0 : (((number % 10 >= 2) && (number % 10 <= 4) && ((number % 100 < 10) (number % 100 >= 20)))? 1 : 2 ) As you can see, in Russian, you can have three different plural forms, each given an index of 0, 1 or 2. For each form, the plural is different, and so the translation is also different. When a translation has different forms due to pluralization, you can provide all the forms as a string separated by a pipe ( ): 'There is one apple There are {count} apples' To translate pluralized messages, use the python_translate.translations.translator.transchoice method translator.transchoice( There is one apple There are {count} apples, 10, { count : 10} ) The second argument (10 in this example) is the number of objects being described and is used to determine which translation to use and also to populate the {count} placeholder. Based on the given number, the translator chooses the right plural form. In English, most words have a singular form when there is exactly one object and a plural form for all other numbers (0, 2, 3... ). So, if count is 1, the translator will use the first string (There is one apple) as the translation. Otherwise it will use There are {count} apples. Here is the French translation: 'Il y a {count} pomme Il y a {count} pommes' Even if the string looks similar (it is made of two sub-strings separated by a pipe), the French rules are different: the first form (no plural) is used when count is 0 or 1. So, the translator will automatically use the first string (Il y a {count} pomme) when count is 0 or 1. Each locale has its own set of rules, with some having as many as six different plural forms with complex rules behind which numbers map to which plural form. The rules are quite simple for English and French, but for Russian, you d may want a hint to know which rule matches which string. To help translators, you can optionally tag each string: 'one: There is one apple some: There are {count} apples' 'none_or_one: Il y a {count} pomme some: Il y a {count} pommes' The tags are really only hints for translators and don t affect the logic used to determine which plural form to use. The tags can be any descriptive string that ends with a colon (:). The tags also do not need to be the same in the original message as in the translated one. 2.3. Pluralization 7
Tip: As tags are optional, the translator doesn t use them (the translator will only get a string based on its position in the string). 2.3.1 Explicit Interval Pluralization The easiest way to pluralize a message is to let the Translator use internal logic to choose which string to use based on a given number. Sometimes, you ll need more control or want a different translation for specific cases (for 0, or when the count is negative, for example). For such cases, you can use explicit math intervals: '{0} There are no apples {1} There is one apple ]1,19] There are {count} apples [20, Inf] There are many apples' The intervals follow the ISO 31-11 notation. The above string specifies four different intervals: exactly 0, exactly 1, 2-19, and 20 and higher. Note that interval definitions will not be present in a translated message, and they have nothing to do with placeholders such as {count}. They will not be formatted, just removed before you get to see them. You can also mix explicit math rules and standard rules. In this case, if the count is not matched by a specific interval, the standard rules take effect after removing the explicit rules: '{0} There are no apples [20,Inf] There are many apples There is one apple a_few: There are {count} apples' For example, for 1 apple, the standard rule There is one apple will be used. For 2-19 apples, the second standard rule There are {count} apples will be selected. You may even represent a finite set of numbers: {1,2,3,4} Or numbers between two other numbers: [1, +Inf[ ]-1,2[ The left delimiter can be [ (inclusive) or ] (exclusive). The right delimiter can be [ (exclusive) or ] (inclusive). Beside numbers, you can use -Inf and +Inf for the infinite. 2.4 Forcing the Translator Locale When translating a message, the Translator uses the specified locale or the fallback locale if necessary. You can also manually specify the locale to use for translation: translator.trans( 'Symfony', {}, 'messages', 'fr_fr' ) translator.transchoice( (continues on next page) 8 Chapter 2. Using the Translator
) (continued from previous page) '{0} There are no apples {1} There is one apple ]1,Inf[ There are {count} apples', 10, {'count': 10}, 'messages', 'fr_fr' 2.5 Retrieving the Message Catalogue In case you want to use the same translation catalogue outside your application (e.g. use translation on the client side), it s possible to fetch raw translation messages. Just specify the required locale: messages = translator.get_messages('fr_fr') The messages variable will have the following structure: { } 'messages': { 'Hello world': 'Bonjour tout le monde', }, 'validators': { 'Value should not be empty': 'Valeur ne doit pas être vide', 'Value is too long': 'Valeur est trop long', } 2.5. Retrieving the Message Catalogue 9
10 Chapter 2. Using the Translator
CHAPTER 3 Adding Custom Format Support Sometimes, you need to deal with custom formats for translation files. The Translation component is flexible enough to support this. Just create a loader (to load translations) and, optionally, a dumper (to dump translations). Imagine that you have a custom format where translation messages are defined using one line for each translation and parentheses to wrap the key and the message. A translation file would look like this: (welcome)(accueil) (goodbye)(au revoir) (hello)(bonjour) 3.1 Creating a Custom Loader To define a custom loader that is able to read these kinds of files, you must create a new class that extends the python_translate.loaders.loader. The :method: python_translate.loaders.loader::load method will get a filename and parse it into an dict. Then, it will create the catalog that will be returned: import re from python_translate.translations import MessageCatalogue from python_translate.loaders import Loader FORMAT_REGEX = re.compile("\(([^\)]+)\)\(([^\)]+)\)") class MyFormatLoader(Loader): def load(self, resource, locale, domain = 'messages'): messages = {} with open(resource, 'r') as f: lines = f.readlines() for line in lines: match = FORMAT_REGEX.match(line) if match: (continues on next page) 11
messages[match.group(1)] = match.group(2) (continued from previous page) catalogue = MessageCatalogue(locale) catalogue.add(messages, domain) return catalogue Once created, it can be used as any other loader: from python_translate.translations import Translator translator = Translator('fr_FR') translator.add_loader('my_format', MyFormatLoader()) translator.add_resource('my_format', './translations/messages.txt', 'fr_fr') print translator.trans('welcome') It will print accueil. 3.2 Creating a Custom Dumper It is also possible to create a custom dumper for your format, which is useful when using the extraction commands. To do so, a new class implementing the python_translate.dumpers.dumper. The must be created. To write the dump contents into a file, extending the python_translate.dumpers.filedumper class will save a few lines: :method:`python_translate.dumpers.dumper::load` from python_translate.translations import MessageCatalogue from python_translate.dumpers import File- Dumper class MyFormatDumper(FileDumper): def format(self, messages, domain = messages ): output = for source, target in messages.all(domain).items(): output += ({0})({1})n.format(source, target) return output def get_extension(self): return txt The :method: python_translate.dumpers.filedumper::format method creates the output string, that will be used by the :method: python_translate.dumpers.dumper::dump method of the FileDumper class to create the file. The dumper can be used like any other built-in dumper. In the following example, the translation messages defined in the YAML file are dumped into a text file with the custom format: from python_translate.loaders import YamlFileLoader loader = YamlFileLoader(); catalogue = loader.load('./translations/messages.fr_fr.yml', 'fr_fr') dumper = MyFormatDumper() dumper.dump(catalogue, {'path': './dumps'}) 12 Chapter 3. Adding Custom Format Support
CHAPTER 4 Using with django There is a separate project that integrates python_translate with django: http://github.com/adamziel/django_translate 13
14 Chapter 4. Using with django
CHAPTER 5 License The code is derived from the Symfony Translation component: (c) Fabien Potencier <fabien@symfony.com> https://github.com/symfony/symfony For the full copyright and license information, please view the LICENSE and LICENSE_SYMFONY_TRANSLATION The documentation and README are derived from the Symfony Documentation: https://github.com/symfony/ symfony-docs http://symfony.com The documentation and README are licensed under https://creativecommons. org/licenses/by-sa/3.0/ 15
16 Chapter 5. License
Index C Components Translation, 1 T Translation, 1 Adding Custom Format Support, 9 Usage, 4 Using with django, 12 17