pylatexenc Documentation

Transcription

1 pylatexenc Documentation Release 1.2 Philippe Faist Apr 28, 2017

2

3 Contents: 1 Simple Parser for LaTeX Code The main LatexWalker class Exception Classes Macro Definitions Data Node Classes and Helpers Simple Latex to Text Converter Latex to Text Converter Class Known Macros and Environments Encode Unicode to LaTeX 13 4 Indices and tables 15 Python Module Index 17 i

4 ii

5 This package provides simple and heuristic conversion of LaTeX to unicode and vice versa. Quick example: >>> from pylatexenc.latex2text import LatexNodes2Text >>> latex = r"""\textbf{hi there!} Here is \emph{an equation}:... \begin{equation}... \zeta = x + i y... \end{equation}... where $i$ is the imaginary unit.... """ >>> print(latexnodes2text().latex_to_text(latex)) Hi there! Here is an equation: ζ = x + i y where i is the imaginary unit. And the other way around: >>> from pylatexenc.latexencode import utf8tolatex >>> text = u"à votre santé!" >>> print(utf8tolatex(text)) {\`A} votre sant{\'e}! Contents: 1

6 2 Contents:

7 CHAPTER 1 Simple Parser for LaTeX Code The latexwalker module provides a simple API for parsing LaTeX snippets, and representing the contents using a data structure based on nodes classes. LatexWalker will understand the syntax of most common macros. However, latexwalker is NOT a replacement for a full LaTeX engine. (Originally, latexwalker was desigend to extract useful text for indexing for text database searches of LaTeX content.) The main LatexWalker class class pylatexenc.latexwalker.latexwalker(s, macro_dict=none, **flags) A parser which walks through an input stream, parsing it as LaTeX markup. Arguments: s: the string to parse as LaTeX code macro_dict: a context dictionary of known LaTeX macros. By default, the default global macro dictionary default_macro_dict is used. This should be a dictionary where the keys are macro names (see MacrosDef.macname) and values are MacrosDef instances. Additional keyword arguments are flags which influence the parsing. Accepted flags are: keep_inline_math=true False If this option is set to True, then inline math is parsed and stored using LatexMathNode instances. Otherwise, inline math is not treated differently and is simply kept as text. tolerant_parsing=true False If set to True, then the parser generally ignores syntax errors rather than raising an exception. strict_braces=true False This option refers specifically to reading a encountering a closing brace when an expression is needed. You generally won t need to specify this flag, use tolerant_parsing instead. get_latex_braced_group(pos, brace_type= { ) Reads a latex expression enclosed in braces {... }. The first token of s[pos:] must be an opening brace. 3

8 Returns a tuple (node, pos, len), where node is a LatexGroupNode instance, pos is the first char of the expression (which has to be an opening brace), and len is its length, including the closing brace. get_latex_environment(pos, environmentname=none) Reads a latex expression enclosed in a \begin{environment}...\end{environment}. The first token in the stream must be the \begin{environment}. If environmentname is given and nonempty, then additionally a LatexWalkerParseError is raised if the environment in the input stream does not match the provided name. This function will attempt to heuristically parse an optional argument, and possibly a mandatory argument given to the environment. No space is allowed between egin{environment} and an opening square bracket or opening brace. Returns a tuple (node, pos, len) with node being a LatexEnvironmentNode. get_latex_expression(pos, strict_braces=none) Reads a latex expression, e.g. macro argument. This may be a single char, an escape sequence, or a expression placed in braces. This is what TeX calls a token (and not what we call a token... anyway). Returns a tuple (<LatexNode instance>, pos, len). pos is the first char of the expression, and len is its length. get_latex_maybe_optional_arg(pos) Attempts to parse an optional argument. Returns a tuple (groupnode, pos, len) if success, otherwise returns None. get_latex_nodes(pos=0, stop_upon_closing_brace=none, stop_upon_end_environment=none, stop_upon_closing_mathmode=none) Parses latex content stored in self.s into a list of nodes. Returns a tuple (nodelist, pos, len) where nodelist is a list of LatexNode s. If stop_upon_closing_brace is given and set to a character, then parsing stops once the given closing brace is encountered (but not inside a subgroup). The brace is given as a character, ] or }. The returned len includes the closing brace, but the closing brace is not included in any of the nodes in the nodelist. If stop_upon_end_environment is provided, then parsing stops once the given environment was closed. If there is an environment mismatch, then a LatexWalkerParseError is raised except in tolerant parsing mode (see py:meth:parse_flags()). Again, the closing environment is included in the length count but not the nodes. get_token(pos, brackets_are_chars=true, environments=true, keep_inline_math=none) Parse the token in the stream pointed to at position pos. Returns a LatexToken. Raises LatexWalkerEndOfStream if end of stream reached. If brackets_are_chars=false, then square bracket characters count as brace_open and brace_close token types (see LatexToken); otherwise (the default) they are considered just like other normal characters. If environments=false, then begin and end tokens count as regular macro tokens (see LatexToken); otherwise (the default) they are considered as the token types begin_environment and end_environment. If keep_inline_math is not None, then that value overrides that of self.keep_inline_math for the duration of this method call. parse_flags() The parse flags currently set on this object. Returns a dictionary with keys keep_inline_math, tolerant_parsing and strict_braces. 4 Chapter 1. Simple Parser for LaTeX Code

9 Exception Classes class pylatexenc.latexwalker.latexwalkererror Generic exception class raised by this module. class pylatexenc.latexwalker.latexwalkerparseerror(msg, s=none, pos=none) Parse error. The following attributes are available: msg (the error message), s (the parsed string), pos (the position of the error in the string, 0-based index). class pylatexenc.latexwalker.latexwalkerendofstream Reached end of input stream (e.g., end of file). Macro Definitions class pylatexenc.latexwalker.macrosdef(macname, optarg, numargs) Class which stores a Macro syntax. macname stores the name of the macro, without the leading backslash. optarg may be one of True, False, or None. if True, the macro expects as first argument an optional argument in square brackets. Then, numargs specifies the number of additional mandatory arguments to the command, given in usual curly braces (or simply as one TeX token) if False, the macro only expects a number of mandatory arguments given by numargs. The mandatory arguments are given in usual curly braces (or simply as one TeX token) if None, then numargs is a string of either characters { or [, in which each curly brace specifies a mandatory argument and each square bracket specifies an optional argument in square brackets. For example, {{[{ expects two mandatory arguments, then an optional argument in square brackets, and then another mandatory argument. pylatexenc.latexwalker.default_macro_dict The default context dictionary of known LaTeX macros. macname) and the values are MacrosDef instances. The keys are the macro names (MacrosDef. Data Node Classes and Helpers class pylatexenc.latexwalker.latextoken(tok, arg, pos, len, pre_space) Represents a token read from the LaTeX input. This is not the same thing as a LaTeX token, it s just a part of the input which we treat in the same way (e.g. a bunch of content characters, a comment, a macro, etc.) Information about the object is stored into the fields tok and arg. The tok field is a string which identifies the type of the token. The arg depends on what tok is, and describes the actual input. Additionally, this class stores information about the position of the token in the input stream in the field pos. This pos is an integer which corresponds to the index in the input string. The field len stores the length of the token in the input string. This means that this token spans in the input string from pos to pos+len. Leading whitespace before the token is not returned as a separate char -type token, but it is given in the pre_space field of the token which follows. Pre-space may contain a newline, but not two consecutive newlines Exception Classes 5

10 The tok field may be one of: char : raw characters which have no special LaTeX meaning; they are part of the text content. The arg field contains the characters themselves. macro : a macro invokation, but not begin or end The arg field contains the name of the macro, without the leading backslash. begin_environment : an invokation of begin{environment}. The arg field contains the name of the environment inside the braces. end_environment : an invokation of end{environment}. The arg field contains the name of the environment inside the braces. comment : a LaTeX comment delimited by a percent sign up to the end of the line. The arg field contains the text in the comment line, not including the percent sign nor the newline. brace_open : an opening brace. This is usually a curly brace, and sometimes also a square bracket. What is parsed as a brace depends on the arguments to get_token(). The arg is a string which contains the relevant brace character. brace_close : a closing brace. This is usually a curly brace, and sometimes also a square bracket. What is parsed as a brace depends on the arguments to get_token(). The arg is a string which contains the relevant brace character. mathmode_inline : a delimiter which starts inline math. This is (e.g.) a single $ character which is not part of a double $$ display environment delimiter. The arg is the string value of the delimiter in question ( $ ) class pylatexenc.latexwalker.latexnode(**kwargs) Represents an abstract node of the latex document. Use nodetype() to figure out what type of node this is. class pylatexenc.latexwalker.latexcharsnode(chars, **kwargs) A string of characters in the LaTeX document, without any special meaning. class pylatexenc.latexwalker.latexgroupnode(nodelist, **kwargs) A LaTeX group, i.e. {...}. class pylatexenc.latexwalker.latexcommentnode(comment, **kwargs) A LaTeX comment, delimited by a percent sign until the end of line. class pylatexenc.latexwalker.latexmacronode(macroname, nodeoptarg=none, nodeargs=[], **kwargs) Represents a macro type node, e.g. textbf class pylatexenc.latexwalker.latexenvironmentnode(envname, nodelist, optargs=[], args=[], **kwargs) A LaTeX Environment Node, i.e. begin{something}... end{something}. Arguments: envname: the name of the environment ( itemize, equation,...) nodelist: a list of LatexNode s that represent all the contents between the begin{...} instruction and the end{...} instruction. 6 Chapter 1. Simple Parser for LaTeX Code

11 optargs: any possible optional argument passed to the begin{...} instruction, for example in begin{enumerate}[label=roman*)] (Currently, only a single optional argument is parsed, but this argument still accepts anyway a list of :py:class: LatexNode s.) args: any possible regular arguments passed to the begin{...} instruction, for example in begin{tabular}{clr}. Currently, only a single regular argument is parsed at maximum, but this is anyway a list of LatexNode s. class pylatexenc.latexwalker.latexmathnode(displaytype, nodelist=[], **kwargs) A Math node type. Note that currently only inline math environments are detected. Arguments: displaytype: either inline or display 1.4. Data Node Classes and Helpers 7

12 8 Chapter 1. Simple Parser for LaTeX Code

13 CHAPTER 2 Simple Latex to Text Converter A simplistic, heuristic LaTeX code parser allowing to returns a text-only approximation. Suitable, e.g. for indexing tex code in a database for full text searching. The main class is LatexNodes2Text. For a quick start, try: from pylatexenc.latex2text import LatexNodes2Text latex = "... LaTeX code..." text = LatexNodes2Text().latex_to_text(latex) Latex to Text Converter Class class pylatexenc.latex2text.latexnodes2text(env_dict=none, macro_dict=none, text_replacements=none, **flags) Simplistic Latex-To-Text Converter. This class parses a nodes structure generated by the latexwalker module, and creates a text representation of the structure. It is capable of parsing \input directives safely, see set_tex_input_directory() and read_input_file(). By default, \input and \include directives are ignored. Arguments to the constructor: env_dict, macro_dict are dictionaries of known environment and macro definitions. default_env_dict and default_macro_dict, respectively. They default to text_replacements are string replacements to apply onto the final string, as final touches. This defaults to default_text_replacements. Additional keyword arguments are flags which may influence the behavior: keep_inline_math=true False: If set to True, then inline math is kept using dollar signs, otherwise it is incorporated as normal text. 9

14 keep_comments=true False: If set to True, then LaTeX comments are kept (including the percent-sign); otherwise they are discarded. latex_to_text(latex, **parse_flags) Parses the given latex code and returns its textual representation. The parse_flags are the flags to give on to the pylatexenc.latexwalker.latexwalker constructor. node_to_text(node) Return the textual representation of the given node. nodelist_to_text(nodelist) Extracts text from a node list. nodelist is a list of nodes as returned by pylatexenc.latexwalker. LatexWalker.get_latex_nodes(). read_input_file(fn) This method may be overridden to implement a custom lookup mechanism when encountering \input or \include directives. The default implementation looks for a file of the given name relative to the directory set by set_tex_input_directory(). If strict_input=true was set, we ensure strictly that the file resides in a subtree of the reference input directory (after canonicalizing the paths and resolving all symlinks). You may override this method to obtain the input data in however way you see fit. (In that case, a call to set_tex_input_directory() may not be needed as that function simply sets properties which are used by the default implementation of read_input_file().) This function accepts the referred filename as argument (the argument to the \input macro), and should return a string with the file contents (or generate a warning or raise an error). set_tex_input_directory(tex_input_directory, latex_walker_init_args=none, strict_input=true) Set where to look for input files when encountering the \input or \include macro. Alternatively, you may also override read_input_file() to implement a custom file lookup mechanism. The argument tex_input_directory is the directory relative to which to search for input files. If strict_input is set to True, then we always check that the referenced file lies within the subtree of tex_input_directory, prohibiting for instance hacks with.. in filenames or using symbolic links to refer to files out of the directory tree. The argument latex_walker_init_args allows you to specify the parse flags passed to the constructor of pylatexenc.latexwalker.latexwalker when parsing the input file. Known Macros and Environments class pylatexenc.latex2text.envdef(envname, simplify_repl=none, discard=false) An environment definition. envname: the name of the environment simplify_repl: the replacement text of the environment. This is either a callable or a string. If it is a callable, it must accept a single argument, the pylatexenc.latexwalker. LatexEnvironmentNode representing the LaTeX environment. If it is a string, it may contain %s which will be replaced by the environment contents converted to text. discard: if set to True, then the full environment is discarded, i.e., it is converted to an empty string. 10 Chapter 2. Simple Latex to Text Converter

15 class pylatexenc.latex2text.macrodef(macname, simplify_repl=none, discard=none) A macro definition. macname: the name of the macro (no backslash) simplify_repl: either a string or a callable. The string may contain %s replacements, in which the macro arguments will be substituted. The callable should accept the corresponding pylatexenc. latexwalker.latexmacronode as an argument. discard: if set to True, then the macro call is discarded, i.e., it is converted to an empty string. pylatexenc.latex2text.default_env_dict The default context dictionary of known LaTeX environment definitions and how to convert them to text. This is a dictionary with keys the environment name (EnvDef.envname) and values are EnvDef instances. pylatexenc.latex2text.default_macro_dict The default context dictionary of known LaTeX macro definitions and how to convert them to text. This is a dictionary with keys the macro name (MacroDef.macname) and values are MacroDef instances. pylatexenc.latex2text.default_text_replacements Default text replacements (final touches) to apply to LaTeX code. (For instance, converting ~ to (space) or '' to ".) This is a list (or tuple) of pairs of (regex-pattern, replacement-text) of replacements to perform Known Macros and Environments 11

16 12 Chapter 2. Simple Latex to Text Converter

17 CHAPTER 3 Encode Unicode to LaTeX The latexencode module provides a single function, utf8tolatex() which allows you to convert a unicode string to LaTeX escape sequences. pylatexenc.latexencode.utf8tolatex(s, non_ascii_only=false, brackets=true, substitute_bad_chars=false) Encode a UTF-8 string to a LaTeX snippet. If non_ascii_only is set to True, then usual (ascii) characters such as #, {, } etc. will not be escaped. If set to False (the default), they are escaped to their respective LaTeX escape sequences. If brackets is set to True (the default), then LaTeX macros are enclosed in brackets. For example, santé is replaced by sant{\'e} if brackets=true and by sant\'e if brackets=false. Warning: Using brackets=false might give you an invalid LaTeX string, so avoid it! (for instance, maître will be replaced incorrectly by ma\^\itre resulting in an unknown macro \itre). If substitute_bad_chars=true, then any non-ascii character for which no LaTeX escape sequence is known is replaced by a question mark in boldface. Otherwise (by default), the character is left as it is. 13

18 14 Chapter 3. Encode Unicode to LaTeX

19 CHAPTER 4 Indices and tables genindex modindex search 15

20 16 Chapter 4. Indices and tables

21 Python Module Index p pylatexenc.latex2text, 9 pylatexenc.latexencode, 13 pylatexenc.latexwalker, 3 17

22 18 Python Module Index

23 Index D default_env_dict (in module pylatexenc.latex2text), 11 default_macro_dict (in module pylatexenc.latex2text), 11 default_macro_dict (in module pylatexenc.latexwalker), 5 default_text_replacements (in module pylatexenc.latex2text), 11 E EnvDef (class in pylatexenc.latex2text), 10 G get_latex_braced_group() (pylatexenc.latexwalker.latexwalker method), 3 get_latex_environment() (pylatexenc.latexwalker.latexwalker method), 4 get_latex_expression() (pylatexenc.latexwalker.latexwalker method), 4 get_latex_maybe_optional_arg() (pylatexenc.latexwalker.latexwalker method), 4 get_latex_nodes() (pylatexenc.latexwalker.latexwalker method), 4 get_token() (pylatexenc.latexwalker.latexwalker method), 4 L latex_to_text() (pylatexenc.latex2text.latexnodes2text method), 10 LatexCharsNode (class in pylatexenc.latexwalker), 6 LatexCommentNode (class in pylatexenc.latexwalker), 6 LatexEnvironmentNode (class in pylatexenc.latexwalker), 6 LatexGroupNode (class in pylatexenc.latexwalker), 6 LatexMacroNode (class in pylatexenc.latexwalker), 6 LatexMathNode (class in pylatexenc.latexwalker), 7 LatexNode (class in pylatexenc.latexwalker), 6 LatexNodes2Text (class in pylatexenc.latex2text), 9 LatexToken (class in pylatexenc.latexwalker), 5 LatexWalker (class in pylatexenc.latexwalker), 3 LatexWalkerEndOfStream (class in pylatexenc.latexwalker), 5 LatexWalkerError (class in pylatexenc.latexwalker), 5 LatexWalkerParseError (class in pylatexenc.latexwalker), 5 M MacroDef (class in pylatexenc.latex2text), 10 MacrosDef (class in pylatexenc.latexwalker), 5 N P parse_flags() (pylatexenc.latexwalker.latexwalker method), 4 pylatexenc.latex2text (module), 9 pylatexenc.latexencode (module), 13 pylatexenc.latexwalker (module), 3 R S (pylatex- method), set_tex_input_directory() enc.latex2text.latexnodes2text 10 U utf8tolatex() (in module pylatexenc.latexencode), 13 node_to_text() (pylatexenc.latex2text.latexnodes2text method), 10 nodelist_to_text() (pylatexenc.latex2text.latexnodes2text method), 10 (pylatex- method), read_input_file() enc.latex2text.latexnodes2text 10 19