keras-pandas Documentation

Transcription

1 keras-pandas Documentation Release Brendan Herger Sep 29, 2018

2

3 Contents: 1 keras-pandas Quick Start Usage Contact Changelog Automater 7 3 lib 11 4 constants 13 5 transformations 15 Python Module Index 17 i

4 ii

5 Welcome to keras-pandas! I d recommend that you start with the keras-pandas section & Quick Start. Contents: 1

6 2 Contents:

7 CHAPTER 1 keras-pandas tl;dr: keras-pandas allows users to rapidly build and iterate on deep learning models. Getting data formatted and into keras can be tedious, time consuming, and difficult, whether your a veteran or new to Keras. keras-pandas overcomes these issues by (automatically) providing: A cleaned, transformed and correctly formatted X and y (good for keras, sklearn or any other ML platform) An input nub, without the hassle of worrying about input shapes or data types An output layer, correctly formatted for the kind of response variable provided With these resources, it s possible to rapidly build and iterate on deep learning models, and focus on the parts of modeling that you enjoy! For more info, check out the: Code Documentation Issue tracker Author s website 1.1 Quick Start Let s build a model with the titanic data set. This data set is particularly fun because this data set contains a mix of categorical and numerical data types, and features a lot of null values. We ll keras-pandas pip install -U keras-pandas And then run the following snippet to create and train a model: 3

8 from keras import Model from keras.layers import Dense from keras_pandas.automater import Automater from keras_pandas.lib import load_titanic observations = load_titanic() # Transform the data set, using keras_pandas categorical_vars = ['pclass', 'sex', 'survived'] numerical_vars = ['age', 'siblings_spouses_aboard', 'parents_children_aboard', 'fare'] text_vars = ['name'] auto = Automater(categorical_vars=categorical_vars, numerical_vars=numerical_vars, text_vars=text_vars, response_var='survived') X, y = auto.fit_transform(observations) # Start model with provided input nub x = auto.input_nub # Fill in your own hidden layers x = Dense(32)(x) x = Dense(32, activation='relu')(x) x = Dense(32)(x) # End model with provided output nub x = auto.output_nub(x) model = Model(inputs=auto.input_layers, outputs=x) model.compile(optimizer='adam', loss=auto.loss, metrics=['accuracy']) # Train model model.fit(x, y, epochs=4, validation_split=.2) 1.2 Usage Installation You can install keras-pandas with pip: pip install -U keras-pandas Creating an Automater The core feature of keras-pandas is the Automater, which accepts lists of variable types (all optional), and a response variable (optional, for supervised problems). Together, all of these variables are the user_input_variables, which may be different than the variables fed into Keras. As a side note, the response variable must be in one of the variable type lists (e.g. survived is in categorical_vars) 4 Chapter 1. keras-pandas

9 One variable type If you only have one variable type, only use that variable type! categorical_vars = ['pclass', 'sex', 'survived'] auto = Automater(categorical_vars=categorical_vars, response_var='survived') Multiple variable types If you have multiple variable types, throw them all in! categorical_vars = ['pclass', 'sex', 'survived'] numerical_vars = ['age', 'siblings_spouses_aboard', 'parents_children_aboard', 'fare'] auto = Automater(categorical_vars=categorical_vars, numerical_vars=numerical_vars, response_var='survived') No response_var If all variables are always available, and / or your problems space doesn t have a single response variable, you can omit the response variable. categorical_vars = ['pclass', 'sex', 'survived'] numerical_vars = ['age', 'siblings_spouses_aboard', 'parents_children_aboard', 'fare'] auto = Automater(categorical_vars=categorical_vars, numerical_vars=numerical_vars) In this case, an output nub will not be auto-generated Fitting the Automater Before use, the Automator must be fit. The fit() method accepts a pandas DataFrame, which must contain all of the columns listed during initialization. auto.fit(observations) Transforming data Now, we can use our Automater to transform the dataset, from a pandas DataFrame to numpy objects properly formatted for Keras s input and output layers. X, y = auto.transform(observations, df_out=false) This will return two objects: X: An array, containing numpy object for each Keras input. This is generally one Keras input for each user input variable. y: A numpy object, containing the response variable (if one was provided) 1.2. Usage 5

10 1.2.5 Using input / output nubs Setting up correctly formatted, heuristically good input and output layers is often Tedious Time consuming Difficult for those new to Keras With this in mind, keras-pandas provides correctly formatted input and output nubs. The input nub is correctly formatted to accept the output from auto.transform(). It contains one Keras Input layer for each generated input, may contain addition layers, and has all input piplines joined with a Concatenate layer. The output layer is correctly formatted to accept the response variable numpy object. 1.3 Contact Hey, I m Brendan Herger, avaiable at Please feel free to reach out to me at 13herger <at> gmail <dot> com 1.4 Changelog Development Adding regression example w/ inverse_transformation (#64) Fixing issue where web socket connections were being opened needlessly (#65) Adding Manifest.in, with including files references in setup.py (#54) Fixed poorly written text embedding index unit test (#52) Added license (#49) Earlier Lots of things happened. Break things and move fast 6 Chapter 1. keras-pandas

11 CHAPTER 2 Automater class Automater.Automater(numerical_vars=[], categorical_vars=[], boolean_vars=[], datetime_vars=[], text_vars=[], non_transformed_vars=[], response_var=none, df_out=false) Bases: object init (numerical_vars=[], categorical_vars=[], boolean_vars=[], datetime_vars=[], text_vars=[], non_transformed_vars=[], response_var=none, df_out=false) x. init (... ) initializes x; see help(type(x)) for signature _check_input_dataframe_columns_(input_dataframe) _check_output_dataframe_columns_(output_dataframe) _create_input_nub(variable_type_dict, input_dataframe) Generate a nub, appropriate for use as an input (and possibly additional Keras layers). Each Keras input variable has on input pipeline, with: One Input (required) Possible additional layers (optional, such as embedding layers for text) All input pipelines are then joined with a Concatenate layer Parameters variable_type_dict ({str:[str]}) A dictionary, with keys describing variables types, and values listing particular variables input_dataframe (pandas.dataframe) A pandas dataframe, containing all keras input layers Returns A Keras layer, which can be fed into future layers Return type ([keras,input], Layer) _create_mappers(variable_type_dict) Creates two sklearn-pandas mappers, one for the input variables, and another for the output variable(s) Parameters variable_type_dict ({str:[str]}) A dictionary, with keys describing variables types, and values listing particular variables 7

12 Returns Two sklearn-pandas mappers, one for the input variables, and another for the output variable(s) Return type (DataFrameMapper, DataFrameMapper) _create_output_nub(variable_type_dict, output_variables_df, y) Generate a nub, appropriate for use as an output / final Keras layer. The structure of this nub will depend on the y variable s data type Parameters variable_type_dict ({str:[str]}) A dictionary, with keys describing variables types, and values listing particular variables output_variables_df (pandas.dataframe) A dataframe containing the output variable. This is necessary for some data types (e.g. a categorical output needs to know how levels the categorical variable has) y (str) The name of the response variable Returns A single Keras layer, correctly formatted to output the response variable provided Return type Layer _suggest_loss(variable_type_dict, y) fit(input_dataframe) Get the data and layers ready for use Train the input transformation pipelines Create the keras input layers Train the output transformation pipeline(s) (optional, only if there is a response variable) Create the output layer(s) (optional, only if there is a response variable) Set self.fitted to True Parameters input_dataframe Returns self, now in a fitted state. The Automater now has initialized input layers, output layer(s) (if response variable is present), and can be used for the transform step Return type Automater fit_transform(input_dataframe) Perform a fit, and then a transform. See transform for return documentation get_transformer(variable) get_transformers() list_default_transformation_pipelines() transform(input_dataframe, df_out=none) Validate that the provided input_dataframe contains the required input columns Transform the keras input columns Transform the response variable, if it is present Format the data for return 8 Chapter 2. Automater

13 Parameters input_dataframe (pandas.dataframe) A pandas dataframe, containing all keras input layers Returns Either a pandas dataframe (if df_out = True), or a numpy object (if df_out = False). This object will contain: the transformed input variables, and the transformed output variables (if the output variable is present in input_dataframe 9

14 10 Chapter 2. Automater

15 CHAPTER 3 lib lib.check_variable_list_are_valid(variable_type_dict) Checks that the provided variable_type_dict is valid, by: Confirming there is no overlap between all variable lists Parameters variable_type_dict ({str:[str]}) A dictionary, with keys describing variables types, and values listing particular variables Returns True, if there is no overlap Return type bool lib.download_file(url, local_file_path, filename) Download the file at url in chunks, to the location at local_file_path Parameters url (str) URL to a file to be downloaded local_file_path (str) Path to download the file to Returns The path to the file on the local machine (same as input local_file_path) Return type str lib.get_variable_type(variable_name, variable_type_dict, response_var) lib.load_iris() lib.load_lending_club() lib.load_mushroom() lib.load_mushrooms() Load the mushrooms data set, as a pandas DataFrame Returns A DataFrame, containing the mushroom dataset Return type pandas.dataframe 11

16 lib.load_titanic() Load the titanic data set, as a pandas DataFrame Returns A DataFrame, containing the titanic dataset Return type pandas.dataframe 12 Chapter 3. lib

17 CHAPTER 4 constants constants.input_nub_categorical_handler(variable, input_dataframe) constants.input_nub_numeric_handler(variable, input_dataframe) constants.input_nub_text_handler(variable, input_dataframe) Create an input nub for text data, by: Finding all derived variables. With a variable name and sequence length of 4, there would be 4 derived variables, name_0 through name_4 - Creating an appropriately shaped input layer, embedding layer, and all future layers - Return both the input layer and last layer (both are necessary for creating a model) Parameters variable (str) Name of the variable input_dataframe (pandas.dataframe) A dataframe, containing either the specified variable, or derived variables Returns A tuple containing the input layer, and the last layer of the nub 13

18 14 Chapter 4. constants

19 CHAPTER 5 transformations class transformations.embeddingvectorizer(embedding_sequence_length=none) Bases: sklearn.base.transformermixin, sklearn.base.baseestimator Converts text into padded sequences. The output of this transformation is consistent with the required format for Keras embedding layers For example the fat man might be transformed into [2, 0, 27, 1, 1, 1], if the embedding_sequence_length is 6. There are a few sentinel values used by this layer: 0 is used for the UNK token (tokens which were not seen during training) 1 is used for the padding token (to fill out sequences that shorter than embedding_sequence_length) init (embedding_sequence_length=none) x. init (... ) initializes x; see help(type(x)) for signature fit(x, y=none) generate_embedding_sequence_length(observation_series) static pad(input_sequence, length, pad_char) Pad the given iterable, so that it is the correct length. Parameters input_sequence Any iterable object length (int) The desired length of the output. pad_char (str or int) The character or int to be added to short sequences Returns A sequence, of len length Return type [] static prepare_input(x) process_string(input_string) Turn a string into padded sequences, consisten with Keras s Embedding layer 15

20 Simple preprocess & tokenize Convert tokens to indices Pad sequence to be the correct length transform(x) Parameters input_string (str) A string, to be converted into a padded sequence of token indices Returns A padded, fixed-length array of token indices Return type [int] 16 Chapter 5. transformations

21 Python Module Index a Automater, 7 c constants, 13 l lib, 11 t transformations, 15 17

22 18 Python Module Index

23 Index Symbols init () (Automater.Automater method), 7 init () (transformations.embeddingvectorizer method), 15 _check_input_dataframe_columns_() (Automater.Automater method), 7 _check_output_dataframe_columns_() (Automater.Automater method), 7 _create_input_nub() (Automater.Automater method), 7 _create_mappers() (Automater.Automater method), 7 _create_output_nub() (Automater.Automater method), 8 _suggest_loss() (Automater.Automater method), 8 A Automater (class in Automater), 7 Automater (module), 7 C check_variable_list_are_valid() (in module lib), 11 constants (module), 13 D download_file() (in module lib), 11 E EmbeddingVectorizer (class in transformations), 15 F fit() (Automater.Automater method), 8 fit() (transformations.embeddingvectorizer method), 15 fit_transform() (Automater.Automater method), 8 G generate_embedding_sequence_length() (transformations.embeddingvectorizer method), 15 get_transformer() (Automater.Automater method), 8 get_transformers() (Automater.Automater method), 8 get_variable_type() (in module lib), 11 I input_nub_categorical_handler() (in module constants), 13 input_nub_numeric_handler() (in module constants), 13 input_nub_text_handler() (in module constants), 13 (Auto- L lib (module), 11 list_default_transformation_pipelines() mater.automater method), 8 load_iris() (in module lib), 11 load_lending_club() (in module lib), 11 load_mushroom() (in module lib), 11 load_mushrooms() (in module lib), 11 load_titanic() (in module lib), 11 P pad() (transformations.embeddingvectorizer static method), 15 prepare_input() (transformations.embeddingvectorizer static method), 15 process_string() (transformations.embeddingvectorizer method), 15 T transform() (Automater.Automater method), 8 transform() (transformations.embeddingvectorizer method), 16 transformations (module), 15 19