Chameleon Metadata s Data Science Basics Tutorial Series. DSB-2: Information Gain (IG) By Eric Thornton,

Transcription

1 Chameleon Metadata s Data Science Basics Tutorial Series Data Science Basics Syllabus for DSB-2 (DSB-2-Infographic-1) Download PDF version here: DSB-2-Information-Gain-V10.pdf DSB-2: Information Gain (IG) By Eric Thornton, ET@ChameleonMetadata.com (20 September, 2017 V11) DSB-1 DSB-2 DSB-3 DSB-4 DSB-5 DSB-6 DSB-7 Links to Chameleon Metadata s Data Science Basics Series Papers INTRODUCTION Links will be activated on an ongoing basis going forward as each paper is published In the first of this series of six tutorials, DSB-1: Information Entropy (Ѥ), we walked through how one calculates Ѥ for datasets, which had a binary target variable with each record classified as a Type 1 or Type 2 record. When we calculate Ѥ for a dataset, the resulting calculation only tells us how much disorder is present with a given dataset. Remember that the dataset s Ѥ value we calculated in DSB-1 applies to the entire dataset as a whole and tells us nothing about the individual attributes (or features) of the dataset. In DSB-1 we were looking for a dataset with higher Ѥ values (higher value = more disordered) as a starting point. Once we have a dataset with a high Ѥ value, the next step is to calculate Ѥ for each individual attribute/feature relative to the same target variable we used in DSB-1. In other words, calculating the number of Type 1 or Type 2 records grouped by each possible value for a single attribute. For example, with possible values for GENDER being { Female or Male }, how many Type 1 and Type 2 records are there for each possible GENDER value (i.e. how many Type 1 or Type 2 records when GENDER= Female vs. how many when GENDER= Male ). In this second of Chameleon Metadata s Data Science Basics tutorial series, DSB-2: Information Gain (IG), we will walk through how to figure out which attribute is the best one to use as your initial split criteria based on its (IG) value relative to our target variable. This process of attribute-driven division of datasets is also the foundation for more complicated calculations we will explore in later tutorials. By DSB-5 and DSB-6 we will learn how to do probability estimates by classification. But for now, let s get started on learning how to calculate Information Gain (IG) for the table uci_dataset_flags_plus_target which. Topics covered in DSB-2: Information Gain (IG) will be: 1. Relationship Between Ѥ and Information Gain (IG) 2. Calculating Attribute Entropy (Æ) for Each Attribute of the UCI Flags Plus Target 3. Calculating Information Gain (IG) for Each Attribute of the UCI Flags Plus Target

2 BULK LEARNING LAB SETUP File Learning Lab Setup & Labs File (DSB-2-SQL-BULK-1.sql) All SQL & DDL used in this tutorial may be downloaded at Chameleon s Learning Lab as file: DSB-2-SQL-BULK-1. URL for DSB-2 Student Lab Setup Bulk SQL File MySQL Syntax (DSB-2-SQL-BULK-1) 1. RELATIONSHIP BETWEEN INFORMATION ENTROPY (Ѥ) INFORMATION GAIN (IG) Review of Information Entropy (Ѥ) In DSB-1, we walked through how to calculate Ѥ for a dataset with a binary target variable. As a reminder, a binary target variable is one with only two possible outcomes like having only Heads or Tails as possible outcomes for a given coin toss, etc. The problem with using the outcomes of a coin toss is that there are very few variables affecting the outcome of each toss. Typically, and assuming the same coin is used, there isn t anything variable about each toss. Sure, you could use multiple different 2 coins (i.e. coins with a face value of 2 Euros) and document differences between different 2 coins (i.e. this 2 coin weighs 0.04 grams less than that 2 coin) and provide another attribute/feature (coin-weight) about each coin. Then it s possible to group the number of Heads or Tails by coin weight. But, that s not how typical coin toss results work out, so we used the UCI Flags Plus Target dataset instead for tutorial DSB-1, because it gave us a good bit of other attributes with which to work. In that dataset, we find attributes/features other than the target variable, the Dependent Variable, which we can use as the Independent Variables to help us understand how much influence it has over the dependent variable. The more disordered the dataset, the higher its Information Entropy value As the proportion of target values skews towards one value, the Information Entropy value decreases Dataset Entropy vs. Attribute Entropy (DSB-2-Infographic-2) When only a single target value exists, the set is Pure and Information Entropy = 0 As we saw in Graph-1 from the first tutorial, and recopied below, we are looking for a dataset with a higher Ѥ value if it is to be of any value in predicting the answers to business questions. Why? Well, think about it, if a trick-coin with Heads on both sides always produces Heads whenever it is tossed, nobody needs data science to tell people it s highly probable that the next toss will be Heads too 100%. And, with such a trick-coin, we also know the probability of getting Tails on a future toss is very close to approaching zero as well 0%. Because of this, notice in Graph-1 below that Ѥ approaches zero as the probability of its binary target values moves towards 100% or 0%.

3 Introduction to Attribute Entropy (Æ) Entropy for Dataset with Binary Target Variable (DSB-1-Graph-1) Attribute Entropy (Æ) is the next piece of this puzzle we need to understand how strong the relationship is and, if one even exists between each of the non-target attributes and their associated target value for each observation. We have been looking for a highly disordered dataset with a high Ѥ value for the entire dataset up to this point. Here is a quick review for calculating the Ѥ value for the UCI Flags Plus Target Dataset from DSB-1: 57.73% Type % Type 2 x x x x x x x x x x = 10 Observations UCI Flags Plus Target Dataset - Chameleon Target Variable Counts (Type 1 vs. Type 2) (DSB-2-Infographic-3) Ѥ Flags Dataset = ((-1) * ( Type-1 * Log 2( Type-1))) + ((-1) * ( Type-2 * Log 2( Type-2))) Ѥ Flags Dataset = ((-1) * ( * ) ) + ((-1) * ( * ) Ѥ Flags Dataset = ( ) + ( ) Ѥ Flags Dataset = Information Entropy Ѥ of the UCI Flags Plus Target Dataset as Calculated in Tutorial DSB-1 (DSB-1-Formula-5)

4 This dataset has a high Parent Ѥ value (Ѥ Flags Dataset = ). Review all the interim calculations used for Ѥ of the UCI Flags Plus Target Dataset at Chameleon s Learning Lab: Link to Chameleon Learning Lab Dashboard (DSB-1-Dashboard-1) (DSB-1-URL-4) However, when we calculate Æ for each non-target attribute, we are looking for an attribute which reduces the disorder with respect to its expected target value. In other words, an attribute with a low Æ value can be used to bring order to a disordered dataset Nice. So, in a nutshell, we want datasets with high Ѥ values like the UCI Flags Plus Target Dataset. Within that dataset we also are looking for it to contain non-target attributes with low Æ. Because when we have high Ѥ at the dataset level it means there is disorder (mixed target values) of the target variable (i.e. a mix of Heads and Tails toss results or a mix of Type-1 and Type-2 records). And if we can discover an attribute within such a dataset which does a good job of predicting what its record s target value will be, it reduces disorder. We know we have a good attribute by its Æ value being much less than the overall Ѥ value calculated across the entire dataset with no consideration given to the value of any non-target attribute. NOTICE EXAMPLES USE MySQL DATABASE The remainder of this tutorial will be done using MySQL examples. For other databases (Oracle, SQL Server, PostgreSQL, etc.) the basic SQL shouldn t change much. But, you'll need to modify the catalog tables and their columns in the SQL as they will be different than the MySQL database catalog. NOTICE USERS OF Python, R, Weka, etc. The same approach would be used with Python, R, Weka, etc. Just keep in mind that where I will use an intermediate results table called uci_dataset_flags_child_attribute_counts. To accomplish the same thing with Python, R, Weka, etc., you would simply use Data Frames instead. To Repeat: the overall approach is the same for Python, R, Weka, etc., as the MySQL examples which will follow below. 2. CALCULATING ATTRIBUTE ENTROPY (Æ) FOR EACH ATTRIBUTE OF THE UCI FLAGS PLUS TARGET DATASET I ll begin by saying the text that follows is a little lengthy. I did this intentionally because so many tutorials I review seem to skip or omit key pieces of information you need to understand the material completely. This means that there is some overlap, especially with the SQL statements which follow. For example, all the functionality of SQL statements below, DSB-2-SQL-5, DSB-2-SQL-6 and DSB-2-SQL-7, could have all be accomplished with just DSB-2-SQL-7. But, the other two, DSB-2-SQL-5 and DSB-2-SQL-6 show the interim results and, hopefully, give the learner more insight.

5 Calculating Attribute Entropy (Æ) Components Using one of my favorite DBA tricks, we will use the database catalog to auto-generate the SQL needed for these commands. Statement DSB-2-SQL-1 generates the first SQL statement we will use. SELECT DISTINCT CONCAT( "UNION SELECT '", information_schema.columns.column_name, "' AS independent_variable, ", "COUNT(DISTINCT ", information_schema.columns.column_name, ") AS independent_value_count ", " FROM ", information_schema.columns.table_name, " GROUP BY independent_variable" ) AS sql_statement FROM information_schema.`columns` WHERE information_schema.`columns`.table_schema = 'ericth5_chameleon_tutorials_data_science' information_schema.`columns`.table_name = 'uci_dataset_flags_plus_target' GROUP BY information_schema.columns.column_name SQL Statement for Mining Attributes and Distinct Value Counts from UCI Dataset Flags Plus Target (DSB-2-SQL-1) NOTICE CATALOG GENERATED SQL ALWAYS NEEDS EDITS BEFORE EXECUTION Whenever generating SQL statements directly from a database catalog, it should be done using a repeatable pattern and UNION which will include each table attribute s results. However, that means if one leaves the first UNION in the generated SQL, it won t run. So, remove the first UNION. IMPORTANT: The first UNION needs to be remove before executing the SQL statement DSB-2-SQL-1 generates or it will not run UNION SELECT 'animate_image' AS independent_variable, COUNT(DISTINCT animate_image) AS independent_value_count FROM uci_dataset_flags_plus_target GROUP BY independent_variable UNION SELECT 'area' AS independent_variable, COUNT(DISTINCT area) AS independent_value_count FROM uci_dataset_flags_plus_target GROUP BY independent_variable UNION SELECT 'bars' AS independent_variable, COUNT(DISTINCT bars) AS independent_value_count FROM uci_dataset_flags_plus_target GROUP BY independent_variable UNION SELECT 'black' AS independent_variable, COUNT(DISTINCT black) AS independent_value_count FROM uci_dataset_flags_plus_target GROUP BY independent_variable UNION SELECT 'blue' AS independent_variable, COUNT(DISTINCT blue) AS independent_value_count FROM uci_dataset_flags_plus_target GROUP BY independent_variable Results not shown UNION SELECT 'white' AS independent_variable, COUNT(DISTINCT white) AS independent_value_count FROM uci_dataset_flags_plus_target GROUP BY independent_variable UNION SELECT 'zone' AS independent_variable, COUNT(DISTINCT zone) AS independent_value_count FROM uci_dataset_flags_plus_target GROUP BY independent_variable ORDER BY 2 DESC While not required, adding the ORDER BY 2 DESC statement will sort the results with highest counts at top Output from DSB-2-SQL-1 with Red Removed and Green Added for Execution Syntax Becomes Query DSB-2-SQL-2 (DSB-2-SQL-1-Results)

6 See the output from DSB-2-SQL-2 at Chameleon s online student lab environment, click here: Output from Query DSB-2-SQL-1 (DSB-2-SQL-1-Results) Next, we modify the output of SQL statement (DSB-2-SQL-1) above and execute that statement (now DSB-2-SQL-2) with its results shown below. The attributes chameleon_key and data_source_url were added as part of a Data Vault onboarding scheme and will be ignored as we go forward because those attributes were not part of the original dataset. Independent Variable Independent Value chameleon_key 194 name 194 area 136 population 48 count_of_sunstars 14 stripes 12 language 10 religion 8 count_of_colors 8 bottom_right_color 8 mainhue 8 top_left_color 7 landmass 6 bars 5 count_of_circles 4 zone 4 count_of_crosses 3 count_of_quarters 3 count_of_saltires 2 text 2 black 2 green 2 white 2 blue 2 icon_inanimate_image 2 red 2 animate_image 2 gold 2 chameleon_target_variable 2 orange 2 cresent 2 triangle 2 data_source_url 1 Output from Query DSB-2-SQL-2 (DSB-2-SQL-2-Results)

7 See the output from DSB-2-SQL-2 at Chameleon s online student lab environment, click here: Output from Query DSB-2-SQL-2 (DSB-2-SQL-2-Results-URL) Now that we have all the columns, let s take a second and discuss what we can learn just from looking at the results above shown as DSB-2-SQL-2-Results. While the first and last columns will be ignored as they weren t part of the original dataset, what else can we learn? The chameleon_key attribute is simply a surrogate key added for data management purposes. But, it also tells us there are 194 rows in this result set. Knowing that, we can immediately decide to ignore the name attribute as it has a 1:1 relationship with the surrogate key and, therefore, cannot be grouped in groups with the same name. The name attribute will have the best IG value, but at the cost of being too narrow in focus at this point in the process. Groups of one are most useful when one knows the exact identity in advance (i.e. with your credit card number, I can find the one person (group of one) associated with that card easily because it will return a group of one). If something can t be used for grouping, it s not a good candidate for classification either So, we dump the attribute name and won t use this attribute going forward. The attributes area and population are a little more complicated. On the one hand, there are a lot of possible values for both; 136 and 48 respectively. But, unlike name, these other two attributes are measurement values, which means we can t so easily dismiss them. While this will be covered more in later tutorials (DSB-4 and DSB-5), for now I ll just say that with variables like these are typically handled using a range of values to create a smaller number of groups than the 136 and 48 distinct values for the two attributes of the original dataset. Population has values ranging from 0 through 1008 in the UCI Flags Plus Target dataset with about 50% of the values being either 0 or 1. Maybe, we split the population attribute into two groups; ( Population <=1 and Population >1)... Maybe? but you get the point, we usually group attributes with many distinct possible values by range rather than each specific value. Once again employing one of my favorite DBA tricks, we use the database catalog to auto-generate the SQL needed without the NOT statements highlighted in red added to remove the chameleon_target_variable along with the other five attributes discussed above.

8 After the NOT statements are added, we now have SQL statement DSB-2-SQL-3: SELECT DISTINCT CONCAT( "UNION SELECT DISTINCT '", information_schema.columns.column_name, "', ", information_schema.columns.column_name, ", ", "chameleon_target_variable, ", -- "COUNT(", information_schema.columns.column_name, ") AS attribute_target_value_xref", " ", " FROM ", information_schema.columns.table_name, " GROUP BY ", information_schema.columns.column_name, ", chameleon_target_variable" ) AS sql_statement FROM information_schema.`columns` WHERE information_schema.`columns`.table_schema = 'ericth5_chameleon_tutorials_data_science' information_schema.`columns`.table_name = 'uci_dataset_flags_plus_target' NOT information_schema.columns.column_name = 'chameleon_key' NOT information_schema.columns.column_name = 'chameleon_target_variable' NOT information_schema.columns.column_name = 'data_source_url' NOT information_schema.columns.column_name = 'name' NOT information_schema.columns.column_name = 'area' NOT information_schema.columns.column_name = 'population' GROUP BY information_schema.columns.column_name SQL Statement for Calculating Target Variable Value Counts for Each Non-Target Attribute s Distinct Values (DSB-2-SQL-3) The results from SQL statement DSB-2-SQL-3 do not include the columns with NOT and, subsequently, becomes DSB-2-SQL-4 once the first UNION is removed to correct the syntax of the overall statement. IMPORTANT: The first UNION needs to be remove before executing the SQL statement DSB-2-SQL-1 generates or it will not run UNION SELECT DISTINCT 'animate_image', animate_image, chameleon_target_variable, COUNT(animate_image) AS attribute_target_value_xref FROM uci_dataset_flags_plus_target GROUP BY animate_image, chameleon_target_variable UNION SELECT DISTINCT 'bars', bars, chameleon_target_variable, COUNT(bars) AS attribute_target_value_xref FROM uci_dataset_flags_plus_target GROUP BY bars, chameleon_target_variable UNION SELECT DISTINCT 'count_of_circles', count_of_circles, chameleon_target_variable, COUNT(count_of_circles) AS attribute_target_value_xref FROM uci_dataset_flags_plus_target GROUP BY count_of_circles, chameleon_target_variable... Most results not shown UNION SELECT DISTINCT 'triangle', triangle, chameleon_target_variable, COUNT(triangle) AS attribute_target_value_xref FROM uci_dataset_flags_plus_target GROUP BY triangle, chameleon_target_variable UNION SELECT DISTINCT 'white', white, chameleon_target_variable, COUNT(white) AS attribute_target_value_xref FROM uci_dataset_flags_plus_target GROUP BY white, chameleon_target_variable UNION SELECT DISTINCT 'zone', zone, chameleon_target_variable, COUNT(zone) AS attribute_target_value_xref FROM uci_dataset_flags_plus_target GROUP BY zone, chameleon_target_variable Output from DSB-2-SQL-3 with First UNION Removed for Execution Syntax Then Becomes Query DSB-2-SQL-4 (DSB-2-SQL-4-Results)

9 See the output from DSB-2-SQL-3 at Chameleon s online student lab environment, click here: Output from Query DSB-2-SQL-3 (DSB-2-SQL-3-Results-URL) The abbreviated results from SQL statement DSB-2-SQL-4 are below (with 213 results not shown). Independent Variable Value Target Value Count of Attribute Value + Target Value animate_image 0 Type 1 88 animate_image 0 Type 2 67 animate_image 1 Type 1 24 animate_image 1 Type 2 15 bars 0 Type 1 95 bars 0 Type 2 64 bars 1 Type 1 1 bars 1 Type 2 5 bars 2 Type 1 3 bars 2 Type 2 4 bars 3 Type 1 12 bars 3 Type 2 9 bars 5 Type Results not shown zone 3 Type 1 11 zone 3 Type 2 5 zone 4 Type 1 34 zone 4 Type 2 24 Abbreviated Output from Query DSB-2-SQL-4 (DSB-2-SQL-4-Results) See the entire output from the DSB-2-SQL-4 query at Chameleon s online student lab environment here: Output from Query DSB-2-SQL-4 (DSB-2-SQL-4-Results-URL) Interim Results Required for Calculation of Attribute Entropy (Æ) The next step is to create the SQL statement, DSB-2-SQL-5, which will return the total number of records by {Independent Variable, Independent Variable Value, Target Value}. We need the results of DSB-2-SQL-5 to know what proportion of each record type is Type 1 vs. how many are Type 2 records. For example, using the attribute animate_image we can see that about 80% of its possible values equals 0 and about 20% equals 1. To get this done, we also need to know that for animate_image = 0, there are 155 records spanning both target values ( Type 1 to Type 2 ). Of those records, 56.77% have their target value equal to Type 1 and 43.23% have target value equal to Type 2. Of course, there s an SQL statement for this too. To make the SQL less lengthy, I ll create a new MySQL table based exactly on SQL statement DSB-2-SQL-4 and call the new table uci_dataset_flags_child_attribute_counts. This new MySQL table is populated with the exact same records as the results from SQL DSB-2-SQL-4 above.

10 Now, I ll need calculations at two different levels: Counts for each {variable + variable value + target value } combination from DSB-2-SQL-4; and Counts for each {variable + variable value} combination from DSB-2-SQL-5 for each attribute. The new table uci_dataset_flags_child_attribute_counts is created and loaded (SQL needed to do it yourself in box below) using the long SQL statement DSB-2-SQL-4_uci_dataset_flags_child_attribute_counts.sql. Since that SQL is really long, I didn t put the NOT statements inline below to save space. In DSB-2-SQL-5 below, we will be joining this temporary table, uci_dataset_flags_child_attribute_counts, to itself to get various component variables needed for the calculations from and forward. Once again, the entire set of calculations discussed here in DSB-2 can also be done with Python, R, Weka, etc. In those cases, data frames would be used instead of my temporary uci_dataset_flags_child_attribute_counts table. SQL to create and load the uci_dataset_flags_child_attribute_counts from the Learning Lab is here: Creates: uci_dataset_flags_child_attribute_counts SQL (DDL) to create and load the uci_dataset_flags_child_attribute_counts (DSB-2-SQL-DDL-1-URL) Remember This tutorial is using a MySQL here. When using Python, R, Weka, etc., you just use Data Frames instead of a MySQL table. Now SQL statement DSB-2-SQL-5 may be executed against the table uci_dataset_flags_child_attribute_counts : SELECT independent_variable_name, independent_value, SUM(target_value_count) FROM uci_dataset_flags_child_attribute_counts GROUP BY independent_variable_name, independent_value ORDER BY independent_variable_name, independent_value SQL Statement for Calculating Attribute Value Counts for Each Non-Target Attribute s Distinct Values (DSB-2-SQL-5) The abbreviated results from SQL statement DSB-2-SQL-5 are below (with 211 results not shown). Notice that regardless the number of possible distinct values for each attribute, the SUM of every attribute s counts for each of its possible values always adds up to 194. The only difference is how many classification groups we get for each. That s why we removed the name and population attributes because they create too many classes to be useful at the value-level.

11 Independent Variable Value Count Records This Value animate_image animate_image 1 39 bars bars 1 6 bars 2 7 bars 3 21 bars 5 1 black black Results not shown white 0 48 white zone 1 91 zone 2 29 zone 3 16 zone 4 58 Abbreviated Output from Query DSB-2-SQL-5 (DSB-2-SQL-5-Results) See the entire output from the DSB-2-SQL-5 query at Chameleon s online student lab environment here: Final Attribute Entropy (Æ) Calculations Output from Query DSB-2-SQL-5 (DSB-2-SQL-5-Results-URL) Here is the basic formula for calculating Æ for an attribute with n possible target values: Attribute Entropy (Æ) = ((p attribute-1 * Log 2(p attribute-1)))) + ((p attribute-2 * Log 2(p attribute-2))))... + ((p n * Log 2(p n)))) Basic Formula for Calculating Attribute Entropy (Æ) (DSB-2-Formula-1) DSB-2-Formula-1 above is the base formula for calculating Æ of an attribute with n possible target values. Let s walk through the formula using one of the attributes from the UCI Flags Plus Target. But, first we need to compute all the variables which go into the Information Gain for each attribute. This is SQL statement DSB-2-SQL-6 which executes against the table uci_dataset_flags_child_attribute_counts.

12 SELECT flags_count_1.independent_variable_name AS attribute_name, flags_count_1.independent_value AS attribute_value, flags_count_1.target_value, SUM(flags_count_2.target_value_count) AS attribute_value_count, flags_count_1.target_value_count AS attribute_and_target_count, ((flags_count_1.target_value_count)/(sum(flags_count_2.target_value_count))) AS proportion_this_target_value, (-1*(LOG2(((flags_count_1.target_value_count)/(SUM(flags_count_2.target_value_count)))))) AS log2_proportion, ((((flags_count_1.target_value_count)/(sum(flags_count_2.target_value_count)))) * ((-1*(LOG2(((flags_count_1.target_value_count)/(SUM(flags_count_2.target_value_count))))))) ) AS value_plus_target_entropy FROM uci_dataset_flags_child_attribute_counts flags_count_1, uci_dataset_flags_child_attribute_counts flags_count_2 WHERE flags_count_1.independent_variable_name = flags_count_2.independent_variable_name flags_count_1.independent_value = flags_count_2.independent_value GROUP BY flags_count_1.independent_variable_name, flags_count_1.independent_value, flags_count_1.target_value, flags_count_1.target_value_count ORDER BY 1,2,3 SQL Statement for Attribute Entropy Base Calculations (DSB-2-SQL-6) The abbreviated results from SQL statement DSB-2-SQL-6 are below (with 279 results not shown). Attribute Name Attribute Value (AV) Target Value (TV) Total AV Count TV Count This AV Proportion AV + TV LOG2 Proportion AV + TV Entropy Attribute + AV + TV animate_image 0 Type animate_image 0 Type animate_image 1 Type animate_image 1 Type bars 0 Type bars 0 Type bars 1 Type bars 1 Type Results not shown white 1 Type white 1 Type zone 1 Type zone 1 Type zone 2 Type zone 2 Type zone 3 Type zone 3 Type zone 4 Type zone 4 Type Abbreviated Output from Query DSB-2-SQL-6 (DSB-2-SQL-6-Results)

13 See the entire output from the DSB-2-SQL-6 query at Chameleon s online student lab environment here: Output from Query DSB-2-SQL-6 (DSB-2-SQL-6-Results-URL) Now that we have the pieces, we execute DSB-2-SQL-7, against table uci_dataset_flags_child_attribute_counts Notice that I am joining the same table to itself three (3) times: once for the Type 1 counts; a second time for the Type 2 counts; and a third time (labeled flags_count_3 ) to hold the SUM of the Type 1 and Type 2 record counts. The flags_count_3 values will be needed to store interim results when we calculate percentage for each target value in DSB-2-SQL-7 below. SQL TIP WHY JOINING A TABLE TO ITSELF IS SO USEFUL You will notice that much of the SQL in this tutorial uses the uci_dataset_flags_child_attribute_counts table by joining the table to itself two or three times. Using SQL statement DSB-2-7 as an example, joining the table to itself three times allows calculations to be done as each result is processed. Blue for Type 1 counts, Green for Type 2 counts and Orange for the SUM of the Type 1 and Type 2 counts. SELECT flags_count_1.independent_variable_name AS attribute_name, flags_count_1.independent_value AS attribute_value, Type 1 Entropy for Each Attribute Value + Target Value ((((flags_count_1.target_value_count)/(sum(flags_count_3.target_value_count)))) * ((-1*(LOG2(((flags_count_1.target_value_count)/(SUM(flags_count_3.target_value_count))))))) ) AS type_1_entropy, -- Type 1 Entropy for Each Attribute Value + Target Value Type 2 Entropy for Each Attribute Value + Target Value ((((flags_count_2.target_value_count)/(sum(flags_count_3.target_value_count)))) * ((-1*(LOG2(((flags_count_2.target_value_count)/(SUM(flags_count_3.target_value_count))))))) ) AS type_2_entropy -- Type 2 Entropy for Each Attribute Value + Target Value -- FROM uci_dataset_flags_child_attribute_counts flags_count_1, uci_dataset_flags_child_attribute_counts flags_count_2, uci_dataset_flags_child_attribute_counts flags_count_3 WHERE flags_count_1.independent_variable_name = flags_count_3.independent_variable_name flags_count_2.independent_variable_name = flags_count_3.independent_variable_name flags_count_1.independent_value = flags_count_3.independent_value flags_count_2.independent_value = flags_count_3.independent_value flags_count_1.target_value = 'Type 1' flags_count_2.target_value = 'Type 2' GROUP BY flags_count_3.independent_variable_name, flags_count_3.independent_value ORDER BY 1,2 SQL Statement for Calculating Attribute Entropy for Attribute Value + Target Value (DSB-2-SQL-7) The abbreviated results from SQL statement DSB-2-SQL-7 are below (with 105 results not shown).

14 Attribute Name Value Type 1 Attribute Entropy Æ Type 2 Attribute Entropy Æ animate_image animate_image bars bars bars bars black black blue blue Results not shown triangle triangle white white zone zone zone zone Abbreviated Output from Query DSB-2-SQL-7 (DSB-2-SQL-7-Results) See the entire output from the DSB-2-SQL-7 query at Chameleon s online student lab environment here: Output from Query DSB-2-SQL-7 (DSB-2-SQL-7-Results-URL) 3. CALCULATING INFORMATION GAIN (IG) FOR EACH ATTRIBUTE OF THE UCI FLAGS PLUS TARGET DATASET Calculating IG for a given attribute (or called a Feature in R) simply involves knowing the difference between the dataset s Ѥ and each attribute s Æ. The greater the difference, the more IG that attribute value provides. Dataset Entropy, Attribute Entropy and Information Gain (DSB-2-Infographic-4)

15 We revisit the basic formula for calculating Æ for an attribute with n possible target values first shown in the text above (immediately after SQL statement DSB-2-SQL-5). Information Gain (IG) EACH ATTRIBUTE = (Information Entropy (Ѥ) DATASET ) - (Attribute Entropy (Æ) EACH ATTRIBUTE ) Basic Formula for Calculating Attribute Entropy (Æ) (DSB-2-Formula-1) Now that we have all the components needed to calculate each attribute s Æ, we just modify SQL statement DSB- 2-SQL-7 above by adding + highlighted in yellow, removing AS type_1_entropy from the Type 1 block of SQL in blue and change the computed column s label to AS classification_entropy. DSB-2-SQL-8 Remember SQL sees two hyphens --' and anything written after them is treated as a comment and ignored SELECT flags_count_1.independent_variable_name AS attribute_name, flags_count_1.independent_value AS attribute_value, -- Total Entropy for Each Attribute Value ( -- Type 1 Entropy for Each Attribute Value + Target Value ((((flags_count_1.target_value_count)/(sum(flags_count_3.target_value_count)))) * ((-1*(LOG2(((flags_count_1.target_value_count)/(SUM(flags_count_3.target_value_count)))))))) -- Type 1 Entropy for Each Attribute Value + Target Value + -- Type 2 Entropy for Each Attribute Value + Target Value ((((flags_count_2.target_value_count)/(sum(flags_count_3.target_value_count)))) * ((-1*(LOG2(((flags_count_2.target_value_count)/(SUM(flags_count_3.target_value_count))))))) ) -- Type 2 Entropy for Each Attribute Value + Target Value ) AS classification_entropy -- Total Entropy for Each Attribute Value FROM uci_dataset_flags_child_attribute_counts flags_count_1, uci_dataset_flags_child_attribute_counts flags_count_2, uci_dataset_flags_child_attribute_counts flags_count_3 WHERE flags_count_1.independent_variable_name = flags_count_3.independent_variable_name flags_count_2.independent_variable_name = flags_count_3.independent_variable_name flags_count_1.independent_value = flags_count_3.independent_value flags_count_2.independent_value = flags_count_3.independent_value flags_count_1.target_value = 'Type 1' flags_count_2.target_value = 'Type 2' GROUP BY flags_count_3.independent_variable_name, flags_count_3.independent_value ORDER BY 1, 2 SQL Statement for Calculating Class Entropy by Attribute Value for Both Target Values (DSB-2-SQL-8) WARNING

16 ENTROPY CALCULATIONS SHOULD NEVER BE NEGATIVE When I first wrote DSB-2-SQL-8, I was getting a couple results which were negative numbers. This can t be from a mathematics standpoint. So, if you ever come up with a negative result you did something wrong. In my case, I forgot to multiply by the percentage of each target value which is shown above as Proportion This AV+TV from: DSB-2-SQL-6-Results above. The abbreviated results from SQL statement DSB-2-SQL-8 are below (with 107 results not shown). Remember that the results from DSB-2-SQL-8 below reflect the entropy of a class of records ( attribute_name + value ) spanning both target values (i.e. both Type 1 and Type 2 ). All that s needed here is to add the Type 1 Æ to the Type 2 Æ to get the entropy for the Attribute Value across both target values. Attribute Name Value Class Entropy animate_image animate_image bars bars bars bars black black blue blue Results not shown white white zone zone zone zone Abbreviated Output from Query DSB-2-SQL-8 (DSB-2-SQL-8-Results) Using attribute value animate_image = 0 as an example, we simply add both results from DSB-2-SQL-8-Results to demonstrate how the first row of DSB-2-SQL-9-Results is calculated. (Type 1 Æ = ) + (Type 2 Æ = ) = Formula for Calculating Attribute Entropy (Æ) for the animate_image attribute of the UCI Flags Plus Target Dataset (DSB-2-Formula-2) Put another way, the lower the Class Entropy the greater the IG of the class. And, the greater the IG of a class, the better that attribute will be in bringing more order (low class entropy) to a disordered dataset (has a high Ѥ value), such as the UCI Flags Plus Target which was used The terms class entropy and attribute entropy will be assumed to mean the same thing from this point forward.

17 See the entire output from the DSB-2-SQL-8 query at Chameleon s online student lab environment here: Output from Query DSB-2-SQL-8 (DSB-2-SQL-8-Results-URL) Since we re looking for the lowest attribute/class entropy statement DSB-2-SQL-8 is rewritten with a new ORDER BY and becomes DSB-2-SQL-9. Since this query sorts on Æ value, and since the lowest Æ (or class entropy) means the most IG, this query will present your past candidate first in the results. The final SQL statement, DSB-2-SQL-9, just changes the ORDER BY of DSB-2-SQL-8 so it returns the ( attribute_name + value ) with the most Information Gain (IG) at the top of the results. Typically, the ( attribute_name + value ) combination with the highest IG is the first branch of a classification tree... however, there are no definitive rules with this work and is why I used typically. Finally, if you were paying real close attention to the details, you have noticed that after query DSB-2-SQL-3 the WHERE clause statements which eliminate the attributes with too many distinct values haven t been used since then. These were identified by the NOT statements which were part of DSB-2-SQL-3 s WHERE clause. Of course, they could be inserted into DSB-2-SQL-4 through DSB-2-SQL-8 too, but were not included so DSB-2-SQL-4 through DSB-2-SQL-8 wouldn t take so much text space. However, since we re now looking for the candidate ( attribute_name + value ) combinations with the lowest entropy with DSB-2-SQL-9, we need to reintroduce the NOT statements now as part of DSB-2-SQL-9 so those attributes are not included in the final results set. The SQL statement DSB-2-SQL-9 is on the next page so the entire statement is on one page.

18 Remember SQL sees two hyphens --' and anything after them as a comment and ignores anything after the two hyphens SELECT flags_count_1.independent_variable_name AS attribute_name, flags_count_1.independent_value AS attribute_value, -- Total Entropy for Each Attribute Value Type 1 Entropy for Each Attribute Value + Target Value ((((flags_count_1.target_value_count)/(sum(flags_count_3.target_value_count)))) * ((-1*(LOG2(((flags_count_1.target_value_count)/(SUM(flags_count_3.target_value_count))))))) ) -- Type 1 Entropy for Each Attribute Value + Target Value Type 2 Entropy for Each Attribute Value + Target Value ((((flags_count_2.target_value_count)/(sum(flags_count_3.target_value_count)))) * ((-1*(LOG2(((flags_count_2.target_value_count)/(SUM(flags_count_3.target_value_count))))))) ) AS classification_entropy -- Type 2 Entropy for Each Attribute Value + Target Value Total Entropy for Each Attribute Value FROM uci_dataset_flags_child_attribute_counts flags_count_1, uci_dataset_flags_child_attribute_counts flags_count_2, uci_dataset_flags_child_attribute_counts flags_count_3 WHERE flags_count_1.independent_variable_name = flags_count_3.independent_variable_name flags_count_2.independent_variable_name = flags_count_3.independent_variable_name flags_count_1.independent_value = flags_count_3.independent_value flags_count_2.independent_value = flags_count_3.independent_value flags_count_1.target_value = 'Type 1' flags_count_2.target_value = 'Type 2' NOT flags_count_3.independent_value = 'chameleon_key' NOT flags_count_3.independent_value = 'chameleon_target_variable' NOT flags_count_3.independent_value = 'data_source_url' NOT flags_count_3.independent_value = 'name' NOT flags_count_3.independent_value = 'area' NOT flags_count_3.independent_variable_name = 'population' GROUP BY flags_count_3.independent_variable_name, flags_count_3.independent_value ORDER BY 3 SQL Statement for Calculating Class Entropy by Attribute Value for Both Target Values (DSB-2-SQL-9)

19 The abbreviated results from SQL statement DSB-2-SQL-9 are below: Attribute Name Value Class Entropy bars count_of_colors landmass triangle stripes language Results not shown count_of_sunstars 6 1 stripes 9 1 language 5 1 count_of_sunstars 3 1 Abbreviated Output from Query DSB-2-SQL-9 (DSB-2-SQL-9-Results) See the entire output from the DSB-2-SQL-9 query at Chameleon s online student lab environment here: Output from Query DSB-2-SQL-9 (DSB-2-SQL-9-Results-URL) That's the end of this tutorial! In the next of this series of tutorials, DSB-3: Loading UCI Flags Plus Target into A Data Vault, I ll walk the learner through how we keep all the data, SQL, images and interim results managed so there are not constantly recalculated over and over again. When that happens, different people recalculate the same metric using different methods, tools (Python vs. R) or data. Why do you think statements like this are heard so often? The numbers from these two reports should match, but they re not even close Tutorial DSB-3: Loading UCI Flags Plus Target into A Data Vault will explain how to avoid that problem with a Data Vault ETL environment working in conjunction with Chameleon Metadata so you do it once and reuse it everywhere. See you then. ### By Eric Thornton, ET@ChameleonMetadata.com