Use JSL to Scrape Data from the Web and Predict Football Wins! William Baum Graduate Sta/s/cs Student University of New Hampshire

Transcription

1 Use JSL to Scrape Data from the Web and Predict Football Wins! William Baum Graduate Sta/s/cs Student University of New Hampshire

2 Just for Fun! I m an avid American football fan Sports sta/s/cs are easily accessible, online My goal: to predict total wins in a season, by team These techniques can be applied to any web data It s fun to see how JMP Scrip/ng Language [JSL] can, at once, automate data clean up and perform modeling With a linle JSL code this really is easy! JMP actually helps you write the code through plaoorms It s straighoorward and can be quite powerful Saves a lot of /me when upda/ng models with new data

3 JSL Resources As a newcomer to JSL scrip/ng I did not always know the best place to turn for answers Many resources exist JMP online community: JMP website & online forums Excellent resources are available from JMP: Help Menu > JSL func/ons Online Scrip/ng Guide & Syntax Reference hnp:// hnp:// Books that will save you loads of /me: Jump Into JMP Scrip>ng, by Wendy Murphrey and Mary Lucas JSL Companion, by Theresa Utlaut, Georgia Morgan, & Kevin Anderson

4 Prepara/on Create folders to store files: Any required directories or subdirectories Good way to organize files created from different data sources Directory names will be used in JSL scripts

5 Scrape Data: Team Sta/s/cs Offense, Defense, Special Teams 1. Set local path for file storage: dir = Pick Directory( Select directory for file storage, show files(0)); 2. Get data from a website & save the file (one for each HTML table): ObjectName=Open( hnp//:url_name1.com, HTML Table(#)) << Set Name( Name ) << Save(dir filename.csv ); Simply copy and paste URL names into your code from your browser To put a link to the website directly in your script helps to open webpage while edi/ng code: Web("hNp://url_name.com"); 3. Why save as CSV file first? Data Import Op/ons, to support automated data import: Import Seqngs(...), Labels(1), Column Names Start(2), Data Starts(3), etc. Delete empty columns! Can also use For loop to eliminate them.

6 The Code: Scrape Data

7 The Files JMP will now automa/cally pull the informa/on from the web and save the files. Tes/ng may be required! No need to save these files, but: Data can be a bit messy Easier for debugging One can see the imported tables and adjust the code

8 Standings Data (Wins, Losses, etc.) Get addi/onal data from an another website Again, using a linle object- oriented scrip/ng: ObjectName = Open( hnp://url_name2, HTML Table(#) << Set Name( TableName ) << Save(dir FileName2.jmp ) This /me, the data do not require any pre- processing to parse correctly Nice! Imported directly into JMP and saved

9 The Code: Standings Data ObjectName = Open( hnp://url_name2, HTML Table(#) << Set Name( TableName ) << Save(dir FileName.jmp )

10 Cell Processing Some/mes even slight varia/ons in individual strings can foul up your code In this case, the playoffs are near To indicate New England has clinched a division /tle, a z is inserted into their name

11 The Code: Cell Processing Cell processing to the rescue! JMP loops through each row of the Team name column For Each Row command makes for easy looping Condi/onal If, then phrase Respec/vely, Contains and Substr commands: returns the posi/on of a specified item within a string returns part of a string: star/ng at a specified posi/on, returns a specified number of characters

12 Reopen Files, Concatenate & Sort Reopen the files, invisibly Join different columns, same row labels Concatenate different rows, same columns Sort the rows of the resultant file by team name, alphabe/cally, to ensure like rows are matched during the joining process, later on Save the new table, which contains the aggregate informa/on

13 The Code: Open, Concatenate & Sort

14 Create a Fresh Table Add the appropriate number of rows: one for each team Create columns with various anributes In this example, JMP simply creates a column of team names in a new table It is ouen useful to create a new table to ensure the data are used appropriately during script execu/on, and are not accidentally lost or altered

15 The Code: Create a Fresh Table

16 Joining Tables This can be annoying without JMP: JMP has a nice GUI for joining tables Tables plaoorm Naming conven/ons are important, but JMP automa/cally renames columns origina/ng from different tables that share the same name Once you have joined the tables, simply copy and paste the table script of the resultant table into your larger, custom script Otherwise, you will have a lot of typing to do Repeat, as necessary, for each join

17 The Code: Joining Tables

18 Rename & Reformat Columns To replace odd- sounding column names: First, create a list of preferred names, then rename the columns using the combina/on of a For Loop and the Set Name func/on

19 Automate Column Reorganiza/on Once the data is collected and new columns are created, one may want to change the order of columns in the data table It is nice to do this automa/cally with JSL Easy with Move Selected Columns command

20 Data Table View

21 Develop the Associa/ve Array Ra/onale: I wanted to compare informa/on contained in cells located in different rows and columns of my table Associa/ve array was very useful for this task In this case, the abbreviated city name for each team is a key Values found in another column, called Strength in the same row. My goal was to predict how strong a team s future performance would be, not only based on their own past performance, But, also rela/ve to their future opponents past performance Strength is calculated from a combina/on of wins & net pts

22 17- week schedule: Keys

23 The Code: Simple Associa/ve Array Create the associa/ve array, then a formula column The formula references the correct value through a subscript of the associa/ve array, using the weekly opponent as the key Create a new column for each week to compare results:

24 Coun/ng Predicted Wins, by Column A simple way to count column results: Create a new formula column Sum a series of If- then condi/ons for each column value In this case, if the difference in Strength was posi/ve, the predic/on was considered a win and counted.

25 Create Predic/ve Models Use any of several JMP modeling techniques to create predic/ons: Con/nuous: Fit Model plaoorm Classifica/on: Neural Net, etc. Develop the op/mal model in one of JMP s plaoorms Copy and paste the model script into custom script, Save future model scripts to the data table, with JSL; The model will be available in the list of data table scripts of the new table, or One can also set the model to run automa/cally

26 Create Model & Save to Table

27 Boosted Neural Net JMP Pro

28 Elas/c Net JMP Pro

29 Model Averaging The average of mul/ple model predic/ons may outperform individual model performance: JMP Pro performs this opera/on with a couple of clicks

30 Model Comparison JMP Pro Average of Neural Net HB and Elas/c Net HB models performed slightly bener

31 Model Comparison Graphics

32 Conclusions With JSL, JMP can scrape data from mul/ple web sources, process the data, perform mul/ple predic/ons, compare them, and display them, all within a single script The effec/ve use of JSL can save a lot of /me, especially when one must repeat an analysis on a regular basis

33 Thank you! Thanks to Wendy Murphrey, for recommending the JSL reference sources Thanks to Peter Mroz, for showing me the usefulness of Associa/ve Arrays Thank you for listening! Have a great JMP Discovery Summit!

34 Ques/ons?