Environment Settings on SIT

Transcription

1 Environment Settings on SIT ~/jhuang/website: The root directory of the jhuang/website web application. Store here HTML, Servlet and JSP files ~/jhuang/website/web-inf: All resources for the web application that are not in the root directory. Store here web.xml, which is not allowed to use on AML server!! ~/jhuang/website/web-inf/classes: Servlet and utility classes ~/jhuang/website/web-inf/lib: Utility JAR files

2 Directory Structure of Your Group Accounts on Linux SIT Server All Web files are put under the website directory. All their applications can read and write anywhere in their group home directories, provided that they set Unix/Linux permissions properly. Your applications are free to connect to the DB servers. All the groups are NOT allowed to change ANY of the port number in the config files (e.g. (1) should not modify files named server.xml and server-minimal.xml; (2) should not create your own web.xml file). Use source /javainit to start Java 1.5.

3

4

5 Referring to your Files in the Browser To open a page called a.htm in the ROOT directory of the jhuang web application: To open a servlet called HelloWorld.class of the dbi web application

6 Running the Programs (via http) (via https)

7 Introduction to JDBC

8 JDBC Introduction JDBC provides a standard library for accessing relational databases API standardizes Way to establish connection to database Approach to initiating queries Method to create stored (parameterized) queries The data structure of query result (table) Determining the number of columns Looking up metadata, etc. API does not standardize SQL syntax JDBC is not embedded SQL JDBC class located in java.sql package Note: JDBC is not officially an acronym; unofficially, Java Database Connectivity is commonly used

9 On-line Resources Sun s JDBC Site JDBC Tutorial bc/ List of Available JDBC Drivers c/drivers/ API for java.sql sql/ package-summary.html

10 JDBC Drivers JDBC consists of two parts: JDBC API, a purely Java-based API JDBC Driver Manager, which communicates with vendor-specific drivers that perform the real communication with the database. Point: translation to vendor format is performed on the client No changes needed to server Driver (translator) needed on client JDBC Driver Manager Database Java Application JDBC API JDBC-ODBC Bridge Vendor Specific ODBC Driver JDBC Driver API Vendor Specific JDBC Driver Database

11 Seven Basic Steps in Using JDBC Load the driver Define the Connection URL Establish the Connection Create a Statement object Execute a query Process the results Close the connection

12 JDBC: Details of Process Load the driver try { Class.forName("oracle.jdbc.driver.OracleDriver"); Class.forName("org.gjt.mm.mysql.Driver"); } catch { ClassNotFoundException cnfe) { System.out.println("Error loading driver: " cnfe); } Define the Connection URL String host = yourcompany.com"; String dbname = "somename"; int port = 1234; String oracleurl = "jdbc:oracle:thin:@" + host + ":" + port + ":" + dbname; String mysqlurl = "jdbc:mysql://" + host + ":" + port + "/" + dbname;

13 JDBC: Details of Process Establish the Connection String username = jimmy_huang"; String password = "secret"; Connection connection = DriverManager.getConnection (oracleurl, username, password); Optionally, look up information about the database DatabaseMetaData dbmetadata = connection.getmetadata(); String productname = dbmetadata.getdatabaseproductname(); System.out.println("Database: " + productname); String productversion = dbmetadata.getdatabaseproductversion(); System.out.println("Version: " + productversion);

14 JDBC: Details of Process Create a Statement Statement statement = connection.createstatement(); Execute a Query String query = "SELECT col1, col2, col3 FROM sometable"; ResultSet resultset = statement.executequery(query); To modify the database, use executeupdate, supplying a string that uses UPDATE, INSERT, or DELETE Use setquerytimeout to specify a maximum delay to wait for results

15 JDBC: Details of Process Process the Result while(resultset.next()) { } System.out.println(resultSet.getString(1) + " " + First column has index 1, not 0 resultset.getstring(2) + " " + resultset.getstring(3)); ResultSet provides various getxxx methods that take a column index or name and returns the data Close the Connection connection.close(); As opening a connection is expensive, postpone this step if additional database operations are expected

16 Example on AML public void connect ( ) { con = null; try { DriverManager.registerDriver(new oracle.jdbc.driver.oracledriver()); String constring = "jdbc:oracle:thin:@sigma.aml.yorku.ca:1521:" + db_name; // System.out.println ("Connection string: + constring); con = DriverManager.getConnection(conString, db_username, db_password); System.out.println("\n\tConnection to " + db_name + " is established\n"); stmt=con.createstatement(); } catch(sqlexception e) { System.out.println("DB.connect => Error: " + e.tostring()); System.exit(1); }

17 Example on AML //query database ResultSet resultset = stmt.executequery( put your SQL query here" ); // process query results StringBuffer results = new StringBuffer(); ResultSetMetaData metadata = resultset.getmetadata(); int numberofcolumns = metadata.getcolumncount(); for ( int i = 1; i <= numberofcolumns; i++ ) { results.append( metadata.getcolumnname( i ) + "\t" ); } results.append( "\n" );

18 Types of Recommendation Systems

19 Product Recommendation in E-commerce Products Recommendations

20 Product Recommendation in E-commerce Products Recommendations

21 Outline Types of recommendation systems Search-based recommendations Category-based recommendations Collaborative filtering Clustering Association rules Content-based recommendations Classifiers

22 Search-based Recommendations The only visitor types a search query data mining customer The system retrieves all the items that correspond to that query The system recommend some of these books based on general, non-personalized ranking (sales rank, popularity, etc.)

23 Search-based Recommendations Pros: Simple to implement Cons: Which criteria to use to rank recommendations? Is it really recommendations? The user only gets what he asked

24 Category-based Recommendations Each item belongs to one category or more. Explicit / implicit choice: The customer select a category of interest (refine search, opt-in for category-based recommendations, etc.). Subjects > Computers & Internet > Databases > Data Storage & Management > Data Mining The system selects categories of interest on the behalf of the customer, based on the current item viewed, past purchases, etc. Certain items (bestsellers, new items) are eventually recommended

25 Category-based Recommendations Pros: Still simple to implement Cons: Which criteria to use to order recommendations? Is it really recommendations? Capacity highly depend upon the kind of categories implemented Too specific: not efficient Not specific enough: no relevant recommendations

26 Collaborative Filtering Collaborative filtering techniques compare customers, based on their previous purchases or page viewed to make recommendations to similar customers It s also called social filtering Follow these steps: Find customers who are similar (nearest neighbors) in term of tastes, preferences, past behaviors Aggregate weighted preferences of these neighbors Make recommendations based on these aggregated, weighted preferences (most preferred, unbought items)

27 Collaborative Filtering An Example: the system needs to make recommendations to customer C Book 1 Book 2 Book 3 Book 4 Book 5 Book 6 Customer A X X Customer B X X X Customer C X X Customer D X X Customer E X X Customer B is very close to C (he has bought all the books C has bought). Book 5 is highly recommended Customer D is somewhat close. Book 6 is recommended to a lower extent Customers A and E are not similar at all. Weight=0

28 Collaborative Filtering Pros: Extremely powerful and efficient Very relevant recommendations (1) The bigger the database, (2) the more the past behaviors, the better the recommendations Cons: Difficult to implement, resource and time-consuming What about a new item that has never been purchased? What about a new customer who has never bought anything? Cannot be compared to other customers no items can be recommended

29 Clustering Another way to make recommendations based on past purchases of other customers is to cluster customers into categories Each cluster will be assigned typical preferences, based on preferences of customers who belong to the cluster Customers within each cluster will receive recommendations computed at the cluster level

30 Clustering Book 1 Book 2 Book 3 Book 4 Book 5 Book 6 Customer A X X Book 1 Book 2 Book 3 Book 4 Book 5 Book 6 Customer AB X X X X X Customer B X Customer C X X Customer Customer D X X Customer E X X Customers B, C and D are clustered together. Customers A and E are clustered into another separate group Typicical preferences for CLUSTER are: Book 2, very high Book 3, high Books 5 and 6, may be recommended Books 1 and 4, not recommended at all

31 Clustering Book 1 Book 2 Book 3 Book 4 Book 5 Book 6 Customer A Book X 1 Book 2 Book 3 Book X 4 Book 5 Book 6 Customer BA X X X X X Customer CB X X X Customer DC X X X Customer DE X X X X Customer EF X X X How does it work? Any customer that shall be classified as a member of CLUSTER will receive recommendations based on preferences of the group: Book 2 will be highly recommended to Customer F Book 6 will also be recommended to Customer F to some extent

32 Clustering Pros: Clustering techniques work on aggregated data: faster It can also be applied as a first step for the selection of relevant neighbors in a collaborative filtering algorithm Cons: Recommendations (per cluster) are less relevant than collaborative filtering (per individual)

33 Association Rules Clustering works at a group (cluster) level Collaborative filtering works at the customer level Association rules work at the item level

34 Association Rules Past purchases are transformed into relationships of common purchases Book 1 Book 2 Book 3 Book 4 Book 5 Book 6 Customer A X X Customer B X X X Customer C X X Customer D X X Customer E X X Customer F X X Customers who bought Also bought Book 1 Book 2 Book 3 Book 4 Book 5 Book 6 Book Book Book Book 4 1 Book Book 6 1

35 Association Rules These association rules are then used to made recommendations If a visitor has some interest in Book 5, he will be recommended to buy Book 3 as well Of course, recommendations are constrained to some minimum levels of confidence Customers who bought Also bought Book 1 Book 2 Book 3 Book 4 Book 5 Book 6 Book Book Book Book 4 1 Book Book 6 1

36 Association Rules What if recommendations can be made using more than one piece of information? Recommendations are aggregated If a visitor is interested in Books 3 and 5, he will be recommended to buy Book 2, than Book 1 Customers who bought Also bought Book 1 Book 2 Book 3 Book 4 Book 5 Book 6 Book Book Book Book 4 1 Book Book 6 1

37 Association Rules Pros: Fast to implement Fast to execute Not much storage space required Very successful in broad applications for large populations, such as shelf layout in retail stores Cons: Not suitable if knowledge of preferences change rapidly.

38 Content-based Recommendations Association rules compare items based on past purchases or viewed Content-based filtering compares items based on their content Also called content-based filtering or information filtering

39 Content-based Recommendations What is the content of an item? It can be explicit attributes or characteristics of the item. For example for a film: Action / adventure Feature Bruce Willis Year 1995 It can also be textual content (title, description, table of content, etc.) Several techniques exist to compute the distance between two textual documents

40 Content-based Recommendations How does it work? A textual document is scanned and parsed Word occurrences are counted (may be stemmed) Several words or tokens are not taken into account. That includes stop words (the, a, for), and words that do not appear enough in documents Each document is transformed into a normed TFIDF vector of size N (Term Frequency/Inverted Document Frequency). TFIDF = ( TF IDF ) 2 N TF The distance between any pair of vector is computed IDF

41 Content-based Recommendations TFIDF = TF IDF ( TF IDF ) 2 N Terms that appear in many different documents are less indicative of overall topic TF i = document frequency of term i = number of documents containing term i IDF i = inverse document frequency of term i, = log 2 (N/ df i) (N: total number of documents)

42 Content-based Recommendations An (unrealistic) example: how to compute recommendations between 8 books based only on their title? Books selected: Building data mining applications for CRM Accelerating Customer Relationships: Using CRM and Relationship Technologies Mastering Data Mining: The Art and Science of Customer Relationship Management Data Mining Your Website Introduction to marketing Consumer behavior marketing research, a handbook Customer knowledge management

43 building data mining applications for crm Accelerating Customer Relationships: Using CRM and Relationship Technologies Mastering Data Mining: The Art and Science of Customer Relationship Management Data Mining Your Website COUNT Introduction to marketing consumer behavior marketing research, a handbook a 1 accelerating 1 and 1 1 application 1 art 1 behavior 1 building 1 consumer 1 crm 1 1 customer data for 1 handbook 1 introduction 1 knowledge 1 management 1 1 marketing 1 1 mastering 1 mining of 1 relationship 2 1 research 1 science 1 technology 1 the 1 to 1 using 1 website 1 your 1 customer knowledge management

44 building data mining applications for crm Accelerating Customer Relationships: Using CRM and Relationship Technologies Mastering Data Mining: The Art and Science of Customer Relationship Management Mastering Data Mining: The Art and Science of Customer Relationship TFIDF Normed Vectors Data Mining Your Website Introduction to marketing consumer behavior Data mining your website marketing research, a handbook customer knowledge management a accelerating Management and application art behavior building consumer crm customer data for handbook introduction knowledge management marketing mastering mining of relationship research Data science technology the to using website your

45 Content-based Recommendations A customer is interested in the following book: Building data mining applications for CRM The system computes distances between this book and the 7 others The closest books are recommended: #1: Data Mining Your Website #2: Accelerating Customer Relationships: Using CRM and Relationship Technologies #3: Mastering Data Mining: The Art and Science of Customer Relationship Management Not recommended: Introduction to marketing Not recommended: Consumer behavior Not recommended: marketing research, a handbook Not recommended: Customer knowledge management