A skeleton file has been provided for you to use in both checkpoints.

Transcription

1 University of Illinois, Urbana-Champaign ECE190 MP5 Website Crawler Program Description: In this MP, you need to write a C program to process a list of websites given to you in a series of text files. For the first checkpoint MP5.1, you will need to be able to properly use a provided API (Application Program Interface) in order to crawl through a website and store necessary information into an array of structures. Each structure will maintain its own singly linked list of website links as well as various other information. In the second checkpoint MP5.2, you will need to be able to find the shortest path between a starting point and end point within the website using a recursive pathfinding algorithm. A skeleton file has been provided for you to use in both checkpoints. Checkpoint 1 For checkpoint 1, you will be using a provided API that is capable of extracting HTML links from a website. Download the MP5.zip file from the website, it contains an object file with executable code and a header file "ece190.h" that you will need to include in your program. The following functions are in the API: void getlinks ( char *url, web_t *website, link_t* (*createlink) (char *url), void (*insertlink)(link_t **, link_t*) ); getlinks takes in four arguments - (1) a pointer to a string that contains a URL of a website, (2) a pointer to one web_t structure, and (3) a pointer to a function that will be called to create a linked list item, and (4) a pointer to a function that will insert a link into a singly linked list. getlinks will only find up to 100 URLs and will cease to function after 100 URLs have been found during the program's execution. For each link that getlinks finds in the webpage, it will call the function createlink that you pointed to when calling getlinks in order to make a new link_t structure. insertlink will then be called to place that previously created link into the linked list structure pointed to from within website.

2 Each time that you call getlinks, you will need to iterate through the linked list and copy data from the nodes that are brand new into unused elements in the array of web_t structures. Then for each new array element that has been populated you will need to run getlinks again for it. Your website crawler will need to be able to maintain records on up to 100 websites, where each website's information will be stored in the web_t structure as defined below. The first element in the array should always describe the homepage URL that was passed in at the beginning of the program. typedef struct { char *url link_t *links; enum visited {YES, NO} flag; int numsitestodestination; } web_t; The variable links is the head pointer to a singly linked list of items. To learn more about singly linked lists, please refer to the course text chapter Please use the following structure for your linked lists: typedef struct { int index; char *url link_t *next; } link_t; Definitions of these two structures are found in ece190.h.

3 In our example, the final data structure might look something like this: Please implement the following functions in your code: void crawlweb (char *url); link_t* createlink (char *url); void insertlink (link_t** list_head, link_t* new_list_item); void deletelist (link_t *head); void freememory (); crawlweb is where you will need to populate your data structures by using createlink, insertlink, and the provided API. createlink is responsible for dynamically allocating a new link_t structure and saving the input string into the new structure, and returning a pointer to the new linked list item. The new structure must have an initial index value of -1 and next pointer equal to NULL. insertlink will insert new_list_item into the singly linked list pointed to by list_head. It is vital that the list be sorted by the url data field. Please see the help files for strcmp as that will be a very useful function for sorting. freememory is called at the end of crawlweb, and must deallocate all memory locations that were dynamically allocated during the program's execution. deletelist is a recursive function that will delete all items in a linked list. You will need to deallocate URL strings. crawlweb will need to print data about all the websites, including the linked list information as indicated in the gold files. After you have found 100 websites (or there are no more websites to be found) and created linked lists of HTML links for each website, you must then populate all the index fields in all the elements of each linked list. The function main has already been compiled in an object file you will need to download (see section for compiling code). Do not try to create a main function in your code.

4 Checkpoint 2 For checkpoint 2, you will need to find the shortest path between two webpages using a recursive sorting algorithm. It is required that you print the shortest path after the recursive search is finished. This can be done using recursion. See the graph below: Please use the following functions in your code: int findpath (int start, int end); void printpath (); findpath must recursively search through the list of links for the shortest path; it returns the value of the shortest path to the destination. From the starting location, set all the visited enums to YES for each link that the start is connected to. Then for each link call findpath again. You may need to store the return values of findpath in numsitestodestination. printpath will print out the shortest path. In our example, the output to the terminal would be: Shortest Path from to => For additional resources and help, please look up breadth-first search on Wikipedia and library texts on algorithms.

5 Additional Graded Questions At the BOTTOM of your code in MP5.2, please answer the following questions in comments. 1) Why is it important to free memory that is dynamically allocated? 2) What happens if we do not keep track of which websites have already been visited in the search algorithm? What happens if we do not set each visited variable to YES before any recursive calls? Compiling Code Before you can test your program you must create an executable. For checkpoint 1, you may do so by typing "gcc -Wall -Werror -g -ansi mp5.1.c ece190.o -o mp5" at the terminal when you are in the same directory as your code. For checkpoint 2, it should be "gcc -Wall -Werror -g -ansi mp5.2.c ece190.o -o mp5". If there are no errors, the binary executable file will be created called mp5. Otherwise you will need to fix any errors found and warnings that are reported. Keep copies of your code and be careful. Do not accidentally overwrite your code by typing the GCC command incorrectly! Testing In order to run mp5, a website address must be provided to the program when calling it from the terminal. The command "mp5 yoururlhere" will start the program with the homepage set to yoururlhere. When typing this command you are actually passing arguments into the main function! If you type "mp5 into the terminal for instance, the webcrawler will crawl through UIUC's website. Several sample websites have been provided to you for testing. Please use the URLs listed below: Gold output files for each test case can be found at the following URLs: You are more than welcome to attempt to use it on other websites as well, like or for instance but we are unable to provide gold files for those websites. When running this code on external websites, it may take several minutes for results to appear.

6 Additional test websites may or may not be uploaded several days before the MP5.1 and MP5.2 deadlines. Please refer to the README for updates. Grading Functionality (70%) Checkpoint 1 (35%) Program properly implements linked lists and dynamic allocation. Program properly reads in website data. Program properly frees all memory dynamically allocated. Checkpoint 2 (35%) Program properly finds shortest path from any node to any other node. Program properly uses recursion. Style and Documentation (20%) Good and consistent indentations (5%) Introductory paragraph (5%) Commenting properly (5%) Using unambiguous variable names (5%) Graded Questions (10%) Concise, thoughtful answers. James Scholar Component Checkpoint JS3 Determine the page rank of each website in your array, where the page rank is defined as the number of times that particular page is linked to by other pages. Finally, print the list of websites where the highest ranked pages are printed first (decreasing order). The web_t structure has a spare integer variable js_page rank for you to use, as well as the function printpageranking.