Project Report on. Google s schema.org Mark-up for Education. Inaugral Regular Internship Program 2014

Transcription

1 Project Report on Google s schema.org Mark-up for Education Inaugral Regular Internship Program 2014 Submitted in partial fullfillment of internship project By Sanket Gaurav (Sikkim Manipal Institute of Technology) Sudhakar Yadav (National Institute of Technology, Raipur) Jithendranath Gupta Y. (Amrita University, Coimbatore) Kasturi BhanuChander (VITAE, Hyderabad) Mentor: Aruna Adil Program Director: Avinash Awate Under the Guidance of Prof. Deepak B. Phatak Department of Computer Science and Engineering, Indian Institute of Technology, Bombay 1

2 Acknowledgement Our heart pulsates with the thrill for tendering gratitude to those persons who helped me in completion of the project. Unfortunately, the list of expressions of thank no matter how extensive is always incomplete and inadequate. Indeed this page of acknowledgment shall never be able to touch the horizon of generosity of those who tendered their help to me. First of all we would like to thank our guide Dr. Deepak B. Phatak, Professor, Department Of Computer Science and Engineering, Indian Institute of Technology, Bombay for hosting the 6 month Regular Internship under which we could complete our 6 month internship and for his kind attitude, invaluable guidance, keen interest, immense help, inspiration and encouragement which helped us in carrying out our application development. Then we are thankful to Google for giving this project to us and also sponsoring our project which takes care of all the expenses for the whole duration of the project. We are extremely grateful to Mr. Avinash Awate, Program Director and Aruna Adil, Project Mentor, Deptt. Of CSE for providing all kinds of possible help throughout the phase of development of application. We are also thankful to Mr. Ashwani Sharma, Program Manager, Google, India and Ewa Gasperowicz, Developer Programs Engineer, Google, London for helping and solving our doubts related to schema.org etc. We are also very thankful to the Department of Computer Science and Engineering, I.I.T, Bombay for allowing us to carry out this internship and providing the resources. Lastly, we thank all those who are involved directly or indirectly in the successful completion of the project work. Sanket Gaurav Sudhakar Yadav Jitendranath Gupta Yenduri Bhanuchander Kasturi 2

3 List of Content Chapter Title Page No. Abstract 8 Introduction 1.1 General Overview of the Problem 1.2 Objective Literature Survey Schema.org LRMI Kritikos Schools Interoperability Framework(SIF) 21 3 Current Scenario Our Approach SRS Software Platform Hardware Platform Project Scheduling Analysis of websites CDEEP Spoken Tutorial ekshiksha Software Development Design Strategy for the solution Context Diagram Test Plan Structured Data Testing Tool Google Webmaster Tool Implementation Details Results and Discussion Result at Structured data testing tool

4 Result at Webmaster Tool Proposed Tags Summary and Conclusion Summary of Achievements Difficulties Encountered Limitations of the Project Future Scope of the Work Any Special Observation 73 User Manual Appendix 1 77 Appendix Appendix References 86 4

5 List of Figures Figure No. 1 2 Figure name The actual flow of web search for information Display of Amitabh Bachchan description on right side panel Page No Display of IIT Bombay description on right side pane Flow at LRMI The search results after adding LRMI and 'schema.org' markup The search results without adding 'schema.org' markup 16 7 Purpose of internet use by school children at KV, Mysore 23 8 Problems faced by children while using Internet 23 9 The various ways search is performed over web The flow of Educational search when we add 'schema.org' Gantt Chart Overall System Design or architecture of the HTML Processing Software Structure of HTML Processing Software Approach used in the HTML Processing Software Level 0 diagram of HTML Processing Software Test at Structured data tool before adding Schema properties Test at Structured data tool after adding Schema properties Test at Structured data tool after adding Schema properties to all the departments listed in CDEEP Test at Structured data tool after adding Schema properties to all the tutorials list in Spoken-tutorial. Test at Structured data tool after adding

6 Schema properties to all the workshop contents listed in Spoken-tutorial Test at Structured data tool after adding Schema properties. Test at Structured data tool after adding Schema properties to all the contents listed in ekshiksha. Test at Structured data tool after adding Schema properties to all the Interactive contents listed in ekshiksha. Data Highlighter of CDEEP Data Highlighter of courses at CDEEP Structured data found in CDEEP Structured data found in Spoken Tutorial site Structured data before and after data 'schema.org' properties in ekshiksha Search queries graph for CDEEP site search queries made for Spoken-tutorial site search queries made for ekshiksha site Change in average position of queries posted of CDEEP site on search engine 'Spoken-tutorial' website improvements after adding structured data. 'ekshiksha' website improvements after adding structured data

7 List of Tables Table No Table Name Page No. 1 2 LRMI Additions to schema.org/creativework Other important Characteristics of LRMI Alignment Object 20 4 Educational Audience 20 5 Mapping with CDEEP attributes with schema.org Mapping with Spoken Tutorials attributes with schema.org 7 Tags to be added (suggestion) 33 8 Tags to be added for workshop event (suggestion) 34 9 Mapping with ekshiksha attributes with schema.org Tags to be added (suggestion) Data Definition The tags related to education that are suggested to schema.org 13 The tags related to education that are suggested to schema.org 14 Advantages and Disadvantages of different parser in 77 Python 7

8 Abstract Today search engines are playing an important role in obtaining the information from the web. Every individual irrespective of age limit (Schoolchildren, College Students, Working professionals, teachers etc.) depends on the search engine to retrieve the information from the web. However, data present in most of the webpages does not have meaning attached to it, which causes the search engine to fail in providing relevant results about the data, what user are searching for, and this consumes a lot of time in getting the relevant results. Our focus is mainly on the retrieval of educational related information effectively when searched and index-able by search engines. A unique path to search for educational related information is not present, which is making the existing search inadequate for discovering all the educational resources. This is very valuable for school children to learn and teachers to use for educational purposes. Schema.org [6] which was initiated by Google, Bing, and Yahoo provides a collection of schemas that help the webmasters to mark-up the pages with different schemas for enhanced search. We will be using these mark-ups for educational content over the web. This will help school children, teachers, etc. to get better relevant results about what they are searching for, in less time. 8

9 1. Introduction 1.1 General Overview of the Problem Currently, Internet is widely used by majority of people in India to access information. Students use internet mainly for watching videos, social networking, playing games, and to complete their homework and projects (educational resources). Most of the students directly depend on search engines to find the right information over large data present on the web. As a result, a lot of time is wasted searching for the exact and appropriate information. Now-a-days search engines are playing a vital role in accessing the online content over the web. In India around 82% [3] of school children use internet and around 53% [3] of them use it to search for educational purposes- such as to do their assignments, gain in depth knowledge about the subjects, take online quizzes, and also to complete their projects. They depend on search engines, especially Google (around 53 % [3] approx.) by using keywords (around 46% [2] approx.) to search these resources. Though there are enough educational resources present on the web, 58% [2] of these school children face problems in accessing the appropriate information in time, and as a result, spend a lot of time on searching. The flow of this search is shown in the Figure 1. All these problems arise as the data present on the web is in unstructured form, i.e., it has no meaning attached to it. Also, search engines cannot understand the meaning of the content that is made available as a search result. These problems clearly specify a need for the retrieval of educational information effectively, when search is performed using search engines. When a search query is made for any personality like Albert Einstein, any organization like IIT Bombay, or a recipe like Potato Salad, we get a summarized information about it on the right hand side of search result page, which is called as Knowledge Panel'. This happens because of structured data markup; when applied as markups to webpages through 'schema.org'. Schema.org categorizes the data, and provides meaning to this data. It is classified into various types like 'Creativework', 'Event', 'Organization', 'Recipe', etc. However, there is no such unique way which makes existing search adequate to discover the required educational resources that are essential for the children to learn, and teachers to teach. 9

10 1.2 Objective: Our work mainly aims to propose some schema properties for education that will help search engines to understand educational resources on web, and will make these resources 'index-able'. As a result, the search time will be reduced which will not only assist teachers to find the appropriate information to teach students, but also students to learn these resources. In addition, it will help publishers to publish their information effectively. Figure 1: The actual flow of web search for information, statistics is based on survey "Use of Internet by the Researchers of Indian Veterinary Research Institute [13]" and "Internet Usage by the Students of Kendriya Vidyalaya[3]" 10

11 2. Literature Survey Here, we discuss the technologies used by various educational websites, and also the properties proposed by schema.org [6] and LRMI [11] to markup the educational websites. 2.1 Schema.org 'Schema.org' provides a set of properties i.e. markups, which can be utilized by the webmasters to markup their webpages, and to make these markups visible in major search engines like Google, Yandex, Yahoo! and Bing. These schema markups help search engine understand the meaning of the website, and filter search results. It helps in finding the right webpage in less time. Search engines have a very limited understanding of what is being discussed on web pages merely by reading, as opposed to the humans. By adding new tags i.e. 'itemscope', 'itemtype', and 'itemprop' in webpages, we can associate the web pages with specific property such as movie, place, person, video, etc. to make search engines understand those webpages clearly. Details of the properties are given below. itemscope: By adding itemscope, we are specifying that the HTML contained in the <div>...</div> block is about a particular item. <div> <h1>avatar</h1> <span>director: James Cameron (born August 16, 1954)</span> <span>science fiction</span> <a href="../movies/avatar-theatrical-trailer.html">trailer</a> </div> After adding Itemscpoe property, the code will be <div itemscope> <h1>avatar</h1> <span>director: James Cameron (born August 16, 1954) </span> <span>science fiction</span> <a href="../movies/avatar-theatrical-trailer.html">trailer</a> </div> 11

12 Itemtype: itemtype attribute is defined immediately after the itemscope. This specifies that the item contained in the div is in fact a Movie/person, recipe..., as defined in the schema.org type hierarchy. Item types are provided as URLs. <div itemscope itemtype=" <h1>avatar</h1> <span>director: James Cameron (born August 16, 1954)</span> <span>science fiction</span> <a href="../movies/avatar-theatrical-trailer.html">trailer</a> </div> Itemprop: To label properties of an item, itemprop attribute is used. <div itemscope itemtype =" <h1 itemprop="name">avatar</h1> <span >Director: <span itemprop="director">james Cameron</span> (born August 16, 1954)</span> <span itemprop="genre">science fiction</span> <a href="../movies/avatar-theatrical-trailer.html" itemprop="trailer">trailer</a> </div> Examples: After adding some schema properties ( like person, organisation etc. ) in the websites, when searched in google, the description about that comes in right hand side of the screen which is called knowledge panel. If we search for a person like Amitabh bachchan in google, the search result will be displayed as shown in Figure 2. 12

13 Figure 2: Display of Amitabh Bachchan description on right side panel If we search for an organisation like IIT Bombay, the search result will be displayed as shown in Figure 3 13

14 Figure 3: Display of IIT Bombay description on right side panel 2.2 Learning Resource Metadata Initiative (LRMI) The Learning Resource Metadata Initiative[11] is an organization that has been working on filtering search results, particularly when a search about educational resources is performed on the web. The purpose is to publish, discover, and deliver quality content easily on the web. The metadata schema developed by the LRMI was adopted by 'schema.org' in LRMI tries to reduce the gap between users and available resources by changing the way of accessing the information. The Figure 4 shows how meta-tagging is deployed in order to facilitate the search and retrieval of instructional resources. Here from right to left the publishers tag the learning resources and 14

15 save them in the learning registry and from left to right the educators and students search for the educational resources and find what they are looking for exactly in the learning registry. Figure 4: The flow at LRMI LRMI works with metadata, i.e., data about the data. It describes information like publication date, author, type of content, and number of pages etc. Each tag proposed by LRMI when added to the webpage, helps in the discovery of the educational re-sources in a better way. When we search for structure of atom after adding LRMI tags, we get categorized results which are shown in Figure 5. On the other hand, Figure 6 shows the results of Google search when no tags were added in the website. In Figure 5, there is a tag option named typicalagerange to filter results according to the age group. So, the user can find right information at right time with less effort. 15

16 Figure 5: The search results after adding LRMI and 'schema.org' markup. Figure 6: The search results without adding 'schema.org' markup. LRMI has proposed 14 properties which mainly markup educational resources. Now these properties are added into the library of 'Schema.org'/Creative work [6]. LRMI allows extension 16

17 since it wants to build the gap between the available resources and users. The survey results conducted by LRMI about the improvement of search after adding a markup shows an increment in retrieval of successful search results by 10% (35.5% %=10%) [11] and decrement of often unsuccessful results by 10% (13.1%-11.2%=10%) [11] when compared with previous year The schema properties that are added from LRMI in schema.org are shown in Table 1,2,3,4. 17

18 18

19 19

20 20

21 2.3 Kritikos This is a customized search engine developed by University of Liverpool, USA [15]. They used a Learning registry approach to capture, connect, and share data about learning resources available online. It aims to reduce the time required to find useful resources for educators and students. The Learning Registry makes the process easier by acting as an aggregator of metadata data about the learning resources available online. They use this technique for higher education and display the results visually as thumbnails. 2.4 SCHOOLS INTEROPERABILITY FRAMEWORK (SIF) ASSOCIATION The SIF [3] Association is comprised of K-12 organizations that have created a set of principles and definitions that enable software programs from different companies to share information and makes it possible for programs within a school or district to share data without additional programming. 21

22 3. Current Scenario In this section, we discuss about the online search requirements of school children, types of problems school children are facing while searching on the web, and if these problems are addressed by search engines or not. We also discuss the new trends in latest educational websites. This work has been done for our research purpose. Problem faced by children while using Internet: Today, most of the school children depend on search engines to do their homework, assignments, and search reference books. Many teachers also suggest their students to search on the web to get more information for better understanding of the topic. When a student types Structure of atom for class 9 in any search engines like Google, Yandex, Bing, or Yahoo!, students get hundreds of results. However, most of these results are irrelevant from students standard and level. Students cannot identify the correct link which gives appropriate information according to their level. The reason behind getting this huge list of results is the use of keyword matching technology used by the search engines to retrieve the results. School children, however, need a friendly search results for educational related information, which can filter the redundant results according to the level and categorize the results according to the content. This will trim down the amount of time spent on identifying the correct content by the children, teachers, and according their requirement. The below data shows the use of internet by children for education and problems faced during finding the information from the web. The results of the survey conducted by Kendriya Vidyalaya (KV) [2], Mysore, about the internet use by school children (among 120 students) say that around 82% children use internet. Analyzing this data, we need to provide a standard way to access the online resources effectively. Students use the internet for many reasons however; we mainly concentrate about how they use it for educational resources. 'Purpose of internet use' of school children is shown in Figure 7. The study shows that around 36% (25% for assignments +11% for educational information) use internet for educational purpose. Comparatively, the statistics of another country (Thailand) [3] states that 60% of total school children use internet for educational purpose. 22

23 Figure 7: Purpose of internet use by school children at KV, Mysore [3]. Figure 8: Problems faced by children while using Internet [2]. School children also face many problems while searching for the information from the web. According to the survey conducted, around 58% [4] find problem in accessing the relevant information from large, overloaded information present on the web as shown in Figure 8. The above analysis strongly highlights the need to provide a solution that will deal with the biggest problem i.e. getting relevant information. Clearly, there is huge scope for providing better 23

24 search facilities for Indian school children in order to increase the percentage of internet use for education. School children search in the web in different ways, according to a survey [5] conducted (among 316 students) 36% of them use search engines, 34% use subscription database and remaining type the web address directly for search. Search can be performed in various ways according to [4], results as shown in Figure 9. Among all different ways of searching, search with keywords is the most common way. Figure 9: The various ways search is performed over web. With the above data, we conclude that if schema.org markups are used in the websites, the search engines can recognize them and provide relevant results. Currently, we have many websites and technologies that have categorized information according to the content for children, related to education information and work in different ways given below: Creation of Custom Search engine. In these websites, search engine is built manually by adding various sites related to the education facilitating a 'custom search'. For example, website Kidrex [15] that is built by custom search. 24

25 Search in the fixed static database. These websites have their own database for educational resources, and perform search in that database. The website Mid-Continent library [12] searches for the educational information from its own database. Query Refinement: Today, many websites, modify the query and add keywords to the query and display the results accordingly to the child s needs. However, the websites also provide irrelevant results as they do not filter the results completely. 'Gogooglians' [13] is a website that works on this technology. This is the latest way to filter the results from the search engine. Search according to Age group: This is a technique which is very flexible used to search educational information according to diverse age groups. Website was built using custom search. 'Kids.gov' [14] is a website that was powered by 'Bing!' Search engine. This technique requires the user to know the specific websites. Thus, we need to provide a solution in which searching in common search engines be done in such a way that only relevant data is displayed. Figure 10 shows the flow diagram of retrieval the results by search engines when 'schema.org' markups implemented to educational websites. 25

26 Figure 10: The flow of Educational search when we add 'schema.org'. 26

27 4. Our approach After going through the literature survey and current scenario, the approach towards the project is as follows: -Analyze the websites and recognize all the properties that are relevant to education. -Create Software which performs the following steps. a) Takes the websites url as input to the software and generates a DOM structure corresponding to each and every webpage. b) Then it crawls through the DOM structure and find the relevant link where education related material is present and creates the in memory structure. c) In the in-memory structure it adds all the schema properties which map to the current properties at schema.org. d) After adding all the properties at schema.org it writes the modified html in a file. -A Hierarchy of the tags for the Indian educational system to be created and mapped with schema.org current properties. Then the properties which are not present in the schema.org are proposed as an extension to Schema.org -A Research paper on A study to assess and enhance educational specific search on web for school children. is written. 4.1 Software Requirement Specification (SRS) Software Platform Operating System: Ubuntu Language: Python (Version: 2.7.3) Library Used: Beautiful Soup (Version: 4.3.2) Parser Used: Python's Html Parser Header Used: bs4, urllib2, re, error, os Testing tool: Google Structured Data testing tool, Google Webmaster Tool Hardware Platform 27

28 Computer RAM: 4GB Processor: Core i5 I/O Devices: Keyboard, Mouse, Monitor 4.2 Project Scheduling The scheduling is shown in Figure 11: Y A B C A C T I V I T Y D E F G H X 1stJan 10th Jan 20th Jan 30th Jan 10th Feb 28th Feb 15th Mar 30th Mar 10th Apr 30th Apr 15th May 30th May 10th Jun 30th Jun TIME Figure 11: Gantt Chart LEGEND: A) Feasibility Study B) Literature Survey C) Design Strategy of Solution D) Implementation E) Adding schema properties to live site F) Testing at Structured Data Testing Tool G) Testing at Webmaster Tool H) Documentation 28

29 4.3 Analysis of websites We have analysed three educational websites [9] owned by IIT Bombay and, identified the tags and mapped them with schema.org. Analyses of the three websites are given below CDEEP (Center for Distance Engineering Education Programme) [16] It is an IIT Bombay s initiative which offers recorded courses to the outside students, working professional, researches etc. to avail IIT Bombay's courses. One can register online for this. After registration, the registered recorded course(s) are available by video streaming. Attributes of courses at CDEEP website mapping with properties of schema.org: Here in Table 5, we have mapped all the attributes present in CDEEP Website with the Properties of Schema.org Table 5: Mapping with CDEEP attributes with schema.org S.No Attributes/Propert 1) 2) 3) y of CDEEP Course Name Course Code Instructor 4) Department 5) 6) 7) 8) 9) 10) 11) 12) 13) Total Credits Number of Lectures Tutorial Practical Pre-requisite Description Last Updated Date Created Date Published 14) Duration/Semester Property Matching in schema.org Expected Type Text Person ->author/creator Organisatio ->department n ->description Text ->datemodified Date ->datecreated Date Date ->datepublished ->duration ISO 8601 date format 15) 16) 17) 18) Level Organisation Type Ratings ->publisher Organisation ->best Number or Text 29

30 19) url 20) related link Rating(Thing>Intangible>Rating) ->url ->relatedlink URL URL Tags need to be added in CDEEP courses (suggestions): These are few tags which are necessary to be added to the courses at the website so that it becomes more informative and more useful. a) Date Created: - It will give the date of creation. b) Date Published: - It will give the date of publishing. c) Level (Under Graduate/Post Graduate):- It is required to tell that the course is for UG or PG level which makes the course more informative. d) Ratings: - It helps both student and teacher to know the performance of the Course. Properties not found at schema.org: a) Total Credits b) Pre-requisite c) Level d) Type e) Course Code f) Number of Lectures g) Tutorial h) Practical Spoken Tutorial [8] The Spoken Tutorial website is the initiative of the Talk to a Teacher activity of the National Mission on Education through Information and Communication Technology (ICT), launched by the Ministry of Human Resources and Development, Government of India. 30

31 Attributes of Spoken Tutorial website matching with schema.org: Here in Table 6, we have mapped all the attributes present in Spoken Tutorial Website with the Properties of Schema.org. Tags need to be added in Spoken Tutorial Table 6: Mapping with Spoken Tutorial attributes with schema.org Property Matching in Expected Sl. No Attributes/Property schema.org Type SPOKEN TUTORIAL 1 Course Name Text 2 3 Language Tutorial Name Creative work->in language Text Text 4 No of visits e>rating>aggregate Rating Number ->Review Count WORKSHOP EVENTS Start Time Location 3 Topic Name WEB PAGE 1. Address 2. Main content of 3. Description Telephone Date Time >door time Person- Contac Point >address or Place Text Postal zation -> Address Web Page ->Creative Work>Web Page Element >description zation Text Integer zation 31

32 Suggestions These are few tags in Table 7 which are necessary to be added to the Tutorials at the "Spoken Tutorial website so that it becomes more informative and more useful. Table 7: Tags to be added (suggestion) SNO Attributes/Property Description Expected Type 1 Uploaded date Date when this media object was uploaded to this site. Date 2 Rating The rating for the content. Number or Text 3 Person Name Name of the person Text 4 Type 5 Level The type tags provides the suitable location of spoken tutorials in tutorial hierarchy We can search the videos based on level Tags need to be added to Workshop Event: Table 8 shows the tags which are necessary to be added to the Workshop Events at "Spoken Tutorial website so that it becomes more informative and more useful. Table 8: Tags to be added for workshop event (suggestion) S. No Attributes/Propert Description Expected Type 1 yrating The rating for the content. Number or Text 2 Academic code centre We can search the Workshop events Based on Academic centre code 32

33 3 Typical age limit We can search the Workshop events Text Based on Age limit 4 Performer Name Name of the Performing the event. Text Properties not found at schema.org: The following are the tags that have to be included in Spoken Tutorials and also not found at Schema.org website. 1. Level 2. Type 3. Academic Centre Code ekshiksha The ekshiksha[1] website is mainly developed for the school level students to learn and prepare them. Any school/institution can register with ekshiksha[1] and download content for distribution to their students through their own labs provided they agree to give valuable feedback. Mapping of attributes with schema.org property: Here in Table 9, we have mapped all the attributes present in ekshiksha website with the properties of Schema.org. Table 9: Mapping with ekshiksha attributes with schema.org No. Attributes Property Matching in schema.org Expected Type Contents Area(subject) Topic/Subtopic Title Author >name Text Text Text Person uthor 33

34 5. 6. Description Ratings >description Text - Number (Thing>Intangible>Rating>Rating Level Mode Thing-> Person Name a-first name b-middle name c-last name Value) Not in Schema.org Not in Schema.org Text ->additional Text name Text name Text additional name Qualification Text Organization/Depa rtment >worksfor (Organization ->Department) 14. Text -> Creative Work->Web Page Main Content Web Language ->maincontentofpage Element ->in Text Image Language Image Object ->primaryimageofpage Number review Count Page Rating>Aggregate Rating ->review Count 19. Contributor 34

35 Tags need to be added in ekshiksha website (suggestions): Table 10 suggests the tags which are necessary to be added to the ekshiksha[1] website so that it becomes more informative and more useful. Table 10: Tags to be added (suggestion) PROPERTIES EXPECTED SCHEMA Related Link TYPE URL Level NUMBER Not in schema.org Description Important links related to website age group Properties not found at schema.org: The following are the tags that have to be included in ekshiksha[9] and also not found at Schema.org website. Mode: It defines the mode of the content. Pre requisite: The pre knowledge and languages required to know for subjects. Level: Defines the level or standard of the content means it is useful for 9 th standard or 10th standard students. Format: It specifies the Type of the content i.e. content either in Text format of Flash/Applet Format. 35

36 5. Software Development 5.1 Design Strategy for the solution: First of all attributes found in the website were mapped with current schema.org properties. The attributes which were mapped with current schema.org were to be inserted in the websites. For this purpose HTML Processing Software was made. Figure 12: Overall System Design or architecture of the HTML Processing Software Figure 12 explains the principle on which HTML Processing Software is made. The software takes homepage url of a site or specifically the html file where schema.org properties to be inserted. Then it uses a Python s default parser which parses the page and 36

37 gives the standard errors and warnings. It also creates the DOM structure of the html page. From the DOM structure, intelligence(contents on which schema.org properties to be added) is extracted which also gives the errors and warnings. After extracting the intelligence, in-memory structure is created and extracted contents are stored in that inmemory structure. On this in-memory structure, processing is done i.e. schema.org properties are added to the contents. After processing, this is written back onto inmemory structure and reverse process is followed. At the end, the modified HTML page i.e. page containing schema.org properties is returned as output which is stored in a file. Figure 13: Structure of HTML Processing Software Figure 13 gives the basic structure of the software. It takes html page or files as input, process it and adds schema properties to it, finally store the modified HTML in a file. 37

38 Approach used to add schema properties to tag courses at CDEEP is shown in Figure 14 Figure 14: Approach used in the HTML Processing Software 38

39 5.2 Context diagram (Dfd Level 0) of the software: Figure 15: Level 0 diagram of HTML Processing Software Table 11: Data Definition Id a b Data Definition home_url Url of home page of CDEEP Original course detail Original html content of course details stored in a file c html Modified d e detail html course details stored in a file Original course html Original html content of course list stored in a file Modified course html Modified html content(with added schema properties to course Modified html content(with added schema properties) of instructor attribute) of course list stored in a file 39

40 6. Test Plan The added schema.org properties to the above 3 educational websites mentioned can tested on two tools of Google. The tools are as follows: 1. Google Structured Data Testing Tool [14]: In this tool the modified html file content are pasted and it recognizes the schema.org properties and display the list of all the schema.org properties in the webpage. URL of the website can also be passed to this tool and it gives the list of all the schema.org properties attached to the passed url. 2. Google Webmaster Tool: This is a very important tool which gives all the required things for a website testing and performance. For using this tool one has to upload Google verification file in the root directory of the website so that Google bot can recognize that this website has to be tested. It is a continuous testing going for the site. This testing is done for live site. Two other file also has to be uploaded to the root directory i.e. robots.txt which mentions the portion of the site that has to be protected from the bot to access and also specifies which bot can access the site. It also contains the link of the sitemap.xml. The other file to be uploaded to the root directory is sitemap.xml which has the preferences of the different levels of the site. It gives the performance of the website in following terms: Structured Data (It gives the schema.org properties added to the registered site) Data Highlighter (It gives the description of schema.org tagged properties on the site will appear on right hand side of the search engine) HTML improvements (It gives HTML improvements on the site.) Search Queries( It gives the average position percentage change of the queries made by the user on the search engine which is calculated on the basis of CTR i.e. click through rate which is calculated by number of clicks by number of impressions for respective query made queries made. For better understandability, a graph is also plotted for clicks and impression progress. ) Description of terms used at Webmaster Tool: Impressions, the number of times pages of the site appeared in search results. 40

41 Clicks, the number of times a user clicked the sites listed in search results for a particular query. CTR (Click Through Rate), the percentage of impressions that resulted on a clicking the site. Average position, the average top position of the site on the search results page for that query. 41

42 7. Implementation Details Language: Python(Version:2.7.3) Library Used:Beautiful Soup(Version:4.3.2) Parser Used:Python's Html Parser Header Used: bs4, urllib2, re, error, os Varibles Used: home_url,keyword,list_jsp_url,course_list_url,schema_link Software Introduction: This software reads the data at the CDEEP site and adds schema.org properties so that course description comes at knowledge panel when anyone searches for it on any search engine. Input to the Software: url of CDEEP site ( Output from the software: Original course details html file Modified html file Description of Functions: 1. process_course_detail(fname,bsoup):- This function processes the input html data and calls necessary functions to add schema properties on the html fname(it is the course code which is used as filename to save the file with.htm extension) bsoup(beautiful soup object of the page containing html data which will be be used for the processing of no return as processing is done and the object(output) is stored in the file inside the function itself 2. process_dept(fname,bsoup): -This function adds schema properties to the author attribute of each course listed in the department and calls necessary function for adding the fname(it is the course code which is used as filename to save the file with.htm extension) bsoup(beautiful soup object of the page containing html data which will be 42

43 be used for the processing of return as processing is done and the object(output) is stored in the file inside the function itself 3. itemscope(tsoup,schema_link): -This function adds schema properties i,e. itemscope and itemtype= to the body tag so that schema.org scope is tsoup(beautiful soup object of the page containing html data which will be be used for the processing of data) schema_link(schema.org link which contains the properties to be added to the page ex: return as it only modifies the soup object and changes are saved to the soup object only 4. itemprop_cname(psoup,cname,itprop):- It finds the given course name in the given html data and its corresponding tag and add 'name' property to it (ex. For course name attribute psoup(beautiful soup object of the page containing html data which will be be used for the processing of data) cname(course name on which schema property to be added) itprop(schema item property to find return as it only modifies the soup object and changes are saved to the soup object only 5. itemprop(psoup,prop,itprop): -It finds the given keyword/attribute and add respective schema property to psoup(beautiful soup object of the page containing html data which will be be used for the processing of data) prop(keyword/attribute on which schema property to be added) itprop(schema item property to find return as it only modifies the soup object and changes are saved to the soup object 43

44 only 6. b_soup_obj(data):- This function is used to create beautiful soup object of the html data passed to it and return the data created from given url by url_data() the beautiful soup object 7. url_data(url):- Function to open the passed url as parameter. It gets the content of the url and returns the content. It first sends request to the url by using urllib2.request(url function) and stores the reply. Then it passes the reply to urlopen() function to open the given url. At the end it reads the content of the url and store in an object and returns that url(url link for which html data to be fetched return the html content of the passed url 8. find_link(obj,key):- This function finds the required link for courses in the home page of CDEEP by matching the keyword passed to it. The course detail link can be accessed from this keyword. It searches in the html content by matching the regular expression as the keyword and finds the corresponding link i.e., href returns the obj(beautiful soup object on which search is to be made) key(keyword by which course link can be opened on the home page of returns the correponding url to the key word. 9. find_dept(obj):- This function finds all departments and displays them. It also opens all the department by passing the department link to dept() obj(beautiful soup object on which search is to be returns nothing. 10. dept(next_link):- It opens the department link and finds the listed courses in each department and search for the courses listed under that department. It opens each course detail link by combining necessary arguments to the jsp. Then it passes the obtained course id and 44

45 course detail object to process_course_detail() next_link(link of each department returns nothing 11. check_error(e_obj): It detects error in the html content of the link. It checks for the basic errors in the html page like error for html/title tags etc. If error is found then it prints the e_obj(beautiful object of the html page for which error to be returns nothing rather only print the errors. Steps carried in the Software: i. Start ii. Input cdeep home page url. iii. Pass the url to url_data() to fetch the html data of home page of cdeep and store the html data in an object. iv. Call b_soup_obj() to create the beautiful soup object of the html data and store the returned bsoup object. v. Call find_link() by passing the bsoup object and keyword to find the course url on the home page. vi. For the course url found call url_data() and b_soup_obj() to create course url content object. vii. Pass the course url object to find_dept() function to find departments at cdeep. viii. At find_dept() it searches for all the departments and print it and combine department name to the jsp to fetch the url of the respective department. ix. It then pass the link for each department to dept() function to further find the courses listed at each department and to open its respective course details link. x. Pass department name corresponding object to process_dept() function for adding schema property to the instructor of each course. xi. Pass course id and course detail object to process_course_deatil() for processing to be done on course detail page. xii. At process_course_deatil() oringinal html page for the respective course is saved to a file 45

46 named at its id and processing is done by passing bsoup object of course detail page to functons itemscope(), itemprop() etc. and also supplying other necessary elements such as tags on which schema properties to be added. xiii. The modified course details object is stored in file named with its id in modified directory. xiv.stop Steps carried in Keyword Insertion Software: i. START ii. First of all Standard Keyword Database for each course under CDEEP is made with the help of internet. iii. Description of a course is extracted. iv. Set of words separated by ;,. : are clubbed together v. Then keyword from standard database are mapped with the set off words and the keywords are matched then that whole set of words is treated as keyword for that respective course. vi. The extracted sets of words are inserted to webpage where course link is present so that the course can be searchable by different keywords. vii. STOP 46

47 8. Results and Discussion 8.1 Results at Structured Data Testing Tool [14] Test result of CDEEP [16] Figure 16: Test result at Structured data tool before adding Schema properties 47

48 Figure 17: Test result at Structured data tool after adding Schema properties 48

49 Figure 18: Test result at Structured data tool after adding Schema properties to all the departments listed in CDEEP 49

50 8.1.2 Result at Spoken Tutorial Figure 19: Test result at Structured data tool after adding Schema properties to all the tutorials list in Spoken-tutorial. 50

51 Figure 20: Test result at Structured data tool after adding Schema properties to all the workshop contents listed in Spoken-tutorial. 51

52 8.1.3 Results at ekshiksha Figure 21: Test result at Structured data tool after adding Schema properties. 52

53 Figure 22: Test result at Structured data tool after adding Schema properties to all the contents listed in ekshiksha. 53

54 Figure 23: Test result at Structured data tool after adding Schema properties to all the Interactive contents listed in ekshiksha. 54

55 8.2 Results at Webmaster Tool: The average position of the website gets improved by two ways, one by the increase in numbers of people who searched about the website and other by adding structured data to the website. The snapshots of search queries from webmaster tool give the information about the improvements in the website. Figure 24: Data Highlighter of CDEEP 55

56 Figure 25: Data Highlighter of courses at CDEEP Figure 24 and 25 gives the view that marked content will be displayed on knowledge panel of the search engine. 56

57 Figure 26: Structured data found in CDEEP 57

58 Figure 27: Structured data found in Spoken Tutorial site 58

59 Figure 28 Structured data before and after data 'schema.org' properties in ekshiksha Figure 26, 27 and 28 shows the structured data i.e. schema.org properties attached to the webpage of sites. It shows significant increase in structured data items in the graph with time. 59

60 Figure 29: Search queries graph for CDEEP site 60

61 Figure 30: search queries made for Spoken-tutorial site 61

62 Figure 31: search queries made for ekshiksha site Figure 29, 30 and 31 gives the analysis of search query graph for the CDEEP site. The graph shows the clicks in red and impressions in blue. There is remarkable increase in the clicks of three sites as shown in the 3 graphs. 62

63 Figure 32: Change in average position of queries posted of CDEEP site on search engine Figure 33: 'Spoken-tutorial' website improvements after adding structured data. 63

64 Figure 34: 'ekshiksha' website improvements after adding structured data. Figure 32 gives the average position change measured for queries made for contents present at CDEEP site in the search engine. In Figure 33 and 34, we can see highlighted columns CTR and average position. Also we can see the respective changes in the very next columns. These columns provide strong evidence that the structured data will help in discovering educational websites. It is observed that by adding the structured data the average position of the website and Click through Rate (CTR) is increased. CTR measures the effectiveness of the webpage. It is calculated based on the clicks and Impression. A high CTR means that it attracts and gets clicks often. A low CTR means it is under performing and should be evaluated further. A 1% CTR means for every 100 impressions there is 1 click. 8.3 Proposed Tags Our next goal is to filter the irrelevant links and provide appropriate information during the retrieval of search results from any search engine. As part of this, we have analyzed the complete Indian educational system hierarchy. We have divided the complete educational system 64

65 into two levels school and University. Further, the school level education is categorized mainly into three ways: access, assimilation and assessment. Access is the way of retrieving resources either by online or by offline and also it deals with the type of resources used for learning like textbooks, printed notes, video lectures and audio lectures etc. Assimilation relates to the mode of learning and can be done by hands on experiment, by computer training or by tutoring. Assessment is a way to test the level of understanding of the subject by the students it can be done either online or offline. It is done according to various levels, age group etc. Table 12 and 13 shows proposed tags related to education. Table 12. The tags related to education that are suggested to schema.org Properties proposed are added under schema.org/school. This has more specific properties as schema.org/student, schema.org/faculty and schema.org/teachingmethodologies. School > TeachningMethodolgies This hierarchy is created in the the current changes in the Teaching Methodologies in view, this can be extended further. Presently it has three more specific properties Access, Assimilation, Assessment. School > TeachingMethodologies > Access Property Expected Type Description modeofacces schema.org/te The mode of accessing the resources. s xt Ex: "online" or Ex: "offline" School > TeachingMethodologies > Access > Traditional The properties in this are 'Book', 'VideoLectures' and 'AudioLectures'. These can be imported from schema.org/book, schema.org/videoobject and schema.org/audioobject. School > TeachingMethodologies > Access > NonTraditional The more specific type under this hierarchy can be schema.org/discussionfourm and schema.org/correspondence. The properties in this hierarchy are discussed below. peertopeer schema.org Here there will be a teacher and a student, /Text teacher will be guiding the student. 65