HOW DOES A SEARCH ENGINE WORK? Hector says... Hi there! Did you know that the World Wide Web is made up of over a trillion web pages? That s more information than you d find in a really big library and it s growing every day. With so much information online, you will need a helper to find what you re looking for on the web. I ll explain more right here! What can I use to find information on the web? On the internet there are two ways of finding information. You can get help from a web directory or from a search engine. Web directories need human input to work. This means they only capture a tiny amount of the information available on the internet. Search engines, however, use software programs to find and study millions and millions of web pages really quickly.
A crawler based search engine will send out armies of spiders to find and study web pages looking for specific words. Once the spider has collected information from a page, it finds links to other pages or websites and explores those too. Each spider sends the information it collects back to a central place called an index. The index holds the information so that it is easy to find. Spiders even visit web pages again to check if any words have changed. If there are changes, the spider will update the index. When Hector uses the key words Silicon Deep in a search engine, the search engine quickly looks through the hundreds of millions of words in the index and sends back a list of pages that have those exact words!
How does a search engine work? Good question! It is important to know how to use a search engine well to get the best results when you search online! Each search engine web page has a special box to type in the key words you are searching for. When you type in key words, the search engine will find those words and give you a list of pages and websites that use them. It sounds simple but there is a lot going on behind the scenes! Many search engines use software programs written by engineers (called crawlers or spiders) to look through web pages all of the time to find information.
How do search engines put these results in order? There are lots of different search engines and each one uses a set of rules to order the results for you. This set of rules is called an algorithm. Different search engines can give you different results depending on the rules or algorithm each uses. You need to look carefully at the web pages a search engine finds. This will help you decide which ones are most useful to you. A teacher or parent can help you learn that skill. If I create a web page, how can I make sure people see it? If you create a web page, you may want to make sure lots of people see it! There are some things web page writers can do to make sure the spiders will see all the information on it. Web page writers can: include key words on the page use text for important information, not pictures. This is because spiders cannot tell what is in a picture include text that clearly describes what is on the website include a link to each page of the website, so the spider can find it
What if I don t want a search engine to index my webpages? A web page writer can use Robots Exclusion Protocol or REP to keep a web page from appearing in the index. When a spider arrives at a website it checks for a robots.txt file on the site. This file tells the spider whether or not it has permission to crawl the site. Hector s Tips for Using a Search Engine Here are some tips that apply to most search engines: If you think carefully about the key words you use in a search engine you will usually find what you are looking for easily. It is always good to talk to parents or teachers about key words you will be using to search the web. Remember to check the spelling of your key words. You can also use special words or symbols that help a search engine know exactly what you are looking for. These words and symbols are called Boolean operators. Here are some of them using the example of looking for the Silicon Deep Tech Cave.
Boolean operator How it looks What it does What you could find tech and cave Tells the search Pages that have both the and tech + cave engine to find pages with BOTH these word tech and the word cave on them. The words + put the word and or a plus sign (+) between the key words on it might not be together on the page or in the same sentence words tech cave Tells the search Pages that have the words put the quote marks around the key words engine to find pages with BOTH these words on it right and tech and cave right next to each other. next to each other tech or cave Tells the search Pages that have the word or put the word or between the key words engine to find pages with the word tech on them OR pages with the word cave tech on them and lots of other pages that have the word cave on them. The words may not be together on them on any pages not Tech not cave Tells the search Pages that have the word - Tech - cave engine to find pages with the word tech tech on them but do not have the word cave put the word not or a on them, but not to minus sign (-) in front of the key word you want to leave out show any pages if they also have the word cave on them
Hector says Find out more about how a search engine works at your local or school library. Here s a link to a Common Craft video where you can learn more about How a search engine works http://www.commoncraft.com/search You can also watch this video which explains how one type of search engine works: http://www.google.com/howgoogleworks/ Other Tech Cave modules have information on safe searching. Check them out! Also keep checking back at the Silicon Deep Tech Cave, as there s new information being added all the time! www.hectorsworld.com
Hector s World Tech Terms search engine key words crawler based search engine A web page that will help you find information on the world wide web The words that describe what you are looking for on the world wide web A search engine that sends out armies of crawlers or spiders to study web pages spider algorithm Boolean operator A software program that looks at specific words on web pages A set of rules Words or symbols that help a search engine know exactly what you are looking for index Robots Exclusion Protocol or REP A central place that holds information about web pages A way you can tell a search engine not to crawl your website