Performance Analysis for Crawling
|
|
|
- Oswald Watkins
- 5 years ago
- Views:
Transcription
1 Scalable Servers and Load Balancing Kai Shen Online Applications online applications Applications accessible to online users through. Examples Online keyword search engine: Google. Web Gmail. News: CNN, NBC news. Web directory: Yahoo!, MSN. Scalability requirements Many simultaneous user accesses; large amount of hosted data, servers Computer systems that host these online applications 11/11/2009 CSC 257/457 - Fall /11/2009 CSC 257/457 - Fall Servers are at the Application Layer Normally on the end hosts, involving no routers Function on transport-layer protocols TCP/UDP Step 1 Crawling Crawling get all these Web pages out there: First retrieve some root pages; Parse their content and follow hyperlinks to retrieve more pages; Depth-first search or breadth-first search? Remove duplicates. Google Yahoo! CNN 11/11/2009 CSC 257/457 - Fall /11/2009 CSC 257/457 - Fall CSC 257/457 - Fall
2 Performance Analysis for Crawling What are the resources involved? CPU processing for TCP/HTTP protocol handling and the parsing of page content writing to disk storage network bandwidth to remote web sites Assume average page size 10KB raw processing power of a single CPU 1000 requests/sec I/O to a single disk 100 seeks/sec up to 100 requests/sec network bandwidth from/to the T1 link (1.5Mbit/s) 12 requests/sec T3 link (45Mbit/s) 360 requests/sec Step 2 Indexing Indexing crawled raw web pages are not easy to search. we index them to formats that are easy to search. As part of indexing, we need to give each page an ID using a hash function. Computer: Page #123 Page #357 Networks: Page #124 Page #468 11/11/2009 CSC 257/457 - Fall /11/2009 CSC 257/457 - Fall Step 3 Online Search Index server Partitioning and Replication Index servers (partition 1) Firewall Web server/ Query handler Local-areaarea network Local-areaarea network Index servers (partition 2) Page server Web server/ Query handlers Scalability, reliability 11/11/2009 CSC 257/457 - Fall Page servers 11/11/2009 CSC 257/457 - Fall CSC 257/457 - Fall
3 Load Balancing over Servers Popular sites like Google or CNN receive tens or hundreds of millions of hits per day. A large number of replicated servers are used at these sites. Key question: how to balance client requests over these servers? Load Balancing on Servers Technique 1 - DNS Rotation IP address of CNN.com? IP address of CNN.com? DNS server 11/11/2009 CSC 257/457 - Fall /11/2009 CSC 257/457 - Fall Discussions on DNS Rotation Advantages Require almost no change on the existing architecture Problems DNS Caching Rigid load balancing policy can t balance based on runtime load changes slow or no adjustment in response to failures Load Balancing on Servers Technique 2 Cooperative Offloading 11/11/2009 CSC 257/457 - Fall /11/2009 CSC 257/457 - Fall CSC 257/457 - Fall
4 Discussions on Cooperative Offloading Can be combined with the DNS rotation. Advantages: More flexible policy is possible Be more responsive to runtime workload and server failures (to a certain degree) Problems: Need software changes on servers Longer delay 11/11/2009 CSC 257/457 - Fall Cooperative Offloading with TCP Handoff [Pai et al. ASPLOS1998] What does 1.3 do? What does 1.4 do? All packets in a TCP connection must offload to one server? 11/11/2009 CSC 257/457 - Fall Cooperative Offloading vs. TCP Handoff Software changes on the servers Delays Load Balancing on Servers Technique 3 Load Balancing Firewall LB /11/2009 CSC 257/457 - Fall /11/2009 CSC 257/457 - Fall CSC 257/457 - Fall
5 More About Load Balancing Summary How deep do we look into the network protocol stack? Network layer (IP)? Transport layer (TCP/UDP)? Application layer? Load balancing policies in LB routers (Goal: transparency, plug-and-play) Simple rotation Least number of active requests Shortest response time Scalable servers partitioning replication Load balancing for servers DNS rotation cooperative offloading (w. TCP handoff) Load balancing router Changes required on the components: DNS server?? Web server?? client?? router?? 11/11/2009 CSC 257/457 - Fall /11/2009 CSC 257/457 - Fall CSC 257/457 - Fall
Files/News/Software Distribution on Demand. Replicated Internet Sites. Means of Content Distribution. Computer Networks 11/9/2009
Content Distribution Kai Shen Files/News/Software Distribution on Demand Content distribution: Popular web directory sites like Yahoo; Breaking news from CNN; Online software downloads from Linux kernel
Relevant?!? Algoritmi per IR. Goal of a Search Engine. Prof. Paolo Ferragina, Algoritmi per "Information Retrieval" Web Search
Algoritmi per IR Web Search Goal of a Search Engine Retrieve docs that are relevant for the user query Doc: file word or pdf, web page, email, blog, e-book,... Query: paradigm bag of words Relevant?!?
Large-Scale Web Applications
Large-Scale Web Applications Mendel Rosenblum Web Application Architecture Web Browser Web Server / Application server Storage System HTTP Internet CS142 Lecture Notes - Intro LAN 2 Large-Scale: Scale-Out
Shen, Tang, Yang, and Chu
Integrated Resource Management for Cluster-based Internet s About the Authors Kai Shen Hong Tang Tao Yang LingKun Chu Published on OSDI22 Presented by Chunling Hu Kai Shen: Assistant Professor of DCS at
CS November 2018
Bigtable Highly available distributed storage Distributed Systems 19. Bigtable Built with semi-structured data in mind URLs: content, metadata, links, anchors, page rank User data: preferences, account
CS November 2017
Bigtable Highly available distributed storage Distributed Systems 18. Bigtable Built with semi-structured data in mind URLs: content, metadata, links, anchors, page rank User data: preferences, account
An Overview of Search Engine. Hai-Yang Xu Dev Lead of Search Technology Center Microsoft Research Asia
An Overview of Search Engine Hai-Yang Xu Dev Lead of Search Technology Center Microsoft Research Asia haixu@microsoft.com July 24, 2007 1 Outline History of Search Engine Difference Between Software and
Data Centers. Tom Anderson
Data Centers Tom Anderson Transport Clarification RPC messages can be arbitrary size Ex: ok to send a tree or a hash table Can require more than one packet sent/received We assume messages can be dropped,
Venugopal Ramasubramanian Emin Gün Sirer SIGCOMM 04
The Design and Implementation of a Next Generation Name Service for the Internet Venugopal Ramasubramanian Emin Gün Sirer SIGCOMM 04 Presenter: Saurabh Kadekodi Agenda DNS overview Current DNS Problems
Introduction. Network Architecture Requirements of Data Centers in the Cloud Computing Era
Massimiliano Sbaraglia Network Engineer Introduction In the cloud computing era, distributed architecture is used to handle operations of mass data, such as the storage, mining, querying, and searching
CS 347 Parallel and Distributed Data Processing
CS 347 Parallel and Distributed Data Processing Spring 2016 Notes 12: Distributed Information Retrieval CS 347 Notes 12 2 CS 347 Notes 12 3 CS 347 Notes 12 4 Web Search Engine Crawling Indexing Computing
CS 347 Parallel and Distributed Data Processing
CS 347 Parallel and Distributed Data Processing Spring 2016 Notes 12: Distributed Information Retrieval CS 347 Notes 12 2 CS 347 Notes 12 3 CS 347 Notes 12 4 CS 347 Notes 12 5 Web Search Engine Crawling
Administrivia. Crawlers: Nutch. Course Overview. Issues. Crawling Issues. Groups Formed Architecture Documents under Review Group Meetings CSE 454
Administrivia Crawlers: Nutch Groups Formed Architecture Documents under Review Group Meetings CSE 454 4/14/2005 12:54 PM 1 4/14/2005 12:54 PM 2 Info Extraction Course Overview Ecommerce Standard Web Search
The Anatomy of a Large-Scale Hypertextual Web Search Engine
The Anatomy of a Large-Scale Hypertextual Web Search Engine Article by: Larry Page and Sergey Brin Computer Networks 30(1-7):107-117, 1998 1 1. Introduction The authors: Lawrence Page, Sergey Brin started
Engineering Goals. Scalability Availability. Transactional behavior Security EAI... CS530 S05
Engineering Goals Scalability Availability Transactional behavior Security EAI... Scalability How much performance can you get by adding hardware ($)? Performance perfect acceptable unacceptable Processors
Lecture 8: Internet and Online Services. CS 598: Advanced Internetworking Matthew Caesar March 3, 2011
Lecture 8: Internet and Online Services CS 598: Advanced Internetworking Matthew Caesar March 3, 2011 Demands of modern networked services Old approach: run applications on local PC Now: major innovation
Be Fast, Cheap and in Control with SwitchKV. Xiaozhou Li
Be Fast, Cheap and in Control with SwitchKV Xiaozhou Li Goal: fast and cost-efficient key-value store Store, retrieve, manage key-value objects Get(key)/Put(key,value)/Delete(key) Target: cluster-level
CSC2231: DNS with DHTs
CSC2231: DNS with DHTs http://www.cs.toronto.edu/~stefan/courses/csc2231/05au Stefan Saroiu Department of Computer Science University of Toronto Administrivia Next lecture: P2P churn Understanding Availability
Web Crawling. Introduction to Information Retrieval CS 150 Donald J. Patterson
Web Crawling Introduction to Information Retrieval CS 150 Donald J. Patterson Content adapted from Hinrich Schütze http://www.informationretrieval.org Robust Crawling A Robust Crawl Architecture DNS Doc.
Mul$media Networking. #9 CDN Solu$ons Semester Ganjil 2012 PTIIK Universitas Brawijaya
Mul$media Networking #9 CDN Solu$ons Semester Ganjil 2012 PTIIK Universitas Brawijaya Schedule of Class Mee$ng 1. Introduc$on 2. Applica$ons of MN 3. Requirements of MN 4. Coding and Compression 5. RTP
Distributed Systems Principles and Paradigms. Chapter 12: Distributed Web-Based Systems
Distributed Systems Principles and Paradigms Maarten van Steen VU Amsterdam, Dept. Computer Science steen@cs.vu.nl Chapter 12: Distributed -Based Systems Version: December 10, 2012 Distributed -Based Systems
THE WEB SEARCH ENGINE
International Journal of Computer Science Engineering and Information Technology Research (IJCSEITR) Vol.1, Issue 2 Dec 2011 54-60 TJPRC Pvt. Ltd., THE WEB SEARCH ENGINE Mr.G. HANUMANTHA RAO hanu.abc@gmail.com
CS November 2017
Distributed Systems 21. Delivery Networks () Paul Krzyzanowski Rutgers University Fall 2017 1 2 Motivation Serving web content from one location presents problems Scalability Reliability Performance Flash
Content Distribution. Today. l Challenges of content delivery l Content distribution networks l CDN through an example
Content Distribution Today l Challenges of content delivery l Content distribution networks l CDN through an example Trends and application need " Some clear trends Growing number of and faster networks
Remote Procedure Call. Tom Anderson
Remote Procedure Call Tom Anderson Why Are Distributed Systems Hard? Asynchrony Different nodes run at different speeds Messages can be unpredictably, arbitrarily delayed Failures (partial and ambiguous)
Distributed Systems. 21. Content Delivery Networks (CDN) Paul Krzyzanowski. Rutgers University. Fall 2018
Distributed Systems 21. Content Delivery Networks (CDN) Paul Krzyzanowski Rutgers University Fall 2018 1 2 Motivation Serving web content from one location presents problems Scalability Reliability Performance
CS November 2018
Distributed Systems 21. Delivery Networks (CDN) Paul Krzyzanowski Rutgers University Fall 2018 1 2 Motivation Serving web content from one location presents problems Scalability Reliability Performance
Crawling the Web. Web Crawling. Main Issues I. Type of crawl
Web Crawling Crawling the Web v Retrieve (for indexing, storage, ) Web pages by using the links found on a page to locate more pages. Must have some starting point 1 2 Type of crawl Web crawl versus crawl
Distributed Data Infrastructures, Fall 2017, Chapter 2. Jussi Kangasharju
Distributed Data Infrastructures, Fall 2017, Chapter 2 Jussi Kangasharju Chapter Outline Warehouse-scale computing overview Workloads and software infrastructure Failures and repairs Note: Term Warehouse-scale
DEPLOYMENT GUIDE Version 1.1. DNS Traffic Management using the BIG-IP Local Traffic Manager
DEPLOYMENT GUIDE Version 1.1 DNS Traffic Management using the BIG-IP Local Traffic Manager Table of Contents Table of Contents Introducing DNS server traffic management with the BIG-IP LTM Prerequisites
Application-layer Protocols
Application-layer Protocols Kai Shen application transport data link physical Network Applications and Application-Layer Protocols application transport data link physical application transport data link
Streaming Video over the Internet. Dr. Dapeng Wu University of Florida Department of Electrical and Computer Engineering
Streaming Video over the Internet Dr. Dapeng Wu University of Florida Department of Electrical and Computer Engineering What is Streaming Video? Download mode: no delay bound Streaming mode: delay bound
Yahoo Traffic Server -a Powerful Cloud Gatekeeper
Yahoo Traffic Server -a Powerful Cloud Gatekeeper Shih-Yong Wang Yahoo! Taiwan 2010 COSCUP Aug 15, 2010 What is Proxy Caching? Proxy Caching explicit client configuration transparent emulate responses
CSE 5306 Distributed Systems
CSE 5306 Distributed Systems Naming Jia Rao http://ranger.uta.edu/~jrao/ 1 Naming Names play a critical role in all computer systems To access resources, uniquely identify entities, or refer to locations
Part 12 殷亚凤. Homepage: Room 301, Building of Computer Science and Technology
Part 12 殷亚凤 Email: yafeng@nju.edu.cn Homepage: http://cs.nju.edu.cn/yafeng/ Room 301, Building of Computer Science and Technology Distributed Web-based systems The WWW is a huge client-server system with
What s Virtual Memory Management. Virtual Memory Management: TLB Prefetching & Page Walk. Memory Management Unit (MMU) Problems to be handled
Virtual Memory Management: TLB Prefetching & Page Walk Yuxin Bai, Yanwei Song CSC456 Seminar Nov 3, 2011 What s Virtual Memory Management Illusion of having a large amount of memory Protection from other
Crawling CE-324: Modern Information Retrieval Sharif University of Technology
Crawling CE-324: Modern Information Retrieval Sharif University of Technology M. Soleymani Fall 2017 Most slides have been adapted from: Profs. Manning, Nayak & Raghavan (CS-276, Stanford) Sec. 20.2 Basic
SJTU 2018 Fall Computer Networking. Wireless Communication
SJTU 2018 Fall Computer Networking 1 Wireless Communication Internet Protocol Stack 2 Application: supporting network applications - FTP, SMTP, HTTP Transport: data transfer between processes - TCP, UDP
Addressed Issue. P2P What are we looking at? What is Peer-to-Peer? What can databases do for P2P? What can databases do for P2P?
Peer-to-Peer Data Management - Part 1- Alex Coman acoman@cs.ualberta.ca Addressed Issue [1] Placement and retrieval of data [2] Server architectures for hybrid P2P [3] Improve search in pure P2P systems
Load Balancing Technology White Paper
Load Balancing Technology White Paper Keywords: Server, gateway, link, load balancing, SLB, LLB Abstract: This document describes the background, implementation, and operating mechanism of the load balancing
Be Fast, Cheap and in Control with SwitchKV Xiaozhou Li
Be Fast, Cheap and in Control with SwitchKV Xiaozhou Li Raghav Sethi Michael Kaminsky David G. Andersen Michael J. Freedman Goal: fast and cost-effective key-value store Target: cluster-level storage for
PNUTS: Yahoo! s Hosted Data Serving Platform. Reading Review by: Alex Degtiar (adegtiar) /30/2013
PNUTS: Yahoo! s Hosted Data Serving Platform Reading Review by: Alex Degtiar (adegtiar) 15-799 9/30/2013 What is PNUTS? Yahoo s NoSQL database Motivated by web applications Massively parallel Geographically
Data Representation and Indexing
Data Representation and Indexing Kai Shen Data Representation Data structure and storage format that represents, models, or approximates the original information What we ve seen/learned so far? Raw byte
DATA MINING - 1DL105, 1DL111
1 DATA MINING - 1DL105, 1DL111 Fall 2007 An introductory class in data mining http://user.it.uu.se/~udbl/dut-ht2007/ alt. http://www.it.uu.se/edu/course/homepage/infoutv/ht07 Kjell Orsborn Uppsala Database
Camdoop Exploiting In-network Aggregation for Big Data Applications Paolo Costa
Camdoop Exploiting In-network Aggregation for Big Data Applications costa@imperial.ac.uk joint work with Austin Donnelly, Antony Rowstron, and Greg O Shea (MSR Cambridge) MapReduce Overview Input file
«Computer Science» Requirements for applicants by Innopolis University
«Computer Science» Requirements for applicants by Innopolis University Contents Architecture and Organization... 2 Digital Logic and Digital Systems... 2 Machine Level Representation of Data... 2 Assembly
Sun N1: Storage Virtualization and Oracle
OracleWorld 2003 Session 36707 - Sun N1: Storage Virtualization and Oracle Glenn Colaco Performance Engineer Sun Microsystems Performance and Availability Engineering September 9, 2003 Background PAE works
Indexing Large-Scale Data
Indexing Large-Scale Data Serge Abiteboul Ioana Manolescu Philippe Rigaux Marie-Christine Rousset Pierre Senellart Web Data Management and Distribution http://webdam.inria.fr/textbook November 16, 2010
Today CSCI Coda. Naming: Volumes. Coda GFS PAST. Instructor: Abhishek Chandra. Main Goals: Volume is a subtree in the naming space
Today CSCI 5105 Coda GFS PAST Instructor: Abhishek Chandra 2 Coda Main Goals: Availability: Work in the presence of disconnection Scalability: Support large number of users Successor of Andrew File System
CSE 530A. B+ Trees. Washington University Fall 2013
CSE 530A B+ Trees Washington University Fall 2013 B Trees A B tree is an ordered (non-binary) tree where the internal nodes can have a varying number of child nodes (within some range) B Trees When a key
UNIT-V WEB MINING. 3/18/2012 Prof. Asha Ambhaikar, RCET Bhilai.
UNIT-V WEB MINING 1 Mining the World-Wide Web 2 What is Web Mining? Discovering useful information from the World-Wide Web and its usage patterns. 3 Web search engines Index-based: search the Web, index
Desktop Crawls. Document Feeds. Document Feeds. Information Retrieval
Information Retrieval INFO 4300 / CS 4300! Web crawlers Retrieving web pages Crawling the web» Desktop crawlers» Document feeds File conversion Storing the documents Removing noise Desktop Crawls! Used
Introduction to MySQL Cluster: Architecture and Use
Introduction to MySQL Cluster: Architecture and Use Arjen Lentz, MySQL AB (arjen@mysql.com) (Based on an original paper by Stewart Smith, MySQL AB) An overview of the MySQL Cluster architecture, what's
Server Selection Mechanism. Server Selection Policy. Content Distribution Network. Content Distribution Networks. Proactive content replication
Content Distribution Network Content Distribution Networks COS : Advanced Computer Systems Lecture Mike Freedman Proactive content replication Content provider (e.g., CNN) contracts with a CDN CDN replicates
Media Pack. Home About User Benefits Advertiser Benefits Advertising Packages
Media Pack About Hotfrog One of Australia s largest online business directories Launched in May 2005 with the purpose to generate targeted sales leads for businesses Owned by the global Reed Elsevier publishing
CIS 601 Graduate Seminar. Dr. Sunnie S. Chung Dhruv Patel ( ) Kalpesh Sharma ( )
Guide: CIS 601 Graduate Seminar Presented By: Dr. Sunnie S. Chung Dhruv Patel (2652790) Kalpesh Sharma (2660576) Introduction Background Parallel Data Warehouse (PDW) Hive MongoDB Client-side Shared SQL
CPSC 426/526. Cloud Computing. Ennan Zhai. Computer Science Department Yale University
CPSC 426/526 Cloud Computing Ennan Zhai Computer Science Department Yale University Recall: Lec-7 In the lec-7, I talked about: - P2P vs Enterprise control - Firewall - NATs - Software defined network
NFSv4 as the Building Block for Fault Tolerant Applications
NFSv4 as the Building Block for Fault Tolerant Applications Alexandros Batsakis Overview Goal: To provide support for recoverability and application fault tolerance through the NFSv4 file system Motivation:
Search Engines. Information Retrieval in Practice
Search Engines Information Retrieval in Practice All slides Addison Wesley, 2008 Web Crawler Finds and downloads web pages automatically provides the collection for searching Web is huge and constantly
Designing Next Generation Data-Centers with Advanced Communication Protocols and Systems Services
Designing Next Generation Data-Centers with Advanced Communication Protocols and Systems Services P. Balaji, K. Vaidyanathan, S. Narravula, H. W. Jin and D. K. Panda Network Based Computing Laboratory
CSC 4900 Computer Networks: Link Layer (3)
CSC 4900 Computer Networks: Link Layer (3) Professor Henry Carter Fall 2017 Link Layer 6.1 Introduction and services 6.2 Error detection and correction 6.3 Multiple access protocols 6.4 LANs addressing,
ScaleArc for SQL Server
Solution Brief ScaleArc for SQL Server Overview Organizations around the world depend on SQL Server for their revenuegenerating, customer-facing applications, running their most business-critical operations
Internet Protocol Stack! Principles of Network Applications! Some Network Apps" (and Their Protocols)! Application-Layer Protocols! Our goals:!
Internet Protocol Stack! Principles of Network Applications! application: supporting network applications!! HTTP,, FTP, etc.! transport: endhost-endhost data transfer!! TCP, UDP! network: routing of datagrams
Dynamic Metadata Management for Petabyte-scale File Systems
Dynamic Metadata Management for Petabyte-scale File Systems Sage Weil Kristal T. Pollack, Scott A. Brandt, Ethan L. Miller UC Santa Cruz November 1, 2006 Presented by Jae Geuk, Kim System Overview Petabytes
Internet Content Distribution
Internet Content Distribution Chapter 1: Introduction Jussi Kangasharju Chapter Outline Introduction into content distribution Basic concepts TCP DNS HTTP Outline of the rest of the course Kangasharju:
big picture parallel db (one data center) mix of OLTP and batch analysis lots of data, high r/w rates, 1000s of cheap boxes thus many failures
Lecture 20 -- 11/20/2017 BigTable big picture parallel db (one data center) mix of OLTP and batch analysis lots of data, high r/w rates, 1000s of cheap boxes thus many failures what does paper say Google
This project must be done in groups of 2 3 people. Your group must partner with one other group (or two if we have add odd number of groups).
1/21/2015 CS 739 Distributed Systems Fall 2014 PmWIki / Project1 PmWIki / Project1 The goal for this project is to implement a small distributed system, to get experience in cooperatively developing a
Distributed Systems. 05r. Case study: Google Cluster Architecture. Paul Krzyzanowski. Rutgers University. Fall 2016
Distributed Systems 05r. Case study: Google Cluster Architecture Paul Krzyzanowski Rutgers University Fall 2016 1 A note about relevancy This describes the Google search cluster architecture in the mid
Uniform Resource Locators (URL)
The World Wide Web Web Web site consists of simply of pages of text and images A web pages are render by a web browser Retrieving a webpage online: Client open a web browser on the local machine The web
CONTENT DISTRIBUTION. Oliver Michel University of Illinois at Urbana-Champaign. October 25th, 2011
CONTENT DISTRIBUTION Oliver Michel University of Illinois at Urbana-Champaign October 25th, 2011 OVERVIEW 1. Why use advanced techniques for content distribution on the internet? 2. CoralCDN 3. Identifying
System Specification
NetBrain Integrated Edition 7.1 System Specification Version 7.1a Last Updated 2018-09-04 Copyright 2004-2018 NetBrain Technologies, Inc. All rights reserved. Introduction NetBrain Integrated Edition features
More on Testing and Large Scale Web Apps
More on Testing and Large Scale Web Apps Testing Functionality Tests - Unit tests: E.g. Mocha - Integration tests - End-to-end - E.g. Selenium - HTML CSS validation - forms and form validation - cookies
LegUp: Accelerating Memcached on Cloud FPGAs
0 LegUp: Accelerating Memcached on Cloud FPGAs Xilinx Developer Forum December 10, 2018 Andrew Canis & Ruolong Lian LegUp Computing Inc. 1 COMPUTE IS BECOMING SPECIALIZED 1 GPU Nvidia graphics cards are
RIGHTNOW A C E
RIGHTNOW A C E 2 0 1 4 2014 Aras 1 A C E 2 0 1 4 Scalability Test Projects Understanding the results 2014 Aras Overview Original Use Case Scalability vs Performance Scale to? Scaling the Database Server
Fault-Tolerant and Scalable TCP Splice and Web Server Architecture ; CU-CS
University of Colorado, Boulder CU Scholar Computer Science Technical Reports Computer Science Winter 1-1-2006 Fault-Tolerant and Scalable TCP Splice and Web Server Architecture ; CU-CS-1003-06 Manish
Monitor your infrastructure with the Elastic Beats. Monica Sarbu
Monitor your infrastructure with the Elastic Beats Monica Sarbu Monica Sarbu Team lead, Beats team Email: monica@elastic.co Twitter: 2 Monitor your servers Apache logs 3 Monitor your servers Apache logs
Scaling Web Service. Capacity planning. CS144: Web Applications
Scaling Web Service Capacity planning Q: How many requests can a machine handle? Really depends on your application Possible bottlenecks Disk/DB IO: disk 100-500MB/sec, 5-10ms avg seek (or ~100K IOs for
CSE 5306 Distributed Systems. Naming
CSE 5306 Distributed Systems Naming 1 Naming Names play a critical role in all computer systems To access resources, uniquely identify entities, or refer to locations To access an entity, you have resolve
Next-Generation Cloud Platform
Next-Generation Cloud Platform Jangwoo Kim Jun 24, 2013 E-mail: jangwoo@postech.ac.kr High Performance Computing Lab Department of Computer Science & Engineering Pohang University of Science and Technology
Search Quality. Jan Pedersen 10 September 2007
Search Quality Jan Pedersen 10 September 2007 Outline The Search Landscape A Framework for Quality RCFP Search Engine Architecture Detailed Issues 2 Search Landscape 2007 Source: Search Engine Watch: US
CSE 124: Networked Services Lecture-16
Fall 2010 CSE 124: Networked Services Lecture-16 Instructor: B. S. Manoj, Ph.D http://cseweb.ucsd.edu/classes/fa10/cse124 11/23/2010 CSE 124 Networked Services Fall 2010 1 Updates PlanetLab experiments
What is the relationship between a domain name (e.g., youtube.com) and an IP address?
DNS, CDNs March 30, 2009 Lecture 14 Sam Madden DNS What is the relationship between a domain name (e.g., youtube.com) and an IP address? DNS is the system that determines this mapping. Basic idea: You
Networking in a Vertically Scaled World
Networking in a Vertically Scaled World David S. Miller Red Hat Inc. LinuxTAG, Berlin, 2008 OUTLINE NETWORK PRINCIPLES MICROPROCESSOR HISTORY IMPLICATIONS FOR NETWORKING LINUX KERNEL HORIZONTAL NETWORK
Advanced Computer Networking. CYBR 230 Jeff Shafer University of the Pacific. Project 2
CYBR 230 Jeff Shafer University of the Pacific Project 2 2 Schedule This Week Mon September 18 Project 1 Work Wed September 20 Project 1 Testing (Grading) Fri September 22 Start Project 2 Next Week Mon
The Design and Implementation of a Next Generation Name Service for the Internet (CoDoNS) Presented By: Kamalakar Kambhatla
The Design and Implementation of a Next Generation Name Service for the Internet (CoDoNS) Venugopalan Ramasubramanian Emin Gün Sirer Presented By: Kamalakar Kambhatla * Slides adapted from the paper -
Lecture 18 Overview. Last Lecture. This Lecture. Next Lecture. Internet Protocol (1) Internet Protocol (2)
Last Lecture Internet Protocol (1) This Lecture Internet Protocol (2) Lecture 18 Overview Source: chapters 19.1, 19.2, 22,1 22.2, 26.6 Next Lecture Transport Control Protocol (1) Source: chapters 24.1,
Got Isilon? Need IOPS? Get Avere.
Got Isilon? Need IOPS? Get Avere. Scalable I/O Performance to Complement Any EMC Isilon Environment By: Jeff Tabor, Director of Product Marketing Achieving Performance Scaling Overcoming Random I/O and
Improve Web Application Performance with Zend Platform
Improve Web Application Performance with Zend Platform Shahar Evron Zend Sr. PHP Specialist Copyright 2007, Zend Technologies Inc. Agenda Benchmark Setup Comprehensive Performance Multilayered Caching
No, the bogus packet will fail the integrity check (which uses a shared MAC key).!
1. High level questions a. Suppose Alice and Bob are communicating over an SSL session. Suppose an attacker, who does not have any of the shared keys, inserts a bogus TCP segment into a packet stream with
Anatomy of a search engine. Design criteria of a search engine Architecture Data structures
Anatomy of a search engine Design criteria of a search engine Architecture Data structures Step-1: Crawling the web Google has a fast distributed crawling system Each crawler keeps roughly 300 connection
Warehouse-Scale Computing
ecture 31 Computer Science 61C Spring 2017 April 7th, 2017 Warehouse-Scale Computing 1 New-School Machine Structures (It s a bit more complicated!) Software Hardware Parallel Requests Assigned to computer
Running Head: HOW A SEARCH ENGINE WORKS 1. How a Search Engine Works. Sara Davis INFO Spring Erika Gutierrez.
Running Head: 1 How a Search Engine Works Sara Davis INFO 4206.001 Spring 2016 Erika Gutierrez May 1, 2016 2 Search engines come in many forms and types, but they all follow three basic steps: crawling,
CSE 124: Networked Services Fall 2009 Lecture-19
CSE 124: Networked Services Fall 2009 Lecture-19 Instructor: B. S. Manoj, Ph.D http://cseweb.ucsd.edu/classes/fa09/cse124 Some of these slides are adapted from various sources/individuals including but
Lecture 16: Router Design
Lecture 16: Router Design CSE 123: Computer Networks Alex C. Snoeren Eample courtesy Mike Freedman Lecture 16 Overview End-to-end lookup and forwarding example Router internals Buffering Scheduling 2 Example:
Cloud Programming. Programming Environment Oct 29, 2015 Osamu Tatebe
Cloud Programming Programming Environment Oct 29, 2015 Osamu Tatebe Cloud Computing Only required amount of CPU and storage can be used anytime from anywhere via network Availability, throughput, reliability
The Google File System
The Google File System Sanjay Ghemawat, Howard Gobioff and Shun Tak Leung Google* Shivesh Kumar Sharma fl4164@wayne.edu Fall 2015 004395771 Overview Google file system is a scalable distributed file system
RAMCloud. Scalable High-Performance Storage Entirely in DRAM. by John Ousterhout et al. Stanford University. presented by Slavik Derevyanko
RAMCloud Scalable High-Performance Storage Entirely in DRAM 2009 by John Ousterhout et al. Stanford University presented by Slavik Derevyanko Outline RAMCloud project overview Motivation for RAMCloud storage:
Diffusing Your Mobile Apps: Extending In-Network Function Virtualisation to Mobile Function Offloading
Diffusing Your Mobile Apps: Extending In-Network Function Virtualisation to Mobile Function Offloading Mario Almeida, Liang Wang*, Jeremy Blackburn, Konstantina Papagiannaki, Jon Crowcroft* Telefonica
Fundamentals of Windows Server 2008 Network and Applications Infrastructure
COURSE OVERVIEW This five-day instructor-led course introduces students to network and applications infrastructure concepts and configurations provided by Window Server 2008. Students will be able to acquire
BIG-IP DNS: Monitors Reference. Version 12.1
BIG-IP DNS: Monitors Reference Version 12.1 Table of Contents Table of Contents Monitors Concepts...5 Purpose of monitors...5 Benefits of monitors...5 Methods of monitoring...5 Comparison of monitoring