Inside Hatena Bookmark's Backend

Transcription

1 Inside Hatena Bookmark's Backend Naoya Ito CTO, Hatena co., ltd.

2 Agenda About Hatena Bookmark System Overview Backend Evolution mod_perl 2.0 Software Updates Caching on Reverse Proxy Other Topics

3 About Hatena Bookmark

4 Hatena Bookmark Social Bookmark Service It's not delicious

5 Overview The most popular social bookmark service in Japan Hatena Keywords and Tags Web APIs Digital Contents Grandprix Award

6 Numbers 35,000 users 45M PV/month 15,000 lines of code

7 Traffic

8 System Overview

9 reverse proxy User's Request bot / RSS / API mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl Other Services mysql mysql mysql mysql mysql (master) mysql

10 Hardware 17 DIY PC Servers 10 app servers 6 DB (1 master / 5 slaves) 1 Reverse Proxy

11 Software LAMP Fedora Core Apache 2.0 / Apache 2.2 MySQL 4.0 not PHP, not Python

12 Other Softwares Senna MRTG Nagios Capistrano (Switch Tower) Subversion

13 Hatena Framework Since 2001 MC of MVC V = TT or H::T DI Container with YAML Plug-In Helper Script / server.pl like Catalyst

14 Backend Evolution

15 Beginning... 3 servers 3 Servers Reverse Proxy (Apache 2.0) mod_proxy + mod_rewrite Application Server (mod_perl 1.99) DB

16 reverse proxy User's Request mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mysql (slave) mysql (slave) mysql (slave) mysql (slave) mysql (master) mysql (slave)

17 Reverse Proxy Setting RewriteRule ^/images/ - [L] RewriteRule ^/css/ - [L] RewriteRule ^/js/ - [L] RewriteRule ^/theme/ - [L] RewriteRule ^/(.*)$ [P]

18 Apache Tuning reverse proxy app server Purpose static files dynamic contents KeepAlive On Off MaxClient MPM prefork prefork memory size / proc Tiny Massive!

19 What is the "load"? I/O swap CPU It's difficult to fully use CPU Limitation of I/O and/or Memory

20 MySQL MySQL 4.0 MyISAM Simple and Easy Almost all of queries are SELECT! So, we don't need InnoDB MERGE table

21 3 + 1 = 4 servers Need additional MySQL High load Replication! MySQL's replication is easy and good.

22 reverse proxy User's Request mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl Other Services mysql mysql (slave) mysql (slave) mysql (slave) mysql (master) mysql (slave)

23 2 DB servers R to Slaves / CUD to Master We have O/R mapper Gathering connection logics in a class. Easy to extend for 2 servers. Class::DBI::Replication

24 On Memory Table Data Technique for MySQL Use tmpfs RAM disk for Linux 2.4 / 2.6 Move some/all files of table to tmpfs on your slave server. No Disk I/O with MySQL! Server crash is not a problem - just replica.

25 How to do it $ sudo mkdir -p /shm/mysql/bookmark $ sudo mount -t tmpfs -o size=1024m tmpfs /dev/shm $ sudo mount -t tmpfs -o size=1024m /dev/shm $ cd /var/lib/mysql/bookmark $ sudo cp *.* /shm/mysql/bookmark $ sudo vi /etc/my.cnf datadir=/shm/mysql/bookmark

26 Be careful Keep your tmpfs beging NOT full alter table, myisamchk needs some space. Need a large memory size of tmpfs + MySQL threads 4GB memory limit

27 MySQL for caching With tmpfs, you can use MySQL for caching like memcached. It can't be distributed completely. Flexible caching "select storabled_data from cache order by timestamp desc limit 5"

28 4 + 2 = 6 servers More Apache Max Client, High load...

29 reverse proxy User's Request mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl Other Services mysql mysql (slave) mysql (slave) mysql (slave) mysql (master) mysql (slave)

30 Load Balance Now we need load balance we can do it with Apache 2.0 mod_proxy + mod_rewrite use ReriteMap

31 Reverse Proxy Setting RewriteRule ^/images/ - [L] RewriteRule ^/css/ - [L] RewriteRule ^/js/ - [L] RewriteRule ^/theme/ - [L] RewriteMap backend rnd:/home/httpd/bookmark/lib/backend.txt RewriteRule ^/(.*)$ [P] # backend.txt

32 More Servers Traffics grow larger and larger. We need more servers. Just buy and add new servers. It's very simple.

33 reverse proxy User's Request mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl Other Services mysql mysql mysql mysql (slave) mysql (master) mysql (slave) mysql (slave)

34 Problems No failover with reverse proxy bots bots bots Too many connections with Apache::DBI

35 Load Balancer Hardware Switch Software UltraMonkey (LVS) Apache 2.2 (mod_proxy_balancer)

36 mod_proxy_balancer Already contained in Apache Easy! failover Load Balancer Manager little change from RewriteMap stable (probably)

37 Reverse Proxy Setting <Proxy balancer://bookmark> BalancerMember loadfactor=10 BalancerMember loadfactor=10 BalancerMember loadfactor=10 </Proxy> RewriteRule ^/(.*)$ balancer://bookmark/$1 [P]

38 Bots Yahoo! Slurp / Googlebot / Bloglines 10-20% requests Now separated from user's request Fasterfox Spidering Tools

39 reverse proxy User's Request bot / RSS / API mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl mod_perl Other Services mysql mysql mysql mysql mysql (master) mysql

40 Fasterfox / Spidering Tools Fasterfox not a big problem (only for.html) robots.txt to prevent prefetching Spidering Tools "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)" kicked them manually with IP addresses We failed building some automation tools

41 Apache::DBI Problem Too many connections 50 procs = 50 connections x 10 servers = 500 connections

42 No more using Apache::DBI use pnotes() of mod_perl API share Perl variables in a Request Store connection handler to pnotes() 1 connection per 1 request Configure MySQL Thread Cache thread_cache_size = 50

43 mod_perl 2.0

44 mod_perl 1.99 Beta version Incompatible with 2.0 CGI.pm does not support 1.99 anymore

45 Why 2.0? A Little faster than 1.0 Apache 2 worker I/O Filtering Clean way to capture the data

46 Migration ( ) to 2.0 "Apache::" to "Apache2::" "handler ($$)" to "handler : method" "use Apache" to "use Apache2::*" RequestRec / RequestIO / Const...

47 ModPerl::MethodLookup $ perl -Mmod_perl2 -MModPerl::MethodLookup -e print_method send_http_header 'send_http_header' is not a part of the mod_perl 2.0 API use 'content_type' instead. To use method 'content_type' add: use Apache2::RequestRec ();

48 Handler example package Sandbox::Hello; use strict; use warnings; use Apache2::ReqeustRec(); use Apache2::RequestIO(); use Apache2::Const -compile => 'OK'; sub handler : method { my ($class, $r) $r->content_type('text/plain'); $r->print('hello, World!'); return Apache2::Const::OK; } 1;

49 ModPerl::VersionUtil use ModPerl::VersionUtil; if (ModPerl::VersionUtil->is_mp2) { require Apache2 (); require Apache2::RequestRec(); require Apache2::RequestIO (); } elsif (ModPerl::VersionUtil->is_mp19) { require Apache2; require Apache::RequestRec(); require Apache::RequestIO (); } elsif (ModPerl::VersionUtil->is_mp1) { require Apache; }

50 Apache2::Request Almost compatible with Apache::Request $apr->param(foo => 'bar') does not work well.

51 Our changes Migrate 1.99 to 2.0 Change request module from CGI to Apache2::Request We wrote the Adapter Use static linked httpd instead of DSO Result: 25% faster

52 Software Updates

53 Software Updates Linux packages... yum CPAN modules... manually :( Our modules... switchtower

54 switchtower # deploy our application to the server $./switchtower deploy # update development enviroment with svn $./switchtower current_update # update production environment $./switchtower -e production current_update # restart production web servers $./switchtower -e production restart

55 Caching on Reverse Proxy

56 Linked by Yahoo! News Massive accesses! Chance to Monetize! But we don't have enough resouces Site goes down

57 mod_hatena_cache Threshold of request #s in some period On reverse proxy C API with memcached Apache2 Output Filter Watching user's cookie

58 Other Topics

59 Restart? I always think if it could be restarted. " " HTML::Template "Copy & Paste", Dirty codes Better framework Need Test

60 We have Lager System Hatena Diary 300M PV/month 100 over servers half of servers are MySQL partitioning / balancing / algorithms... (like mixi.)

61 Next Step Solve "Single Point of Failure" Reverse proxy Master DB Distributed File System? Intelligent Load Balancer like UltraMonkey? from IT company to High-Tech company

62 Thank you!