High Availability High Performance Plone

Transcription

1 High Availability High Performance Plone Guido Stevens Social Knowledge Technology

2 Plone Worldwide

3 Resilience

4 Please wave, to improve my speech

5

6 Plone as usual Aspeli: über-buildout for a production Plone server Regebro: Plone-Buildout-Example nginx frontend varnish cache haproxy balancer 4x plone instance zeo backend

7 Plone as usual

8 Plone as usual webserver :80

9 Plone as usual caching

10 Plone as usual balancing across Plone instances

11 Plone as usual Plone instances

12 Plone as usual ZEO backend

13 Meet the client High-profile internet technology NGO Slashdot traffic levels 0.4 million page views / peak day 4 40 million page views / month million hits / month Mission-critical web presence 100% uptime previous 5 years Non-Plone sysadmins High security

14 No can do SPOF WTF? SPOF

15 Architecture Goals Must convince file-based 100% uptime sysadmins No SPOF eliminate all Single Points Of Failure Automated failover no manual intervention Extreme performance Extreme resilience killall -9 Plone

16 Meet Paul Stevens My brother mod_wodan + DBmail Plone developer pjstevns on irc/github/etc NFG Net Facilities Group premium hosting 24/7 MySQL HA since stone age

17 Plone as usual

18 3-tier

19 Plone as usual

20 Duplicate setup

21 Load Balancer

22 Load Balancer Client provided hardware load balancer Alternative: Linux Virtual Server + HAproxy 2x HAproxy in active/passive config this would be an EXTRA layer of HAproxy not shown in diagram use highly available virtual IP address monitor with Heartbeat or comparable failover virtual IP addres with arping broadcasts Alternative: AWS

23 Load Balancer

24 Ensure physical separation Ensure redundancy across physical servers no use to fail over on same machine separate machines in separate data centers Gotcha: moving virtuals around Disable HA facilities of virtualization platform We'll do our own HA

25 Full cluster

26 Replacing ZEO

27 ZEO versus Relstorage ZEO Relstorage ZEO protocol ZEO protocol filestorage MySQL or PostgreSQL object pickles object pickles: no alchemy! ZRS Replication MySQL replication $$$ at the time done that 24/7 since 2001 later opensourced widely used No hot-failover slave master reconfig Hot failover multi-master

28 Relstorage on MySQL

29 Blobstorage Not shown in diagram Client provided Netapp Metrocluster NFS disks no need to care about replication and HA for those Alternatives: DRBD + NFS AWS Elastic Block Device F-sniper + rsync + NFS Why not run database on that? disk replication + NFS + ZEO what can possibly go wrong?

30 Full cluster

31 Apache + Wodan

32 mod_wodan Caching module for Apache C Originally by ICS for nu.nl Now maintained by NFG Store response body + headers on disk BOFH attitude to caching policies Used in anger Alternative: stxnext.staticdeployment

33 Varnish Wodan Proxy process Apache module RAM memory cache Persistent disk cache restart empty cache restart full cache expired gone expired keep fallback Plays nice BOFH request + response headers my way or the highway etag split-view single cache file per page purge API plone.app.caching Cronjobs maintenance crawl sitemap delete removed pages

34 Varnish plus Wodan Varnish Wodan unload Plone failsafe content delivery plone.app.caching policies hard policy config pages 1 hour pages 1 minute resources longer resources longer purge on edit edit 1-minute refresh etag split-view per-user page versions cache authenticated Gotcha: anonymous only editors bypass Wodan

35 Failure Modes

36 Full cluster

37 MySQL failover

38 Multi Master MySQL multi-master cross replication any can be master each slaves the other hot failover and failback Gotcha: use only 1 master at a time Relstorage is not multi-master avoid replication errors mmm_agent server (not shown in diagram) monitors mysql health and replication manages virtual MySQL HA ip address think: Heartbeat for MySQL

39 Blade failure

40 Wodan only

41 Plone as usual file-based content delivery

42 Readonly Rescue Mode File-based content delivery mod_wodan full cache of all pages + resources cached search results (Subject / tag cloud) AJAX-driven graceful degradation detect backend down via non-cached lightweight view disable interactive elements via CSS search bar, personal tools display:none Gotcha: anonymous not a full page: minimal rendering overhead down for authenticated until manual reconfig Gotcha: ErrorDocument pre-cache nice page but preserve http error status code

43 No-downtime maintenance

44 Full cluster

45 cosent.nl/blog