+ + a journey to zero-downtime

Transcription

1

2 + + a journey to zero-downtime

3 James Halsall Technical Team Inviqa

4 Christian Dornhoff Head of IT toom

5 A bit about toom baumarkt One of Germany s biggest DIY companies

6 Annual turnover of ~ 2.6 billion

7 Around 350 markets

8 Technical challenges Translating the toom model to Magento

9 Some technical challenges Existing distributed architecture Market specific pricing and stock Huge volumes of stock changes Large catalog provided by a middleware Zero-downtime deployments

10 toom s distributed system Fitting Magento into the existing architecture

11 Problem: Different backends for different routes

12 Solution: Varnish as a HTTP router (and HTTP cache)

13 Browser HTTP Request GET Varnish VCL config GET /c/* GET /p/* GET /* GET /bestellen CMS Shop Typo3 Magento 2.2

14 Problem: Multiple services used to handle a single request

15 Solution: Varnish ESIs Varnish assembles the page a

16 Browser HTTP Request GET /p/name/sku Varnish <esi:include src= /> VCL config Composition Microservice Shop Magento 2.2

17 Market specific pricing and stock Enabling online market reservations

18 350+ markets We needed to model these in Magento

19 Magento stores? Increased load time, bigger indexes a

20 Solution: custom Market model Bespoke stock / price model too

21 Replace Magento s stock registry

22 Replace Magento s price pool types

23 Custom price + stock indexes Increased runtime performance a

24 Go microservices for stock + price

25 Shop MySQL Magento 2.2 Magento DB Microservice stock-service Microservice stock-changeimport Kafka stocksales

26 Kafka provides real-time updates Stock and price changes as they happen

27 Problem: Millions of stock updates weekly How do we keep indexes up-to-date?

28 Solution: Index updates with MView

29 Building huge indexes Tens of millions of entries

30 Problem: Large query results loaded into PHP memory by default

31 Queries are using the buffered mode by default. This means that query results are immediately transferred from the MySQL Server to PHP and then are kept in the memory of the PHP process

32 Solution: Unbuffered queries a use_buffered_query => false

33 Unbuffered MySQL queries execute the query and then return a resource while the data is still waiting on the MySQL server for being fetched. This uses less memory on the PHP-side, but can increase the load on the server.

34 $conn = $resourceconnection->getconnection($name); $config = $conn->getconfig(); $config['use_buffered_query'] = false; $unbufferedconn = $objectmanager->create( get_class($conn), ['config' => $config] );

35 MySQL server now buffers the result Much smaller memory footprint in PHP process a

36 Lesson: Indexes are powerful, use them Partial index updates are great, but benchmark for your use case

37 Lesson: Profile your code or

38 Middleware managed catalog Improving performance of the Magento REST API

39 Large product catalog 100,000+ products, attributes and 900+ sets

40 Middleware receives data from PIM +

41 MySQL Magento Magento DB REST API POST /rest/v1/products HTTP 200 Success S4T Connect PIM Middleware Product Data

42 Problem: slow full catalog import Needed to improve this

43 Solution: parallelise the REST API RabbitMQ + background processing

44 Request is stored in RabbitMQ HTTP 202 Accepted response code

45 RabbitMQ catalog-requests Magento Magento Magento Magento queue processor queue processor queue processor REST API POST /rest/v1/products HTTP 202 Accepted MySQL S4T Connect Magento DB Middleware

46 bin/magento queue:consumers:start --max-messages=100

47 API response time decreased 850% response time improvement

48 Product creation time improved 6x faster to persist 1000 products in the catalog

49 Lesson: Think outside the box Magento provides us with the tools to create tailored solutions

50 Zero-downtime deployments Maintenance page-free deploys

51 Database migrations Automated database updates executed during the deployment

52 Native Magento database migrations module.xml and setup_version

53 Problem: module version mismatches Magento will return a 5xx response

54 Browser Magento 2 MySQL GET /p/name/sku Check module versions HTTP 500 Versions returned

55 Maintenance page = downtime Breaks zero-downtime for toom

56 Our solution: Phinx migrations and Magento a

57 Provides new trigger for DB changes No need to change setup_version in module.xml

58 New database migration process for deploys 1. Run bin/magento setup:db:status 2. If exit code is 2, run bin/magento setup:upgrade 3. Run bin/phinx migrate 4. Continue as normal

59 Make DB changes backwards compatible Should work with both old and new code

60 Problem: shared cache data During deploy both old and new code can be hit by web traffic

61 Redis Cache DB Magento Magento Magento Magento New Code New Code Old Code Old Code HTTP Request

62 Solution: Alternate Redis database

63 Redis New DB Old DB Magento Magento Magento Magento New Code New Code Old Code Old Code HTTP Request

64 Zero Downtime a

65 Lesson: Try and avoid setup:upgrade on every deployment

66

67 Key take-aways What we learned from toom + Magento

68 1. Magento is extremely flexible We use Magento as part of a distributed system with ease

69 2. Big customisations are achievable

70 3. Parallelise big tasks with RabbitMQ

71 4. Custom indexes are powerful

72 5. Always profile your solutions

73 6. Zero-downtime deploys are real

74 +

75 Thanks a

76