welcome to production Graeme Foster Spanish Software Developer

Transcription

1 welcome to production Graeme Foster Spanish Software Developer

2 5 years ago, I co-designed, and built a MONOLITH We based in on the buzzwords and ideas of the time like ORMs, DDD, and web services. Like proud parents, we released it into the wild in 2011

3 System Architecture Fat Client (WPF) Web Services via WCF, aka Death-* Object Oriented Domain Azure SQL IIS IIS load balancer Queue Some DDD concepts but not the important ones ORM (NHibernate) & a SQL database Warehouse populated via audit-trail Store WPF WPF WPF Warehouse

4 Move to Azure (Singapore) Single Database Per Store Small number of users Hosted locally

5 Issues begin to appear Performance Scaling Concurrency

6 How will we diagnose issues? Log Files Application Profilers Network Profilers SQL profiling tools Intuition

7 Dear Diary The following incidents are based on real world incidents that occurred to this system. Name have been altered to protect the innocent!!

8 Support users are struggling to complete large sales Server Log files Fast operation

9 As seen by Fiddler

10 The Chatter-Box Api

11 Caused by looped service calls in code class SaleViewModel { public void OnFinishSale(SaleRequest request) { DoItemPreCheck(request.Items); } } private void DoItemPreCheck(Items[] items) { } if (items.any(!stockservice.isavailableforsale)) { ShowCannotCompleteSaleMessage(...); }

12 Why did it happen? The ping time from a terminal to Singapore is about 350ms Calling a service multiple times serially, on the main UI thread, caused the application to lock Most of the time was due to network latency

13 Resolution We own the producers and consumers of the API. It s simple for us to make the interface much less chatty. Don t use chatty api s over the network

14 Support Balancing tills is taking a long time Network Profiler Application Profiler Single service call from terminal Slow

15 As seen by the profiler

16 The family of N+1 s

17 Caused by looping over sub-collections class TillGroup { public decimal Balance { }... } class MoneyBag { } get { return moneybags.sum(t => t.balance); } Balance { get { return payments.sum(t => t.amount); } }

18 Resolution We hoisted the balance property onto the Till, and pre-calculated. Getting the balance of a till-group was now a simple read from the database public void AddTransaction(Transaction tran) { transactions.add(tran); Balance += tran.moneybags.sum(t => t.amount); } public decimal Balance { get; protected set; }

19 Support Till balancing doesn t always work Application Profiler Fast

20 Found in the log file Timestamp Thread Message :32: (1) retail sale begin :32: (2) retail sale begin :32: (1) retail sale end :32: (2) retail sale end Balance

21 Poor concurrency choice

22 Caused by our Sql Isolation Level public void AddTransaction(Transaction tran) { transactions.add(tran); Balance += tran.moneybags.sum(t => t.amount); } public decimal Balance { get; protected set; } session.begintransaction(isolationlevel.readcommitted);

23 Resolution We didn t nail the concurrency model early on We took the database default of Read Committed Optimistic concurrency models are well supported on ORM s like Hibernate but moving to one could be a costly exercise for us. But they have a hidden side effect which disables sql batching But luckily, we have single stores per database, and a small number of users session.begintransaction(isolationlevel.serializable);

24 Hive databases (up to 50 stores) Large number of users Hosted in Singapore Database approaching 40GB in size >400 concurrent users

25 Sometimes it all goes wrong!! Support Victoria can t trade. The entire system is down Application Profiler Lit up like a Christmas tree

26 Suspicious log file entries Timestamp Thread Message :32: (1) finish stocktake :32: (2) update stock :32: (3) new retail sale :32: (4) find stock :33: (2) update stock timeout :33: (8) finish stocktake :33: (3) new retail sale timeout :33: (4) find stock timeout

27 What does finish stocktake do?

28 Self induced denial of service

29 What the!#!##!? The Stock Take operation greedily locked rows causing other operations to block The ORM was not able to batch effectively and took a long time to flush Multi-tenanting had rendered some of our sql indexes useless The terminal timed out, and the user kept retrying The server-side did not time out

30 Potential fix #1 The hibernate family of ORM have flags to improve batching Batching enables more sql to be sent to the database per call Can be a real improvement when latency to the database is high But batch operations will still take some time

31 Potential fix #2 Move work to background processes to spread the load! For many use-cases your user s don t care if things don t happen instantly.

32 Quick resolution Sometimes you need to get the network moving again... if (time is between 8am and 7pm) { throw new DomainException( "Sorry. Please try again outside of business hours"); }

33 Conclusions Avoid chatty distributed components Keep your indexes tuned Think about concurrency Watch that ORM!!! Fetch strategies, horrid joins, fetching too much, fetching too little!

34 Thank you for listening Graeme