Do-It-Yourself: WebSphere Commerce Problem Determination

Transcription

1 IBM Software Group Do-It-Yourself: WebSphere Commerce Problem Determination Mike Callaghan David La Gamba WebSphere Support Technical Exchange

2 Agenda Problems that lay ahead Typical web request flow Isolating the problem Questions to ask and how to debug: Web Browser Web Server Database Application Server (and Commerce application) Integrations Example: Request flow How to approach performance problems Where to go for more help WebSphere Support Technical Exchange 2 of 37

3 Problems? What problems?! Can face many types of problems during any phase of the project: Design and planning Development stage Testing Launch Maintenance or upgrades Runtime staging/qa/production issues Important to isolate specific problem, know how and where to find answers Commerce Non-runtime problems include install, configuration, deployment, feature enablement, development Commerce Runtime problems include orders, payments, web services, messaging, users and cookies, performance We will focus on troubleshooting of runtime issues here WebSphere Support Technical Exchange 3 of 37

4 But what s in it for me? Self-sufficiency will save you time, money, frustration! Don t want to become dependant on us when we are not around Save time working through the issue yourself before contacting us Narrow down the specific issue or progress the problem determination so we can pick up where you left off At very least you could already have the information we will request available As you understand what causes the problems, you learn to avoid problems altogether, or learn how to solve them quicker WebSphere Support Technical Exchange 4 of 37

5 Where did my request go? Runtime issue could be anywhere along the way Browser: client side Javascript, AJAX requests, cookie acceptance, cookie/url limits, pop-ups Web Server: virtual hosts, SSL certificate, rewrites/redirects, ports WAS Plug-in: mapped modules, cluster, load balancing, transports, virtual hosts Application Server: servlet filters, Commerce servlets, caching Database: connections, SQLs, contention, data itself WebSphere Support Technical Exchange 5 of 37

6 Ok I have a problem, but where is it? Need to know what to look for at each tier Often easiest to work from application outwards When focusing of Commerce, component-specific MustGathers are a good place to start SystemOut.log and trace.log can point you to the next step: No request at all in log (check the web server logs or browser) See the logged request, but input seems wrong (check browser) Application exceptions (application itself is the issue, or the data) Rollback, timeout, locking exceptions (consider the database) Just simply wrong results (check the application code, or the data and config) To find the right answers you need to start by asking the right questions WebSphere Support Technical Exchange 6 of 37

7 Questions to ask: Web Browser Think what can happen between the request being initiated by web browser and it arriving at web server (and vice-versa) Questions to ask yourself: Internet Explorer vs. Firefox vs. Safari (and release of each) Is the request as expected? Or is the response the problem? (bad response = server side problem instead?) What is in between the browser and web server? (proxy, firewall, hosts entry, network) Secure (https) or non-secure (http)? Is there a problem with cookies? What cookies do you see? (JSESSION, WC_AUTHENTICATION, WC_USERACTIVITY) WebSphere Support Technical Exchange 7 of 37

8 How to debug: Web Browser javascript:alert(document.cookie) Debug in Firefox if you can (Firebug, TamperData tools) WebSphere Support Technical Exchange 8 of 37

9 Questions to ask: Web Server Are the virtual hosts defined correctly? (httpd.conf) Listening on right ports and IP? (test by telnet) Local or remote web server? Is there a load balancer in front? SSL Accelerator? Does the request even make it to the web server? What is the response back from the application server? Can you hit the application server directly? WebSphere Support Technical Exchange 9 of 37

10 Questions to ask: WAS Plug-in Are the virtual hosts defined correctly? (plugin-cfg.xml) Can the web server communicate with the transports: What is the response from the app server? Does the plugin get the request to the right app server? Are the right modules mapped to the right virtual hosts? Are the web modules mapped to the right servers? WebSphere Support Technical Exchange 10 of 37

11 How to debug: Web Server & WAS Plug-in access_log shows timestamps, IP, and return code of each request, static content (served locally) and dynamic (routed to WAS via plugin) Use to track return code (200, 404, 500) Can modify httpd.conf to show response time for request [30/June/2009:16:25: ] "GET /webapp/wcs/stores/servlet/topcategoriesdisplay? langid=-1&storeid=10001&catalogid=10001 HTTP/1.1" Enable plugin trace (plugin-cfg.xml) to see the routing of the request to the right virtual host group, transport See if nodes marked as down Keep in mind that session affinity is only at the application server level, not the webserver WebSphere Support Technical Exchange 11 of 37

12 Questions to ask: Database Functional problem? Performance problem? (deadlocks/timeouts?) Or is the data just wrong? Are there too many SQLs running overall? Should I even have any SQLs running? (should be cached?) Are there long running SQLs? Is there contention against certain objects? (revisit the code) Are you performing regular database maintenance? dbclean on guest users, junk order, scheduler tables runstats, reorg, rebind regularly (DB2) Any potentially problematic data? Large orders, recursive catalog, huge category tree WebSphere Support Technical Exchange 12 of 37

13 How to debug: Database (DB2) db2diag.log is a key diagnostic error log, application receiving those errors will log this information (ie. Check for lock escalations) Database snapshots (capture info about database and any connection apps, observe trends, predict problems Locks, statements, sorts, bufferpools, tables, etc For problematic queries db2expln/db2advis (inefficient SQLs or missing index) SYSCAT.tables, SYSCAT.indexes stats (regular maintenance) Actual data (cardinality, LOBs, etc) Event monitors if you need to see actual statements Deadlock vs. Statement monitors Javacores could help pin down the source for long running queries Lock waits vs. lock timeouts vs. deadlocks What else is there? data loads, staging props, dbclean, messaging feeds WebSphere Support Technical Exchange 13 of 37

14 Questions to ask: Application (Commerce) Current behaviour compared to the expected behaviour? Did it ever work? Have there been any recent changes? configuration, code or code deployments, data propagation Reproducible? Can you reproduce locally? Intermittent? Functional problem? Or a performance problem? Happens on production? Development? Or all environments? Is this a cached or non-cached request? Does this happen for one particular user? Or for all users? What levels of Commerce, WAS, DB2/Oracle are you on? (release, fixpack, APARs) Is this a data issue? Or a code issue? OOB code or custom? Working case you can compare to? Clustered environment? Multiple tiers? What else is in the picture? (MQ, web services, Akami, Endeca, Omnifind, F5 Load Balancer, proxy, data feeds, scheduled jobs) WebSphere Support Technical Exchange 14 of 37

15 How to debug: Application (Commerce) Typical flow once request hits application server: Each present its own problems and configurations Component specific traces always help WebSphere Support Technical Exchange 15 of 37

16 How to debug: Application (Commerce) Runtime servlet filter: Check here for everything cookie and session related Cache filter (dynamic caching): Check if should be cached in cachespec.xml Check if actually cached in cache monitor, or if request hits ECActionServlet in trace (or intercepted before) Stores servlet Bread and butter here Struts config files, controller commands Database EJBs, datasource, connection pooling Presentation logic (Views) Struts config again, JSPs, databeans, AJAX calls WebSphere Support Technical Exchange 16 of 37

17 [Aside] Why do you keep asking me for traces? Easiest way to see which code is involved, Which classes, which methods, what inputs, what outputs Comparison to other cases Compare your working case to your non-working case Compare our local working case to your non-working case IBM OOB code vs. custom code, when does behaviour change Find which traces to enable in InfoCenter page, or MustGather documents WC_SERVER is a great default in all cases Tracks the who, what, where, when, how You will need to find out the why! Very verbose, be warned! Important: Be sure to include traces in your own custom code too so it is serviceable! WebSphere Support Technical Exchange 17 of 37

18 How to debug: Reading the traces! No silver bullet, there are various techniques depending on the scenario and component General techniques we use include tracking: JSESSION cookie (common between browser requests) WC_USERACTIVITY cookie (common to a particular user) Java thread ID (common through the life of one request) Component specific (ORDERS_ID, USERS_ID, etc) Compare to working case, find differences Compare two failing cases, find similarities Exceptions are your friend! (half the work is done for you) Custom code involved? We need to know Always good to look up the corresponding data in the DB WebSphere Support Technical Exchange 18 of 37

19 How to debug: Still reading the traces WebSphere Support Technical Exchange 19 of 37

20 Questions to ask: What else do you have? We only know what you tell us Spikes in CPU or common period for an outage should make you at least think of the bigger picture beyond your storefront/order flow Integrations often are a key difference in behaviour between your environment and out-of-the-box one Scheduled jobs Messaging, web services Caching (Akami, Edge caching, replication, etc) Catalog search (Endeca, OmniFind, etc) Proxies (SSL accelerator/load balancer) Data feeds, staging propagations WebSphere Support Technical Exchange 20 of 37

21 Show me! (Browser, Firebug, Access logs) Web browser (Add to cart) Firefox Firebug debugging WebServer Access Log [28/Jul/2009:10:36: ] "GET /webapp/wcs/stores/servlet/orderitemadd? storeid=10851&catalogid=10151&orderid=.&url=setpendingorder%3furl%3dordercalculate%3furl%3dorderitemdisplay% 26updatePrices%3d1%26calculationUsageId%3d-1%26orderId% 3d.&catEntryId=10451&quantity=1&errorViewName=ProductDisplayErrorView HTTP/1.1" WebSphere Support Technical Exchange 21 of 37

22 Show me! (Plugin trace, server trace) Plugin trace WC_SERVER trace TRACE: ws_common: websphereshouldhandlerequest: trying to match a route for: vhost='pogge'; uri='/webapp/wcs/stores/servlet/orderitemadd TRACE: ws_common: webspherevhostmatch: Comparing 'pogge.torolab.ibm.com:443' to 'pogge:80' in VhostGroup: VH_demo (case 2) TRACE: ws_common: webspherevhostmatch: Comparing 'pogge:80' to 'pogge:80' in VhostGroup: VH_demo (case 2) TRACE: ws_common: webspherevhostmatch: Found a match 'pogge:80' to 'pogge:80' in VhostGroup: VH_demo with score 5, exact match 5 TRACE: ws_common: websphereurimatch: Found a match '/webapp/wcs/stores/*' to '/webapp/wcs/stores/servlet/orderitemadd' in UriGroup: VH_demo_server1_WC_demo_node_Cluster_URIs with score 19, exact match 19 TRACE: lib_htrequest: htrequestseturl: Setting the url /webapp/wcs/stores/servlet/orderitemadd TRACE: lib_htrequest: htrequestseturl: Setting the query string storeid=10851&catalogid=10151&orderid=.&url=setpendingorder%3furl% 3dOrderCalculate%3fURL%3dOrderItemDisplay%26updatePrices%3d1%26calculationUsageId%3d-1%26orderId% 3d.&catEntryId=10451&quantity=1&errorViewName=ProductDisplayErrorView TRACE: ws_common: webspherefindtransport: Setting the transport(case 2): pogge.torolab.ibm.com on port 9060 TRACE: ws_common: webspheregetstream: socket 17 connected to pogge.torolab.ibm.com:9060 TRACE: lib_htrequest: htrequestwrite: Writing the request: GETwebapp/wcs/stores/servlet/OrderItemAdd?storeId=10851&catalogId=10151&orderId=.&URL=SetPendingOrder%3fURL%3dOrderCalculate%3fURL %3dOrderItemDisplay%26updatePrices%3d1%26calculationUsageId%3d-1%26orderId%3d.&catEntryId=10451 HTTP/1.1 TRACE: lib_htresponse: htresponseread: Reading the response: c TRACE: HTTP/ Found [7/28/09 10:34:12:909 EDT] b WC_SERVER 3 com.ibm.commerce.webcontroller.runtimeservletfilter dofilteraction URLINFO - Method = POST ServletPath = /servlet PathTranslated = /opt/websphere/appserver/profiles/demo/installedapps/wc_demo_cell/wc_demo.ear/stores.war/orderitemadd RequestURI = /webapp/wcs/stores/servlet/orderitemadd PathInfo = /OrderItemAdd.. HeaderNames: Cookie Header = WC_PERSISTENT=yREvsAWFOVXfQeN9IqAQ%2fgqBk3w%3d%0a%3b2009%2d07%2d24+11%3a06%3a20%2e623% 5f %2d2%5f10701%5f%2d1002%2c%2d1%2cUSD%5f10701; JSESSIONID=0000QfpAH7_71FzNjhAs7qLj0Mt:-1;. WebSphere Support Technical Exchange 22 of 37

23 Questions to ask: Performance issues Exactly what is having performance issues? Surfing the store front Are particular pages slow? Adding items to cart Completing an Order Is it slow in the tooling? (Accelerator, Management Center, etc) Does the JVM appear to be the source? Does the database appear to be the source? Is the proper network cable being used in the right network port? WebSphere Support Technical Exchange 23 of 37

24 How to debug: Performance issues Sometimes the answers to the previous questions may be unknown But you DO know where to look to figure it out: Verbose Garbage Collect logs Heap Dumps Java Cores HTTP Access logs Server logs (SystemOut, SystemErr, trace) Database snapshots OS specific monitoring (I/O, paging, native memory usage, CPU usage, etc) Try to pin down which tier it is, start with the database usually Don t forget the network! WebSphere Support Technical Exchange 24 of 37

25 Verbose GC: What good is this? Can see what s happening over time with Java Heap When you want to consider it: High CPU? Check if GC is always running Out-of-memory yet again? How big was the allocation? General high usage in memory? Check what sized objects are causing the failures Assist with memory tuning, determine optimal heap size Find out what large objects may be getting allocated into memory Minimal overhead, always leave it enabled! WebSphere Support Technical Exchange 25 of 37

26 Verbose GC: How do I read it? Can be viewed manually in a text editor (if you dare!) Different tools exist to analyze these files Supports favourite is PMAT (Pattern Modeling and Analysis Tool for Java Garbage Collector) Allows: Visualization over time of heap utilization How much time is being taken up performing garbage collection Patterns of memory utilization What to look for: How frequent are the GC cycles? How large are the allocations? How much heap is free How large is the current heap compared to the configured MAX heapsize? WebSphere Support Technical Exchange 26 of 37

27 Verbose GC: What does it look like? Which would you rather take home for mom to put on the fridge? WebSphere Support Technical Exchange 27 of 37

28 Heapdumps: What good are these things? A time and a place for everything: Verbose GC = Heap usage over time Heapdump = snapshot of objects in heap at time picture is taken Can be very, very (very) large in size Generated automatically by JVM when trouble hits, or can generate one manually Excellent source of information when looking into memory problems (OOM, unnecessarily high memory utilization) Provide good view of exactly what is happening in the heap when taken over period of time Find a large number of common objects allocated Find large objects allocated Good for finding memory leaks Support tool of choice? IBM Memory Analysis Tool WebSphere Support Technical Exchange 28 of 37

29 Heapdumps: What am I looking for? Leak suspects Need to look at the type of objects that are taking up a lot of the heap Large number of a particular type of object Not interested in char[] and Strings as they should be the most numerous objects in memory Beans do not belong at the top of the list GOOD (normal) Not so good WebSphere Support Technical Exchange 29 of 37

30 Javacores: What good are these things? Snapshot of what was running in the JVM Fraternal twin to heapdump combo package What can it tell you though? What was running at time of picture What was queued up to run Most importantly the java stack associated to the active threads But when do I want to look at them? For hangs, helps isolate the cause (compare threads from one core to the next) For OOM issues, can help show potential culprit that may have allocated large object (ie. Sitemap!) For general slowdowns, will tell you which code is running, and what it is in contention with View using IBM Thread and Monitor Dump Analyzer for Java Excellent when viewed in groups (ie. core s taken every x minutes) WebSphere Support Technical Exchange 30 of 37

31 Javacores: What am I looking for? You now have the power to pinpoint where the code was during its execution Gives you a starting point for investigation CURTHDINFO will tell you the current thread that was active Do not always just focus on the current running thread Innocent thread could be running don t be quick to judge! Check what the webcontainer threads are doing, if anything Compare threads between javacores Search for code that is always there Look for JSPs in the stacktrace (_jsp.) WebSphere Support Technical Exchange 31 of 37

32 My site went down! What do I do now? 1. Restart it (if hung, get javacores first) 2. Gather all logs, heapdumps, javacores (don t delete them yet!) 3. Remember what just happened. Specifically. (just loaded 1000 promotions) 4. Breathe. 5. Read the MustGather for when it happens next time (ie. hang, 100% CPU, or OOM) so you know what to gather 6. Join us next week for more pointers! ( Using IBM Tooling to improve your self-help abilities for WebSphere Commerce ) 7. Open a PMR WebSphere Support Technical Exchange 32 of 37

33 Summary Any site can face many troubles at any stage of the ballgame Troubleshooting is about asking the right questions first Once you know what is wrong, fixing it is another battle Think of the common trends and issues, rule them out Traces are your friend, the more, the merrier Isolate to a reproducible scenario Remember what makes your site unique to the OOB case, or another site you have implemented WebSphere Support Technical Exchange 33 of 37

34 Additional Troubleshooting Resources Next week: Using IBM Tooling to improve your self-help abilities for WebSphere Commerce MustGather: Read first for all WebSphere Commerce products WebSphere Commerce Trace Components com.ibm.commerce.admin.doc/refs/rlslogging.htm IBM Support Assistant Maintaining a WebSphere Commerce DB2 Database IBM Thread and Monitor Dump Analyzer for Java Memory Analyzer Pattern Modeling and Analysis Tool for Java Garbage Collector WebSphere Support Technical Exchange 34 of 37

35 Additional WebSphere Product Resources Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: developerworks/websphere/community/ Learn about other upcoming webcasts, conferences and events: software/websphere/events_1.html Join the Global WebSphere User Group Community: Access key product show-me demos and tutorials by visiting IBM Education Assistant: View a webcast replay with step-by-step instructions for using the Service Request (SR) tool for submitting problems electronically: support/d2w.html Sign up to receive weekly technical My Notifications s: support/einfo.html WebSphere Support Technical Exchange 35 of 37

36 Join WebSphere Support Technical Exchange on Facebook! Stay up-to-update on upcoming webcast sessions Suggest future topics Suggest program improvements Network with other product users And More Become a fan now! WebSphere Support Technical Exchange 36 of 37

37 Questions and Answers WebSphere Support Technical Exchange 37 of 37