The Z-Files: Field reports from the world of business critical PHP applications By Axel Schinke Senior Manager of Global Operations for Global Services
About this webinar Field reports from Zend Consulting Different topic areas Performance Architecture & Scalability Coaching & Mentoring Zend Software Sometimes it s not PHP Anonymized selection from Zend s global customer base To ensure data privacy: No security issues will be dealt with Overview: Zend Consulting 2
Performance
Performance Case #1 Our website is running stable, but slow. It was built using a selfdeveloped framework. The site needs definitely more hardware ressources than we have. 4
Performance - Case #1 3 days Time & Material Consulting - Remote Analysis of the architecture Because of security reasons parts of the data were saved encrypted Decrypting and displaying this data again, made the system slow down massively Definition and implementation of different caching scenarios (caching on disk, memory etc. / also with Zend Server) Use of the Zend Server JobQueue for long running tasks 5
Performance - Case #2 Our website has a performance problem. Normally the site is performing well except on Wednesdays. We have already invested a lot of effort in finding the problem, but weren t successful so far. 6
Performance - Case #2 Performance Audit -Remote Every Wednesday several web spiders called a certain page of the customer website which showed the whole overview of products 1,100 DB requests per user per page view Result: The 1,100 queries could be reduced to 11 queries doing actually the same (1.5 sec. instead of 15 sec./call). 7
Performance - Case #3 Our webshop is under heavy load in christmas time. We have to make sure that it will perform well at these times. We are connecting to quite a few external web services. Can this be the reason? 8
Performance - Case #3 Performance Audit Remote Result of the audit: The external web services (connected via SOAP interface) were running using a very low bandwidth connection At the same time the amount of transferred data was very large Solution: Many parts of the external web services were reproduced locally. In addition some intelligent caching mechanisms have been implemented Final result: The ecommerce website was working 3-10 times faster than before 9
Performance - Case #4 Our portal for partners/distributors has performance problems. Sometimes a page needs 30-60 sec. to load instead of normally 0.5-4 sec. Our partners are already complaining about this situation 10
Performance - Case #4 Performance Audit Remote Result: The configuration of the Apache web server was suboptimal Solution: Modification of the configuration brought the time to render the page again into normal dimensions Small changes to the database setup saved another 0.3 seconds per call for even better performance Alongside: Various security issues have been found in the code and have been brought to the attention of the customer 11
Architecture & Scalability
Architecture & Scalability - Case #1 We are planning an extended version of our web application. The complexity will be 10 times higher than before. We are not sure whether our existing application and DB architecture is capable to handle the planned changes. 13
Architecture & Scalability - Case #1 Architecture/Scalability Audit on-site Application with a very complex rights management component Obstacle: Can the database handle the expected load? Approach: Analysis of the existing source code Solution: Outlining a new database structure, which can cope with the new application architecture without any problems 14
Architecture & Scalability - Case #2 Our new Web 2.0 intranet is online since 1.5 years. Some pages need more than 10 sec. to load We have built it using Zend Framework, but lately the response times of the site are getting unacceptable. 15
Architecture & Scalability - Case #2 5 days Time & Material Consulting on-site Approach: Use of the profiling features of Zend Studio to localize the problems Solution: Minor changes to the architecture Result: Response time dropped from 10 sec. to under a second without the need to rewrite the whole application Alongside: Together with the customer s developer team several other optimizations have been made to the architecture to ensure sustainability of the code 16
Architecture & Scalability - Case #3 An important internal application needs sometimes 5 minutes to deliver the results. We have many thousand customers, which suffer from this and cannot use our service in a comfortable way. 17
Architecture & Scalability - Case #3 3 days Time & Material Consulting on-site (initially) Guidelines from the customer: The Zend Consultant had only access to the PHP code changes to the database or the architecture were not allowed for delivering a solution. Approach: Analysis on-site followed by additional remote work Result: The performance could be raised by the factor of 4,100 (!). Average performance gain in all application modules: factor 328. 18
Architecture & Scalability - Case #4 We have millions of customers. We have to be technically state of the art to foster additional growth. We have problems with scalability. Our current infrastructure has to be reviewed and updated. 19
Architecture & Scalability - Case #4 5 days Time & Material Consulting on-site (initially) Cause: The very complex application of the customer has been constantly extended based only on customer requests without having an overall plan or vision. The grown application structure has prevented the option for scalability The original database structure has massive problems coping with the current amount of data Solution: Continuous consulting of the customer s development team during the whole time of the development 20
Coaching & Mentoring
Coaching & Mentoring - Case #1 Our development team knows PHP already We have already trained ourselves concerning Zend Framework. We have the impression that our developers are not feeling confident concerning the architecture 22
Coaching & Mentoring - Case #1 3 days Time & Material Consulting on-site Initial situation: Knowledge about Zend Framework was existing, but not much experience concerning the architecture of applications Solution: Staying 3 days on-site, the Zend Consultant defined a robust and scalable structure together with the customer s developer team. This structure is still working until now and was already extended independently by the customer himself 23
Coaching & Mentoring - Case #2 We need a prototype within a short timeframe and it has to be done in PHP and to run on IBM System i. At the same time the PHP prototype has to communicate with our legacy systems to exchange data. 24
Coaching & Mentoring - Case #2 5 days Time & Material Consulting on-site Approach: Together with a developer from the customer, a Zend Consultant designed an application via training on the job Result: The newly created application was able to call existing RPG programs and to exchange data with them Alongside: Afterwards the developer was able to extend the application on his own and to interface to additional legacy data sources in his company infrastructure. 25
Coaching & Mentoring - Case #3 Our application, used by many important customers, is standing at the crossroads concerning the architecture. The application has grown over the years, but we failed on consistently monitoring the architectural guidelines. 26
Coaching & Mentoring - Case #3 3 days training and 3 days Time & Material Consulting on-site 1 st step: On-Site-Training Zend Framework Fundamentals for the development team of the customer 2 nd step: Evaluating the application together with the development team Result: Over 25 recommendations concerning the models, database, documentation, unit testing, Zend Server, error handling, Apache configuration, ACL implementation etc. Alongside: Afterwards the developers were able to extend the application on their own. 27
Coaching & Mentoring - Case #4 We want to build a new application based on the complete Zend stack. We need support for a successful start of the project. The application shall operate with 100,000 users and over 40,000 parallel users. 28
Coaching & Mentoring - Case #4 10 days Time & Material Consulting on-site Solution: Zend Consulting was involved in the project in a very early stage Approach: Implementation of aggressive caching strategies into the application In addition, the system and code integrity was analyzed in the different phases of the project and valuable feedback was given to the development teams. Before the starting the development: Installation and optimized configuration of the Zend Software on all related customer systems by a Zend Consultant 29
Get the maximum out of the Zend software
Get the maximum out of the Zend software We are not sure, whether our installation and configuration of the Zend software makes sense. The time for calling the most complex page got reduced from 1.5 sec. to 180 ms but maybe we can reach an even better performance? 31
Get the maximum out of the Zend software 1 day Time & Material Consulting on-site Approach: Explaining the substantial possibilities of performance optimization with the Zend software on-site by a Zend Consultant Optimizing the configuration of the Zend Software and also the PHP code of the customer to achieve best results Result: The time to display the mentioned page could again be reduced by 50% (180 ms 80 ms). Alongside: The Zend Consultant gave many helpful hints to the customer s development team in how to write well performing PHP code 32
Sometimes it s not PHP
Sometimes it s not PHP - Case #1 Every time, when sending out an Email newsletter to our customers, our website has enormous amounts of visitors Sadly our PHP is breaking down every time when this happens. 34
Sometimes it s not PHP - Case #1 Performance Audit Remote Approach: Performance Audit Result: The customer system was running Windows and the allowed number of incoming connections was limited concerning the ports Solution: The wrong setup of the TCP stack was re-configured and optimized and afterwards the problem was gone 35
Sometimes it s not PHP - Case #2 Our PHP is running too slow! The rendering of the pages in our intranet application is slow. Probably the reason for this is the bad performance of PHP on Windows systems. 36
Sometimes it s not PHP - Case #2 1 day Time & Material Consulting on-site Approach: Profiling of the intranet application Solution: The database in use was not optimally configured for the current purpose 37
Sometimes it s not PHP - Case #3 We have a very strange problem when opening files via PHP We see very strange error messages, which only show up on our production environment and not on our development system 38
Sometimes it s not PHP - Case #3 1 day Time & Material Consulting - Remote Approach: Using the Zend Debugger in connection with Zend Studio Solution: A PHP internal function for working with the file system showed completely different behavior in FreeBSD (production system) than in Linux (development system) although the source code and the PHP version were identical Result: Modifications to the PHP code, because FreeBSD was mandatory for the production system 39
Sometimes it s not PHP - Case #4 Actually our PHP based website is running fine, but with many parallel visitors it happens sometimes that a login is not possible anymore. After a restart of the Apache web server, everything runs fine, but after 30 minutes the same problem occurs again. 40
Sometimes it s not PHP - Case #4 1 day Time & Material Consulting - Remote Reason: Number of Apache processes was limited. In addition the keep alive time was quite high. Result: No new Apache processes could be created Solution: Re-configuration of the web server Alongside: Several security issues have been brought to the attention of the customer. A security audit followed. 41
Overview: Zend Consulting
Advantages of Zend Consulting Unlike others Zend consultants have access to the creators of PHP and thus can bring in a very strong level of knowledge when it comes to PHP and LAMP environments. In 2009 they worked with worldwide over 90 customers in multiple industries with a lot of different application types Zend experts use internally developed methodologies and tools, which are constantly tweaked and optimized during the multiple audits that Zend conducts at customer sites. 43
Performance Audit Selection of topics to be dealt with (depending on the customer situation) Identification of application bottlenecks Audit of the executed PHP code itself Profiling of the scripts / optimization potential of the scripts OS / file system performance Web server configuration Database query analysis / optimization potential Analysis of the load balancing Analysis of used caching techniques 44
Architecture/Scalability Audit Selection of topics to be dealt with (depending on the customer situation) Analysis of the horizontal scalability of the application Analysis of the database scalability Analysis of the session clustering Analysis of the load balancing Does the architecture and configuration of the web server(s) make sense? 45
Security Audit The Security Audit is divided in two phases Pre-Audit (1 day) Black-Box Test, to find some of the very obvious problems Complete Security Audit (several days) Detailed Security Audit incl. optional audit of the source code 46
Security Audit Selection of topics to be dealt with during a complete Security Audit (depending on the customer situation) Penetration Testing Cross Site Request Forgeries Analysis of the PHP config JavaScript Vulnerabilities Output Analysis Denial Of Service Analysis Input Filtering Analysis Header Injection SQL Injection Script Analysis Session Security Cross Site Scripting Vulnerabilities Analysis of the Shell Execution Security 47
Zend Audits Advantages A detailed written REPORT with recommendations is delivered at the end of the audit: By knowing what the issues are, the customer can make informed decisions regarding next steps to address them and ensure that his applications run reliably and securely. After the audit the customer has the freedom of choice to hire Zend to help him fix the found issues or bring in someone else to do this 48
Custom Consulting Based on Time & Material, for example PHP application architecture: Design & Mentoring Zend Framework applications: Design & Mentoring Analysis of scalability and performance problems Support with the integration of Zend products (i.e. Job Queue system design, caching optimizations etc.) Audit services for projects with more than 50K LOC (lines of code) 49
Thank you! For any questions, please contact: axel.schinke@zend.com