Cloud-Based Information Systems Management Rising to the Challenge of Truly Distributed Computing

Transcription

1 Clud-Based Infrmatin Systems Management Rising t the Challenge f Truly Distributed Cmputing By Richard Levy The migratin f infrmatin systems frm npremise t the Clud is well under way. Fueled by the grwth f services such as SalesFrce.cm, Amazn s Web Services and new platfrms like Micrsft s Windws Azure, clud cmputing is a paradigm that entails assembling systems whlly frm remtely accessed, third-party prviders f applicatin servers, databases, hardware, and s frth. The advantages f clud cmputing are many, including giving system develpers the ability t deply slutins rapidly withut the need t acquire IT resurces, hardware, sftware, netwrk infrastructure, and datacenter capacity. Instead, all f these essential cmpnents f system develpment and delivery can be rented and accessed as demand dictates. At the same time, clud cmputing presents sme significant infrmatin systems management challenges. Frm the perspective f serius prductin systems, such as thse used in critical applicatins by large crpratins r gvernment agencies, clud cmputing must address a number f security and perfrmance cncerns befre it can be readily embraced as an alternative t traditinal npremises deplyment. This paper will lk at sme f these challenges and explre emerging best practices that can take sme f the guesswrk and stress ut f mving t the clud. Security fr the clud-based system Data security pses the biggest challenge t the clud-based cmputing mdel. The amunt f skepticism and parania that exists in the industry tday arund clud applicatins and their ability t prtect yur data is unfathmable. Just think abut clud applicatins cmpared t the traditinal npremise applicatin mdel, where the data is stred behind yur firewall n yur servers in yur secure facility. Even large multi-billin-dllar rganizatins with large IT departments struggle t btain and maintain the necessary talent t build a secure IT infrastructure. Yu need t have DBAs, netwrk engineers, and system administratrs fr the different perating systems yu supprt. In additin, yu need security experts t help advise yu n best practices and plicies fr adhering t industry standards such as SAS 70, Sarbanes Oxley, PCI, HIPAA, etc. On the netwrk side alne yu have multiple technlgies t understand and t implement such as firewalls, prxy servers, DNS servers, IDS systems, Single Sign On security servers, etc. t select, install and manage. Finding and maintaining experienced technical staff that can understand yur crprate needs is a challenge. Then yu must research and identify the best prducts t address these security requirements, install and cnfigure them and supprt them in prductin peratins. These are nt easy tasks. What percentage f tday s large crpratins can bast a SAS 70 Type II certified facility? Cmpare this t the percentage f cludbased systems that can prvide a SAS 70 Type II audit reprt n demand. The majrity f tday s clud-based prducts are hsted by prfessinal third-party hsting service prviders. The applicatin vendrs have realized that their skills are in applicatin develpment and nt in prviding hsting services. The marriage f these tw talent pls prvides a pwerful end result which, in the majrity f cases, will prvide a higher degree f data security than the typical internal crprate systems team can deliver. This same argument applies t all ther areas f security in the hsting facility. Databaselevel security can be implemented in several different ways. Yu can use firewall rules t limit access t the database server prts; yu can use database-level security t restrict access; and yu can use data encryptin ptins prvided by the database vendrs r by third-party vendrs. Experienced DBAs may have dne this in the past s again yu

2 are limited by yur talent pl. The same issues apply t applicatin servers, perating systems. etc.; they all prvide a multitude f security ptins and cnfiguratins, s yu end up being as secure as yur talent pl allws. It makes sense t cnsider the fact that a Clud applicatin vendr will live r die by hw secure their custmers data is, and thus will find the best hsting partner t wrk with and ensure their success. One data breach and it s game ver fr a clud-based prduct. Mnitring the clud-based system It is extremely imprtant t design and implement a cmprehensive set f mnitrs and a system health dashbard which allw the system management and supprt teams t easily identify issues and reslve them. The mnitrs shuld span the deplyment architecture frm the hardware layer all the way up thrugh the applicatin layer. There are several prducts available t implement a mnitring framewrk. A mnitring framewrk directly accesses system and applicatin infrmatin including lgs, etc., t reprt status and t send alerts based n predefined threshlds. System Mnitrs: Mnitrs shuld be set up t ensure the server is running ptimally. Checking the CPU utilizatin, disk I/O, disk space available, RAM usage, page swapping, and ther perfrmance data is key t mnitring the health f the server. Best practices include setting threshlds which d nt cause alert verlads but instead highlight ptential issues befre they impact the applicatin perfrmance r causes the applicatin t crash. Ensuring that warnings are delivered in time fr the peratin team t make the necessary adjustments befre service level agreements (SLAs) are impacted is a key factr t the success f the applicatin. Fr example, yu can set up an alert n CPU utilizatin when it hits 90%. This culd be a prblem since yu may have spikes t 90% frequently during standard peratins s, in ur example, t ensure the alert is nly generated when there is a ptential issue, yu shuld cnfigure the mnitr t send the alert nly if it finds that the 90% threshld has been exceeded acrss 3 mnitr intervals. Mnitr intervals shuld be set t a value which des nt cause excessive resurce usage n the server s we can chse a 3-secnd interval. In this scenari the alert wuld nly g ut if the CPU remained at ver 90% fr 9 secnds. Applicatin Mnitrs: Mnitrs shuld be set up t ensure that the applicatin tier is running and is perfrming in line with the SLAs. The nature f these mnitrs is typically determined by the applicatin architecture. A typical clud applicatin has a three-tiered architecture, brwser frnt end, web applicatin server middle tier and a database as the backend tier. T ensure the SLAs are being met yu wuld need mnitrs that can drive applicatin transactins and thus can measure the user respnse time. Other applicatin mnitrs shuld be tracking user authenticatin failures and ther applicatinlevel cmpnents which need mnitring. Similar functinality shuld be set up fr these applicatin-level mnitrs as described fr system mnitrs. If the transactin turnarund exceeds the predefined threshhlds fr 3 successive attempts, fire the alerts. Mnitr the URL s that if it is nt accessible (meaning the site is dwn) fire the alerts immediately, etc. Database Server Mnitrs: Mnitring the database server t ensure it is running and is perfrming ptimally is imprtant. Adding mnitrs that track number f pen cnnectins, memry usage, disk space available etc. will help t identify the pint f failure sner. Yu can als mnitr specific database server features like Oracle 10G / 11G advisry infrmatin, which gather statistics and prvide recmmendatins n SGA and PGA memry settings fr ptimal perfrmance etc. In general when yu lg in t the system mnitr prtal yu shuld see a simple graphical view f the facility with a few basic clrs t indicate and highlight prblems.

3 Typically red wuld indicate an issue, green wuld mean health and yu can chse ther clrs t indicate that a cmpnent is mving tward a red status r that a cmpnent is under maintenance r Return T Service (RTS) activities, etc. N mnitring service is cmplete withut a cmplete set f autmated and clearly dcumented RTS prcedures in place t supprt them. Waiting fr the peratrs t determine the applicatin is dwn, and then t lg in and determine what t d t bring the applicatin back nline, is nt an effective slutin and can have a serius impact n the system SLAs. Autmated RTS prcedures shuld be very tightly cupled t the mnitrs. Fr example, if the mnitr shws that the applicatin URL is dwn (which means the applicatin is nt accessible) then what shuld the autmated RTS prcedure be t bring it back nline? It culd be t simply redeply the applicatin if the status shws stpped, r it culd be t ping the applicatin server cnsle URL and see if the applicatin server is up and, if nt, call a system script t restart the applicatin server which wuld autmatically bring the applicatin back nline. Even lwer level is t check the hardware; if the server has crashed then the RTS prcedures shuld be t autmatically rebt, and yu shuld include the applicatin server startup scripts in the server s initializatin prcess. Perfrmance and Service Level Agreements (SLAs) System perfrmance and managing t yur SLAs is anther area f cmplexity which requires highly skilled resurces t accmplish. System perfrmance and stability has a majr impact n SLAs, s ensuring that the right system resurces and perfrmance experts are in place t meet yur SLAs is a key factr t yur success. Let s cnsider what it takes t ensure that a simple three-tiered Java applicatin is perfrming ptimally and can cmply with yur SLAs under maximum lad. The factrs that play a rle in the perfrmance f this standard applicatin architecture include: Hardware (servers, perating systems, etc.) Netwrks (bandwidth, DNS servers, prxy servers, lad balancers, switches, etc.) Applicatin server (DB cnnectin pl settings, thread settings, JVM ptins, etc.) Database server (disk cnfiguratin, memry, blck sizes, SQL ptimizatin, etc.) Web server (cnfiguratin, plug-in settings, lad balancer / failver settings, etc.) T ensure ptimal applicatin perfrmance yu need t rely n perfrmance experts t set all f this hardware and sftware ptimally accrding t the applicatin characteristics. Mst applicatins have very unique perfrmance characteristics s yu cannt simple rely n a generic perfrmance prfile t ensure the ptimal perfrmance f the applicatin. Sme applicatins have many small simple transactins; thers have mre cmplex transactins with large data vlumes. These different characteristics require yu t cnfigure the systems differently in rder t ensure ptimal applicatin perfrmance. The recmmended apprach t validating the system cnfiguratin t ensure it will allw yu t meet yur SLAs is using standard lad testing sftware t simulate the usage scenaris and expected lad. The steps fr this task are: Identify a lad testing tl such as HP Lad Runner, JMeter, etc. Script the usage scenaris. Determine what the percentage distributin acrss these usage scenaris shuld be. Determine a ramp time and valid think times. Run the lad test. Once yu have established that the system cnfiguratin will satisfy yur perfrmance and will prvide up-time necessary t meet the SLAs, yu need t build the SLA tracking services. This task is nt simple and is tightly cupled with t the applicatin architecture

4 and the system cnfiguratin that is supprting it. Tracking and reprting n system up-time is nt cmplex. Yu must track all scheduled maintenance utages and all unscheduled utages which ccur within the facility and the wide area netwrk. Tracking perfrmance-based SLAs is a mre cmplex task and requires mnitrs that track the end user experience and respnse time. Finding a technically savvy system administratr t determine hw t measure this, based n the applicatin and system architecture, is nt easy. Failver and Disaster Recvery: Determining the best High Availability (HA) slutin fr the applicatin requires the Systems team t understand the applicatin architecture, SLAs, and the brad range f HA ptins available. Let s assume the applicatin is a three-tiered Enterprise Java applicatin and explre sme f these ptins, first fr the primary site and then fr the case where the SLAs demand a secndary site. Primary site failver can be accmplished via hardware-based r sftware-based slutins r, in sme cases, yu may chse a cmbinatin f bth. First yu need t determine if the applicatin architecture can supprt HA. Can yu simply cnfigure tw standalne applicatin servers and use a lad balancer t distribute the lad, r d yu require applicatin server clustering and the use f multi-casting r RMI t synchrnize sessin data? Once yu understand the applicatin architecture then yu can chse the crrect applicatin server n which t deply the applicatin. The next step is t determine what lad balancer yu want t use, hw yu want t set up the lad balancer s as t distribute the wrklad acrss the cluster, and hw t handle failver cnditins. Chsing a hardware-level lad balancer such BIG-IP s F5, Citrix s NetScaler, etc., is the first hurdle. D yu have the expertise t cnfigure and supprt this device? Alternately yu culd set up a web server like Micrsft IIS, Apache, r IBM HTTP Server with an applicatin server plug-in and cnfigure it t distribute the lad acrss the backend applicatin servers. Either way, fr HA purpses, yu will have t set up at least tw f whichever ptin yu chse t ensure HA. Nw yu start t see the csts assciated with this basic setup fr HA in a single site. Yu have the cst f purchasing the dual sets f hardware and sftware, plus the csts f cnfiguring them fr failver t ensure that yu dn t have any single pint f failure in the facility and the applicatin. Again, we are back t the same issue: D yu have the right resurces t make the best decisin fr yur cmpany and yur applicatin architecture? These resurces are expensive and retaining them is expensive. If the SLAs require that yu have a Disaster Recvery (DR) plan in place then yu must have access t a secndary site in anther gegraphic lcatin. Yu can chse t d this n yur wn using a branch ffice and setting up a secndary facility, r yu can find a partner that ffers a hsting facility and rent space plus hardware, etc., frm them. Building ut a DR site als pens up many ptins, each f which cmes with its wn assciated csts. Yu need t answer sme basic questins such as, hw lng can the applicatin be ff line fr in the case f a disaster, and can yu sustain a data lss? If the business requires zer dwntime and cannt tlerate any data lss, then the DR requirements will be a lt mre cmplicated. Yu will require a ht failver site with a live clustered applicatin server, database replicatin t make sure the database is in sync at all times, and glbal DNS t ensure that if ne site went dwn the ther wuld seamlessly take ver the traffic. Questins that cme up include, d yu require sessinlevel failver? This is an expensive ptin but ensures that the user s request will be prcessed even in the event that ne site failed while they were executing n it; this requires that the user s sessin cntext be shared acrss the applicatin server cluster. If this is nt a requirement then yu can g with less expensive ptins where the user receives an errr message and has t lg back int the applicatin.

5 All f these requirements need t be flushed ut. Yu need a team with the necessary experience t first ask the right questins, then t design and implement the facilities necessary t supprt these requirements. In additin, yu need all f the hardware and sftware needed t supprt these requirements. Maintaining the Clud Applicatin The typical clud-based applicatins are designed t supprt multi-tenant usage, which means the applicatin tier will be shared by many clients. This can be achieved in different ways. Yu culd use a single applicatin URL with the cmpany name n the end f the user ID (e.g. userid@cmpanyname.cm ) t identify the client and then, by string the client ID with each rw f data in the database tables, yu can segregate the data. Anther ptin is t present a different URL t each client and create a separate database schema per client, ttally segregating the data but still allwing the applicatin tier t be shared. Whichever way yu chse t develp the applicatin yu still face the same challenges f applying applicatin maintenance, adding client unique custmizatins t the applicatin, prviding a develpment and test instance fr these custmizatins, prviding the end users with acceptance test envirnments prir t a prductin upgrade, and s frth. These challenges with maintaining cludbased applicatin are cmplex t slve in such a way that yu dn t verburden the IT staff that are supprting and maintaining the applicatin. Yu als have t take int accunt that maintenance windws may be small due t different user time znes; they may nly be available at 11:00 pm 2:00 am, s yu need t have a team t supprt these activities. Thus applying patches and updates with clear rllback prcedures is crucial t the success f these applicatins. This requires well dcumented change cntrl prcedures and clear audit trails t ensure that maintenance changes are traceable. Thus if there is a prblem with a custmer after the maintenance is applied the infrmatin is readily available t the supprt rganizatin. Allwing clients t test their applicatin and unique custmizatins in a test instance prir t the upgrade f the prductin instance is a gd practice. This requires that yu set up a test instance by making a cpy f the prductin database and upgrade this test instance t the latest cde updates, then ntify custmers that they have, fr example, a ne-mnth windw t test and prvide them with the test URL and lgin. Having a 24/7 supprt team with the necessary skills n hand t prvide these services is crucial t the success f all cludbased applicatins. This team needs t cnsist f team members wh can quickly identify and reslve issues at all levels f the applicatin stack. Building and maintaining an applicatin supprt team with these 24/7 capabilities is cstly. Chsing the right partner fr clud cmputing Betting yur cmpany s business and future n the Clud is a decisin that is being made nt just by applicatin vendrs but als by large and small enterprises tday. I culd cntinue t pint ut the cmplexities invlved in develping, deplying and supprting a clud-based applicatin. I believe the pattern will be cnsistent with the pattern that is apparent frm the areas highlighted abve. T be clear, the cst f setting up the facilities t hst the applicatin and then t supprt and manage the applicatin t its SLAs will be a much bigger challenge than the develpment f the clud-based applicatin. Thus, as a clud applicatin vendr, finding a hsting services prvider t partner with and share the risk will be a key factr in yur success. There are many prviders ut there wh all prvide similar services, s yu need t create an RFP and submit it t thse hsting service prviders wh have the facilities and expertise t supprt yur applicatin stack.

6 Remember t be very clear abut what yur applicatin stack is and t determine if the service prvider has the necessary expertise in key staff rles t supprt the applicatin stack 24/7: Hardware netwrk knwledge DBAs Netwrk administratrs Security experts Unix / Windws Server administratrs Applicatin server experts Richard Levy Richard.levy@mitratech.cm Mitratech Hldings, Inc Wilshire Bulevard, Suite 300 Ls Angeles, CA While yu, the applicatin vendr, are respnsible fr prviding applicatin level supprt, and need t create a 24/7 applicatin supprt team, yu can cnsider training yur new partner (the hsting service prvider) t take n this supprt rle as well. Abut the Authr: Richard Levy serves as Mitratech s Chief Technlgy Officer, verseeing the architecture and develpment f the cmpany s prducts and applicatin platfrm. Richard s respnsibilities include building and managing a distributed 24/7 engineering rganizatin, as well as leveraging leading-edge Web 2.0 technlgies t enhance and extend the functinality f the existing prducts and applicatin platfrm, and creating the next generatin clud-based develpment platfrm fr cllabrative accuntability applicatins. In additin Richard is driving the develpment f an autmated end-t-end Sftware Develpment Life Cycle management prcess t allw adpters f the clud-based develpment platfrm t manage all aspects f their develpment prcess nline. Prir t jining Mitratech, Richard spent fur years at SeeBeynd Technlgies/SUN Micrsystems as Vice President f Engineering, where he managed a crsscntinent team f engineers wrking n a suite f Enterprise Applicatin Integratin prducts, including a full range f adapters, an Integratin Brker, Enterprise Service Bus, Business Prcess Manager and a Business 2 Business Exchange Server. Richard was als respnsible fr establishing an Offshre Develpment Center with ver 200 engineers. Richard has als wrked as Vice President f Technical Operatins fr Infrmatin Builders Inc./Iway Sftware