Course&updates& Tes=ng& Ariane&5&rocket& Mars&Polar&Lander& Therac[25&radia=on&therapy&machine& 4/3/17& Real%programmers%need%no%tes/ng!

Transcription

1 4/3/17 Courseupdates LiteraturereviewfeedbackonMoodle Allpresenta=onsgraded ProjectplanassignmentdueApril11 Tes=ng Midtermnextweek,April13 We llstartreviewthisthursday,finishnexttuesday HW1,2,and3solu=onsonMoodle Allpresenta=onslidesonclasswebpage Homework4extendedun=lApril20,9AMEDT Real%programmers%need%no%tes/ng! 5)Iwanttogetthisdonefast,tes=ngisgoingtoslowmedown. 4)IstartedprogrammingwhenIwas2.Don tinsultmebytes=ngmy perfectcode! 3)Tes=ngisforincompetentprogrammerswhocannothack. 2)WearenotNotreDamestudents,ourcodeactuallyworks! 1) Mostofthefunc=onsinGraph.java,asimplemented,areoneor twolinefunc=onsthatrelysolelyuponfunc=onsinhashmapor HashSet.Iamassumingthatthesefunc=onsworkperfectly,andthus thereisreallynoneedtotestthem. an#excerpt#from#a#student s#e1mail Ariane5rocket Therocketself[destructed37secondsa\erlaunch Reason:Acontrolso\warebugthatwentundetected Conversionfrom64[bitfloa=ngpointto16[bitsignedintegervaluehad causedanexcep=on Thefloa=ngpointnumberwaslargerthan32767(max16[bitsignedinteger) Efficiencyconsidera=onshadledtothedisablingoftheexcep=onhandler. Programcrashed!rocketcrashed TotalCost:over$1billion Therac[25radia=ontherapymachine Causedexcessiveradia=on,killingpa=entsfrom radia=onpoisoning Whathappened? Updateddesignhadremovedhardwareinterlocksthat preventtheelectron[beamfromopera=nginitshigh[energy mode.nowallthesafetychecksaredoneintheso\ware. Theso\waresetaflagvariablebyincremen=ngit. Occasionallyanarithme=coverflowoccurred,causingthe so\waretobypasssafetychecks. Theequipmentcontroltaskdidnotproperlysynchronize withtheoperatorinterfacetask,sothatracecondi=ons occurrediftheoperatorchangedthesetuptooquickly. Thiswasevidentlymissedduringtes=ng,sinceittooksome prac=cebeforeoperatorswereabletoworkquicklyenough fortheproblemtooccur. MarsPolarLander Sensorsignalfalselyindicatedthatthecra\hadtoucheddown whenitwas130[feetabovethesurface. thedescentenginestoshutdownprematurely Theerrorwastracedtoasinglebadlineofso\warecode. NASAinves=ga=onpanelsblameforthelander sfailure, "arewellknownasdifficultpartsoftheso\ware[engineering process, 1

2 4/3/17 Tes=ngisforeverysystem Examplesshowedpar=cularlycostlyerrors Buteverylikleerroraddsup Insufficientso\waretes=ngcosts $22[60billionperyearintheU.S. [NISTPlanningReport02[3,2002] Ifyourso\wareisworthwri=ng, it sworthwri=ngright Buildingqualityso\ware WhatImpactstheSo\wareQuality? External Correctness Does#it#do#what#it#suppose#to#do?# Reliability Does#it#do#it#accurately#all#the#:me?# Efficiency Does#it#do#with#minimum#use#of#resources?# Integrity Is#it#secure?# Internal Portability Can#I#use#it#under#different#condi:ons?# Maintainability Can#I#fix#it?# Flexibility Can#I#change#it#or#extend#it#or#reuse#it?# QualityAssurance Theprocessofuncoveringproblemsandimprovingthequalityofso\ware. Tes=ngisamajorpartofQA. Thephasesoftes=ng UnitTes=ng Iseachmoduledoeswhatitsupposetodo? Integra=onTes=ng Doyougettheexpectedresultswhentheparts areputtogether? Valida=onTes=ng Doestheprogramsa=sfytherequirements SystemTes=ng Doesitworkwithintheoverallsystem UnitTes=ng Atestisatthelevelofamethod/class/interface Checkthattheimplementa=onmatchesthe specifica=on. Blackboxtes=ng Choosetestdatawithout#lookingatimplementa=on Glassbox(whitebox)tes=ng Choosetestdatawith#knowledgeofimplementa=on Howistes=ngdone? Basicstepsofatest 1)Chooseinputdata/configura=on 2)Definetheexpectedoutcome 3)Runprogram/methodagainsttheinput andrecordtheresults 4)Examineresultsagainsttheexpected outcome What ssohardabouttes=ng? "justtryitandseeifitworks..." int proc1(int x, int y, int z) // requires: 1 <= x,y,z <= 1000 // effects: computes some f(x,y,z) Exhaus=vetes=ngwouldrequire1billionruns! Soundstotallyimprac=cal CouldseehowinputsetsizewouldgetMUCHbigger Keyproblem:choosingtestsuite(setofpar==onsofinputs) Smallenoughtofinishquickly Largeenoughtovalidatetheprogram 2

3 4/3/17 Approach:par==ontheinputspace Inputspaceverylarge,programsmall "behavioristhe same forsetsofinputs Idealtestsuite: Iden=fysetswithsamebehavior Tryoneinputfromeachset Twoproblems 1.No=onofthesamebehaviorissubtle Naiveapproach:execu=onequivalence Bekerapproach:revealingsubdomains 2.Discoveringthesetsrequiresperfectknowledge Useheuris=cstoapproximatecheaply Naiveapproach:execu=onequivalence int abs(int x) { // returns: x < 0 => returns -x // otherwise => returns x if (x < 0) return -x; else return x; All x < 0 are execution equivalent: program takes same sequence of steps for any x < 0 All x >= 0 are execution equivalent Suggests that {-3, 3, for example, is a good test suite Whyexecu=onequivalencedoesn twork Considerthefollowingbuggycode: int abs(int x) { // returns: x < 0 => returns -x // otherwise => returns x if (x < -2) return -x; else return x; {[3,3doesnotrevealtheerror! Two$executions:.!x#<#%2#!!x#>=#%2! Three$behaviors:$..!x#<#%2#(OK)!x#=#%2#or#%1#(bad)!#x#>=#0#(OK)! Revealingsubdomainapproach Same behaviordependsonspecifica=on Saythatprogramhas samebehavior ontwoinputsif 1)givescorrectresultonboth,or 2)givesincorrectresultonboth Subdomainisasubsetofpossibleinputs Subdomainisrevealingforanerror,E,if 1) Eachelementhassamebehavior 2) IfprogramhaserrorE,itisrevealedbytest Trickistodividepossibleinputsintosetsofrevealing subdomainsforvariouserrors Example Forbuggyabs,whatarerevealingsubdomains? int abs(int x) { if (x < -2) return -x; else return x; { 1 { 2 { 2, 1 { 3, 2, 1 { 2, 1 Whichisbest? % Heuris=csfordesigningtestsuites Agoodheuris=cgives: fewsubdomains errorseinsomeclassoferrorse, highprobabilitythatsomesubdomainis revealingfore Differentheuris=cstargetdifferentclassesoferrors Inprac=ce,combinemul=pleheuris=cs 3

4 4/3/17 Black[boxtes=ng Heuris=c:explorealternatepathsthroughspecifica=on theinterfaceisablackbox;internalsarehidden Example int max(int a, int b) // effects: a > b => returns a // a < b => returns b // a = b => returns a 3paths,so3testcases: (4,3)=>4(i.e.,#any#input#in#the#subdomain#a#>#b) (3,4)=>4(i.e.,#any#input#in#the#subdomain#a#<#b)# (3,3)=>3(i.e.,#any#input#in#the#subdomain#a#=#b) Black[boxtes=ng:advantages Processnotinfluencedbycomponentbeing tested Assump=onsembodiedincodenotpropagatedtotest data. Robustwithrespecttochangesinimplementa=on Testdataneednotbechangedwhencodeischanged Allowsforindependenttesters Testersneednotbefamiliarwithcode Amorecomplexexample Writetestcasesbasedonpathsthroughthe specifica=on int find(int[] a, int value) throws Missing // returns: the smallest i such // that a[i] == value // throws: Missing if value not in a[] Twoobvioustests: ([4,5,6],5) =>1 ([4,5,6],7) =>throwmissing HaveIcapturedallthepaths? ( [4, 5, 5], 5 ) => 1 Musthuntformul=plecasesineffectsorrequires Heuris=c:boundarytes=ng Createtestsattheedgesofsubdomains Whydothis? off[by[onebugs forgettohandleemptycontainer overflowerrorsinarithme=c programdoesnothandlealiasingofobjects Smallsubdomainsattheedgesofthe main subdomainshaveahighprobabilityof revealingthesecommonerrors Boundarytes=ng Todefineboundary,mustdefineadjacentpoints Oneapproach: Iden=fybasicopera=onsoninputpoints Twopointsareadjacentifonebasicopera=onaway Apointisisolatedifcan tapplyabasicopera=on Example:listofintegers Basicopera=ons:create,append,remove Adjacentpoints:<[2,3],[2,3,3]>,<[2,3],[2]> Isolatedpoint:[](can tapplyremoveinteger) Pointisonaboundaryifeither Thereexistsanadjacentpointindifferentsubdomain Pointisisolated Commonboundarycases Arithme=c Smallest/largestvalues Zero Objects Null Circular Sameobjectpassedtomul=plearguments (aliasing) 4

5 4/3/17 Boundarycases:arithme=coverflow public int abs(int x) // returns: x Testsforabs whataresomevaluesorrangesofxthatmightbeworthprobing? x#<0(flipssign)orx# #0(returnsunchanged) aroundx#=0(boundarycondi=on) Specific#tests:#say#x#=#11,#0,#1# How#about # int x = ; // this is Integer.MIN_VALUE System.out.println(x<0); // true System.out.println(Math.abs(x)<0); // also true! FromJavadocforMath.abs: NotethatiftheargumentisequaltothevalueofInteger.MIN_VALUE,the mostnega=verepresentableintvalue,theresultisthatsamevalue, whichisnega=ve# Boundarycases:duplicatesandaliases <E> void appendlist(list<e> src, List<E> dest) { // modifies: src, dest // effects: removes all elements of src and // appends them in reverse order to // the end of dest while (src.size()>0) { E elt = src.remove(src.size()-1); dest.add(elt) Whathappensifsrcanddestrefertothesamething? Aliasing(sharedreferences)iso\enforgoken Clear(glass,white)[boxtes=ng Goals: Ensuretestsuitecovers(executes)alloftheprogram Measurequalityoftestsuitewith%coverage Assump=on: Highcoverage! (noerrorsintestoutput!fewmistakesinprogram) Focus:featuresnotdescribedbyspecifica=on Control[flowdetails Performanceop=miza=ons Alternatealgorithmsfordifferentcases Glass[boxmo=va=on Therearesomesubdomainsthatblack[boxtes=ngwon'tcatch: boolean[] primetable = new boolean[cache_size]; boolean isprime(int x) { if (x>cache_size) { for (int i=2; i<x/2; i++) { if (x%i==0) return false; return true; else { return primetable[x]; Importanttransi=onaroundx=CACHE_SIZE Glass[boxtes=ng:advantages Insightintotestcases Whicharelikelytoyieldnewinforma=on Findsanimportantclassofboundaries ConsiderCACHE_SIZEinisPrimeexample Needtochecknumbersoneachsideof CACHE_SIZE CACHE_SIZE-1,CACHE_SIZE,CACHE_SIZE+1 If CACHE_SIZEismutable,wemayneedtotest withdifferentcache_size s Glass[boxchallenges Defini=onofalloftheprogram Whatneedstobecovered? Op=ons: Statementcoverage Decisioncoverage Loopcoverage Condi=on/Decisioncoverage Path[completecoverage 100%coveragenotalwaysreasonabletarget 100% may be unattainable (dead code) High cost to approach the limit increasing number of cases (more or less) 5

6 4/3/17 Regressiontes=ng Wheneveryoufindabug Reproduceit(beforeyoufixit!) Storeinputthatelicitedthatbug Storecorrectoutput Putintotestsuite Then,fixitandverifythefix Whyisthisagoodidea? Helpstopopulatetestsuitewithgoodtests Protectsagainstregressionsthatreintroducebug Ithappenedonce,soitmightagain RulesofTes=ng Firstruleoftes=ng:Do#it#early#and#do#it#o-en# Besttocatchbugssoon,beforetheyhaveachancetohide. Automatetheprocessifyoucan Regressiontes=ngwillsave=me. Secondruleoftes=ng:Be#systema1c Ifyourandomlythrash,bugswillhideinthecornerun=lyou're gone Wri=ngtestsisagoodwaytounderstandthespec Thinkaboutrevealingdomainsandboundarycases Ifthespecisconfusing!writemoretests Speccanbebuggytoo Incorrect,incomplete,ambiguous,andmissingcornercases Whenyoufindabug!fixitfirstandthenwriteatestforit Tes=ngsummary Tes=ngmakers Youneedtoconvinceothersthatmoduleworks Catchproblemsearlier Bugsbecomeobscurebeyondtheunittheyoccurin Don'tconfusevolumewithqualityoftestdata Canloserelevantcasesinmassofirrelevantones Lookforrevealingsubdomains( characteris=c tests ) Choosetestdatatocover Specifica=on(blackboxtes=ng) Code(glassboxtes=ng) Tes=ngcan'tgenerallyproveabsenceofbugs Butitcanincreasequalityandconfidence 6