Measuring cloud-based anti-malware protection for Office 365 user accounts

Measuring cloud-based anti-malware protection for Office 365 user accounts Ferenc Leitold Veszprog fleitold@veszprog.hu Anthony Arrott CheckVir aarrott@checkvir.com William Kam Trend Micro william_kam@trendmicro.com Abstract Microsoft Office 365 user accounts were tested for the efficacy of anti-malware protection provided as part of the cloud-based components of Office 365 productivity software-as-aservice: Exchange, OneDrive, and SharePoint. Multiple threat types (malware binaries, infected documents, malicious hyperlinks) were applied through multiple attack vectors (e-mail, file transfers, social media posts). Cloud-based third party enhanced anti-malware protection is compared to the cloudbased self-protection provided by Microsoft Office 365. While the cloud-based user account self-protection provides protection comparable to endpoint-based anti-malware, the 3rd party protection is shown to provide significantly enhanced protection for file transfers to cloud-drives and for malicious links for all attack vectors. Microsoft has incorporated cloud-based anti-malware protection into the basic design of Office 365 using multiple filters, anti-virus engines, and sandboxes (Figure 2) [6]. Analogous to 3rd party endpoint anti-malware protection for Windows, Trend Micro s Cloud App Security (CAS) product provides additional internet security as a cloud-based add-on to Microsoft Office 365 (Figure 3) [7]. Index Terms anti-malware product testing, Office 365 user accounts, Office 365 security, cloud-based anti-malware protection, complete end-user protection. I. INTRODUCTION I N contrast to endpoint-based Microsoft Office, users of cloud-based Office 365 services exchange email, files, and social media posts through intermediary cloud-based services including Exchange, OneDrive, and SharePoint. For defending against malicious attacks, Office 365 affords more opportunities for scrutiny and detection before malware reaches the user endpoint (Figure 1). Fig. 2. Cloud-based anti-malware self-protection of Office 365 user accounts provided by Microsoft. Fig. 3. Cloud-based anti-malware 3rd party protection of Office 365 user accounts provided by Trend Micro Cloud App Security. Fig. 1. Microsoft Office 365 user account vulnerability to malware attack. This paper presents the results of a first attempt to adapt the testing methods and metrics for endpoint-based protection to measure the additional protection a 3rd party anti-malware

product provides to Office 365 user accounts above and beyond the self-protection provided by Microsoft Office 365 itself. II. EXPERIMENT DESIGN A. Systems under Test (SUTs) Two systems were tested: A Microsoft Office 365 user account (E3) with its anti-malware self-protection fully enabled ( user account self-protection ) and a Microsoft Office 365 user account (E3) protected with both Microsoft self-protection and 3 rd party enhanced anti-malware protection from Trend Micro s Cloud App Security ( 3 rd party cloud antimalware ). Test procedures followed the methodology of the Veszprog Endpoint Test Battery [4], modified to accommodate testing the user account of SaaS productivity products. User accounts of SaaS applications operate across several endpoint devices (e.g., desktop, laptop, tablet, smartphone) whereas endpoint-based anti-malware applications protect specific instances of endpoint operating systems on individual devices. The testing methods employed are consistent with AMTSO standards and correspond to other established industry practices for testing endpoint-based antimalware products [5]. B. Stimulus and Steady State Workloads Three high-level types of attacks are considered in measuring the protection of Office 365 user accounts: (A) malware binaries; (B) infected documents; and (C) malicious hyperlinks (Figure 4). Fig. 4. High level distinction of separate attack types on Office 365 user accounts: (A) malware binaries; (B) infected (weaponized) documents; and (C) malicious hyperlinks. Three high-level attack vectors are considered in measuring the protection of Office 365 user accounts: (1) email; (2) file transfers; and (3) social media posts (Figure 5). Attack samples (both malware binaries and malicious links) used in the tests are obtained from the AMTSO Real-Time Threat List (RTTL) service [1]. The attack samples were further prepared by the testers for application through the tested attack vectors (e.g., email attachment, social media post). After testing is completed, malware binaries are analyzed and categorized as belonging to one or more of four malware binary threat types: (i) Trojans, (ii) adware, (iii) ransomware, and (iv) APT components. Malicious links are processed and analyzed separately. Fig. 5. High level distinction of separate attack vectors on Office 365 user accounts: (1) email; (2) file transfer; and (3) social media post. For the attacker, there are several options when combining each high-level threat type (A, B, C) with each high-level attack vector (1, 2, 3). For example, when using a malicious binary (threat type A) to attack through SharePoint (attack vector 3), the attacker can upload to: Document Library Form Library Wiki Page Library Picture Library Asset Library Data Connection Library Report Library For these high-level tests, only one file transfer option was tested for each threat type and attack vector. More comprehensive testing would be required to assess differences among the available options. In order to further simplify the experiment, the stimulus workload was reduced to six combinations of threat type and attack vector. Two primary threat types were used: (i) malicious binaries as stand-alone files and (ii) malicious links embedded in text files. Both types were used to attack the Office 365 user account through each of the three SaaS services: Exchange, One-Drive, and SharePoint (Figure 6). C. Metrics For all threat types and all attack vectors. protection efficacy was measured for each attack sample applied to each applicable attack vector as: FAILURE if attack sample was available to the Office 365 user account logged in at a Windows 7 endpoint (prior to

the application of local endpoint anti-malware protection). SUCCESS if attack sample was not available to the Office 365 user account logged in at a Windows 7 endpoint (prior to the application of local endpoint anti-malware protection). Fig. 6. Reduction of high level threat types and attack vectors to simplify current test. Two primary threat types were used: (i) malicious binaries as stand-alone files and (ii) malicious links embedded in text files. Unlike traditional PC endpoints, Office 365 user accounts provide cyber-attackers multiple paths to infect personal user infrastructure (e.g. shared cloud file storage services vs. PC physical hard drive). An appropriate metric for overall protection efficacy is thus to consider the blocking rate in a swarm attack. In a swarm attack, the attacker attempts to infect the target using all available attack vectors. We therefore define the overall blocking rate ( swarm attack ) as the number of attempts blocked at each available attack vector divided by the total number of attempts using all threat types through all attack vectors. Tests using malware binaries were conducted in 12 tranches in December 2016 and January 2017. Each tranche consisted of 200 malware binaries and 800 benignware binaries. Following the tests, the malware binaries were sorted by threat type using security intelligence metadata, service vendor logs, and by examination. Four threat type categories were used for the malware binaries: (i) Trojans, (ii) adware, (iii) ransomware, and (iv) APT components. A single malware binary could be tagged in more than one category (e.g., Trojan and ransomware). Many systems for malware categorization exist [9]. For our limited purposes here, we adopted our own heuristic categorization corresponding to current popular interest in what the malware is used to accomplish. Tests using malicious links were conducted separately under similar conditions earlier in 2016. Each malicious link tested consisted of a link to a file location containing a malicious binary (e.g., http://bad.com/bad.exe). The malware binary associated with each malicious was extracted and applied to a battery of endpoint-based anti-malware products more than two weeks after testing the malicious link with the Office 365 user accounts. The endpoint products used included Bitdefender, F-Secure, Kaspersky, Symantec, and Sophos. The results were used to restrict the attack sample set to malware samples (n=140) that met the conditions for consensus malware as defined by Colon Osorio et al [3]. From the full set of malicious links tested, n=140 were chosen for which both protections provided 100% blockage in the case of threat type A (malware binaries) and attack vector 2 (file transfer to OneDrive). All 140 malware attack samples were deemed consensus malware by the Colon Osorio criteria using endpoint-based anti-malware. This was done to establish a reference for how the malicious links were handled in each of the attack vectors tested (link embedded in email attachment, transferred file, or social media post). III. RESULTS A. False positives For all tests of benignware samples in each attack vector (email attachment, file transfer, social media post), the root mean square variation of false positive detections is less than one percent among the two test articles (user account selfprotection and 3 rd party cloud anti-malware) and the reference endpoint-based anti-malware. From this we conclude that false positive rate is not a distinguishing difference among any of the anti-malware protections tested. B. Office 365 user account self-protection Office 365 user account self-protection (Microsoft Office 365 Security) [6] provides comparable anti-malware protection for some of the tested threat type / attack vector combinations (in general, against malicious binaries) while providing no measurable protection for others (in general, against malicious links) (Table 1). Cloud App Self-Protection 3 rd party Cloud Antimalware Endpoint Device attack vector cloud social web cloud social web threat type drive media mail drive media mail (ref) Trojans 84% 99% 100% 99% 100% 100% 100% Adware 91% 93% 92% 98% 99% 100% 100% Malicious Links 0% 0% 0% 93% 93% 88% 93% Ransomware 87% 100% 98% 100% 100% 98% 85% APT chain kill 90% 96% 90% 100% 100% 90% 61% max 91% 100% 100% 100% 100% 100% 100% median 87% 96% 92% 99% 99% 92% 93% min 0% 0% 0% 93% 93% 88% 61% Table 1. Percent of malware attacks blocked from 5 threat types using 3 attack vectors for both self-protected and 3 rd party protected Office 365 user accounts. Protection from direct attacks on a Windows PC protected by traditional endpoint-based anti-malware is shown as reference. Compared to the reference endpoint-based anti-malware ( endpoint device ), user account self-protection is comparable for Trojans attacking through social media or email and for ransomware attacking through cloud drive file transfer. User account self-protection is significantly better

than endpoint-based protection for ransomware attacking though social media or cloud drive file transfer and for blocking APT components through any of the tested attack vectors. User account self-protection shows no ability to protect against malicious links for any of the tested attack vectors. malicious links embedded in text files. C. 3rd party cloud anti-malware protection 3rd party cloud anti-malware (Trend Micro CAS) provides measurably enhanced protection for 11 of 15 threat-type and attack vector combinations (Table 1). 3rd party protection is comparable for 4 of the 15 combinations (Trojans and ransomware attacking through social media and web mail; and APT segments utilizing email). Overall, median protection for all five threat types is significantly enhanced for the cloud drive attack vector; moderately enhanced for the social media attack vector; and comparable for the web mail attack (Figure 7). Fig. 8. Anti-malware protection from attacks by recently identified ransomware. 3rd party anti-malware protects significantly better for files written directly to cloud drives a favorite vector for ransomware lateral movement in organizations. Protection against all attack vectors is comparable or significantly better for user accounts than protection provided by traditional endpoint-based antimalware. Fig. 7. Median protection from attacks by 5 threat types using 3 attack vectors for both self-protected and 3rd party protected Office 365 user accounts. Protection from direct attacks on a Windows PC protected by traditional endpoint-based antimalware is shown as reference. For the cloud drive attack vector (file transfers directly to the OneDrive cloud drive), 3rd party anti-malware is consistently effective (99% for malicious binaries; 93% for malicious links embedded in text files). 3rd party anti-malware blocked 100% of ransomware attacks through the cloud drive compared to 87% for user account self-protection and 85% for endpoint-based anti-malware (Figure 8). For detecting malicious links embedded in text files, 3rd party anti-malware is comparable to the effectiveness of endpoint-based anti-malware (Figure 9). Office 365 user account self-protection is unable to provide protection against Fig. 9. Anti-malware protection from attacks by recently identified malicious links. User account self-protection is nonexistent while 3rd party cloud-based anti-malware protection is comparable to protection provided by traditional endpointbased anti-malware.

D. Swarm attack through all available attack vectors A goal in these tests is to make an initial measurement of the overall protection provided by cloud-based anti-malware protecting cloud-based Office 365 user accounts. An appropriate metric for assessing overall protection efficacy is protection against a swarm attack through all the tested threat types and attack vectors. This is the case when attackers attempt to deliver malware through all available attack vectors simultaneously. Summary result for blocking rate of all malicious items through all attack vectors. Results indicate that 3 rd party antimalware provide significantly enhanced protection over the self-protection of Office 365 user accounts (Figure 10). 3 rd party anti-malware enhancement provided by Trend Micro CAS is due primarily to two factors: (1) the better 3 rd party detection of malware binaries that are transferred directly to the cloud drive (Trojans, ransomware, and APT components); and (2) the inability of Office 365 user account self-protection against malicious links delivered by any of the three attack vectors. 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% Protection against Swarm Attack 97% 0% 75% user account self-protection Fig. 10. Average anti-malware protection against swarm attacks by recently identified malware (i.e., attacks using all 5 threat types through all 3 attack vectors). 3 rd party cloudbased anti-malware provides significantly better protection compared to user account self-protection. IV. DISCUSSION 3rd party anti-malware n = 2540 unique attacks in each vector A. Measuring protection for cloud app user accounts The measurements presented here utilize methods wellestablished for traditional endpoint-based anti-malware. In endpoint-based anti-malware, the system-under-test is the operating system (e.g., Windows 7). In contrast, here the system-under-test is the cloud application user account (Office 365 E3). The adapted methods successfully provide visibility into the effectiveness of anti-malware protection applied before malware is available to any of the multiple potential endpoints that can be connected to the cloud-based user account. This facilitates direct comparisons of the effectiveness of anti-malware provided by user account selfprotection (Microsoft Office 365 Security) with both traditional 3 rd party protection at the endpoint and with 3 rd party anti-malware applied to the cloud application itself (Trend Micro CAS protection of Office 365 user accounts). B. More complete end-user protection Users of Office 365 conduct information transactions in the office, at home, on the road, or anywhere in between. They are not always working on a protected network, device, or application. Effective security needs to follow users wherever they go: different devices on different networks. When multiple anti-malware techniques are used to stop malware attacks, the multi-layer protection needs to share threat intelligence across the different layers, networks, and endpoints. Cloud-based antimalware applied to cloud app user accounts can consolidate the latest threat knowledge for application across all threat vectors regardless of the user s transient connectivity or configuration environment. The single-most significant improvement that both cloudbased Microsoft Office 365 security and Trend Micro 3rd party cloud-based Cloud App Security offer over traditional end-point-based anti-malware is the detailed examination of files before they are stored inside the user environment. Both apply multiple anti-malware filters before a file is stored in cloud-based OneDrive or equivalently for Exchange and SharePoint. Where the 3 rd party protection demonstrates greatest enhancement over the self-protection of Office 365 is when malware is delivered as a malicious link (e.g., hyperlink in email of the form abc.com/xyz.exe) rather than as a direct malware executable (e.g., malware binary email attachment of the form xyz.exe). 3 rd party anti-malware protection is also enhanced by better detection of malware binaries written directly to the cloud drive (OneDrive). One of the innovative features of Trend Micro CAS is its incorporation of a fast-responding sandbox that is a key element in Trend Micro s enterprise breach detection product, Deep Discovery. The sandbox service, Virtual Analyzer [8], is faster and more thorough than the sandbox service provided by Microsoft Office 365 Security ( detonation chamber shown in figure 2). A comparative analysis of cloud-based sandboxing vs. other anti-malware techniques is reported in a separate study [2]. C. Lateral movement visibility and protection The traditional weakness of network gateway security is that it provides little protection against lateral movement of malware among endpoints within an enterprise network. The method described here adapts traditional endpoint-based antimalware testing techniques to cloud-based user accounts. This facilitates the direct measurement of vulnerabilities to attacks

from other user accounts in the same enterprise network. Such attacks are not visible to protections at the network gateway. Nor are measurements at the endpoint able to easily distinguish between attacks from outside or inside the enterprise. Lateral movement vulnerability detection is particularly relevant for protection against advanced persistent threats (APTs). Also, cloud applications that synchronize shared drives are a favorite ready-made lateral movement mechanism in ransomware attacks. D. Broader perspective: What is not measured The improvements offered by cloud-based anti-malware protection contribute significantly to protecting Office 365 user accounts in ways more effective than traditional endpoint-based anti-malware. This is achieved primarily by interrupting attack vectors prior to malware or malicious links ever reaching endpoint devices. For document-sharing and collaboration among information workers in the same Office 365 environment, this is particularly valuable. However, it is important to place these protections in the overall context of complete end-point and user account protection. Elements of Complete End-User Protection platform description protection capabilities Central Management Manage threat and data - central management protection across the - user-centric visibility enterprise. Endpoint Security Secure physical and virtual - anti-malware endpoints using the broadest - advanced threat protection range of threat and data - data protection protection techniques across - vulnerability protection all devices and applications. - application control - web filtering - desktop virtualization Mobile Security Secure, track, monitor, and manage enterprise employee's mobile devices and company data. Secure real-time collaboration and stop targeted attacks, spam, phishing, viruses, spyware, and inappropriate content from impacting the enterprise. Safeguards the web gateway from web threats. Email and Collaboration Security Secure Web Gateway - mobile device management - data protection - Office 365 security - email gateway protection - email server protection - collaboration portal protection - instant message security - hosted email security - data protection - anti-malware - advanced threat protection - URL filtering - application control Table 2. Cloud-based anti-malware for SaaS user accounts (such as Office 365) is only one element in the total protection of SaaS end-users. The enhanced cloud-based storage, email, and collaboration security provided by the products and services tested in this paper are only part of a comprehensive overall approach to complete end-user protection. Just as end-user-based antimalware is an incomplete solution to end-user anti-malware protection, so too, cloud-based anti-malware protection of Office 365 user accounts is, by itself, an incomplete solution to protecting Office 365 user accounts. A more complete view is presented in Table 2. V. CONCLUSIONS Compared to network gateways and endpoint PCs, user accounts for cloud-based applications provide malware attackers with more attack vectors for infecting and exploiting users. At the same time, cloud-based services provide defenders with systematic collaborative means for interrupting malware attacks on user accounts across multiple parallel attack vectors. For Office 365, component services within the application (e.g., Exchange, OneDrive, SharePoint) facilitate self-protection against malware attack. Self-protection for Office 365 user accounts is comparable to endpoint antimalware protection for PC users better for some threat types (e.g., ransomware) and worse for others (e.g., malicious links). Third-party cybersecurity protection for cloud-based applications can further leverage the cloud to provide enhanced anti-malware protection to user accounts. In the case of Office 365 user accounts, Trend Micro Cloud Application Security (CAS) provides anti-malware protection that is comparable or better than either Office 365 selfprotection or traditional endpoint-based anti-malware. Using anti-malware efficacy measurements adapted from traditional endpoint anti-malware product testing, Office 365 self-protection and a third party protection application (Trend Micro CAS) were tested and compared. Of the three attack vectors tested (cloud drive file transfer, social media post, and email), third party protection was most enhanced for cloud drive file transfers. For the five threat types tested (Trojans, adware, malicious links, ransomware, and APT components), third party protection was most enhanced for malicious links (regardless of attack vector). From the results, we can infer (but not conclude) that cloudbased anti-malware leverages dedicated global services (such as cloud-based sandboxes and dynamic reputation services) that are unavailable (or at least not as timely) for on-premise and endpoint-based anti-malware. VI. SUGGESTIONS FOR FURTHER RESEARCH Finally, a note of caution with these results: As mentioned in the experiment design section above, the sheer complexity of the systems under test requires further exploration of the correspondence between attack vector options for the attackers and the routing and ordering of protection processes by the defenders. The work presented here is very much merely a first attempt at measuring the protection of Office 365 user accounts with cloud-based anti-malware defenses. Cyber-attackers have several options when combining each

high-level threat type (malware binaries, infected documents, malicious hyperlinks) with each high-level attack vector (email, file transfers, social media posts). The tests reported here are reduced and simplified subsets. Refinements of the test design could establish additional relevant properties of Office 365 user account protection. Among these are: a. Varying the means of insertion or attachment of malicious binaries or links into the SaaS services (e.g., inserting malicious links directly into the body of email messages); b. Embedding malicious code in standard Office and other documents (e.g., Word documents, PDFs); c. Malicious links that are neither phishing URLs nor are direct links to the location of malicious binary files (e.g., links containing malicious java script, links as part of more complex exploits or advanced persistent threats); d. Sorting out routing and ordering effects of component antimalware processes within the Microsoft self-protection and when combined with the Trend Micro 3 rd party protection (e.g., suspicious files routed to sandboxes may or may not see all of the anti-malware filters and engines of both services). REFERENCES [1] AMTSO. Real Time Threat List (RTTL). http://www.amtso.org/rttl/ [2] CheckVir. Effectiveness of sandbox analysis prior to storage and synchronization in cloud-based Office 365 productivity networks (in preparation). [3] Colon Osorio FC, F Leitold F, C Pickard C, S Miladinov, A Arrott. Measuring the effectiveness of modern security products to detect and contain emerging threats A consensus-based approach. In: Malicious and Unwanted Software:" The Americas"(MALWARE), 2013 8th International Conference on 2013 Oct 22 (pp. 27-34). IEEE 2013. [4] Leitold F, K Yu, and A Arrott. Component Protection Metrics for Security Product Development: CheckVir Endpoint Test Battery. Veszprog Ltd., Veszprem, Hungary 2014. Available: http://www.checkvir.com/data/files/checkvir-case-study-ti6-2012.pdf [5] Marx A, A Decker, A Arrott. Component protection metrics for security product development: I. AV-TEST Full Product Tests. In Malicious and Unwanted Software (MALWARE), 2012 7th International Conference on 2012 Oct 16 (pp. 54-61). IEEE 2012. [6] Microsoft. Office 365 Security and Compliance. Microsoft, 2016. Available: https://www.microsoft.com/en-us/download/details.aspx?id=26552 [7] Trend Micro. Cloud App Security Datasheet. Trend Micro, 2016. Available: http://www.trendmicro.com/cloudcontent/us/pdfs/business/datasheets/ds_cloud_app_security.pdf. [8] Trend Micro. Virtual Analyzer. Deep Discovery Inspector Online Help. Available: http://docs.trendmicro.com/all/ent/ddi/v3.7/enus/ddi_3.7_olh/va_sensors.html [9] Zaytsev O. Rootkits, spyware/adware, keyloggers and backdoors: detection and neutralization. БХВ-Петербург; 2006.