AutoCAD 2007 vs. AutoCAD 2004

Transcription

1 WHITE PAPER AutoCAD 2007 vs. AutoCAD 2004 The Effect on WAN Optimization Solutions

2 INTRODUCTION Riverbed Technology has hundreds of architecture, engineering and construction (AEC) customers using Steelhead appliances around the world. In late 2007, we started hearing from some of our AEC customers that they were seeing slower performance with AutoCAD 2007 than they had seen with AutoCAD After investigating the problem with our customers and with Autodesk, Riverbed learned that the new.dwg file format used in AutoCAD 2007 writes data to disk in a way that is very different than in earlier versions; namely that in many cases the new file format scrambles bytes even when the file is largely or completely unchanged. Scrambling the bytes in turn defeats the benefits of de-duplication, which is a key element of most WAN Optimization solutions as well as many primary and secondary storage solutions and backup / replication software, and can reduce the performance of AutoCAD 2007 for end users, especially those who work in branch offices connected by narrow or congested wide area network (WAN) links. Since deduplication is only one of three core techniques that Riverbed uses to accelerate applications, reducing the effectiveness of deduplication does not completely eliminate the performance gains, especially in terms of response time for the end users. Data presented in this document shows that it s possible to have little to no reduction in WAN traffic, yet still deliver a significant improvement in response time. This report, validated by Taneja Group, a leading storage analyst firm, describes test results of AutoCAD 2007 used in conjunction with four WAN Optimization products, and compares those test results to the same experiments using AutoCAD The vendors tested were Riverbed, Cisco, Blue Coat and Silver Peak. The validated and objective test results outlined below show the following findings: All products that use de-duplication are affected by the new file format Customers can still see significant time savings by using WAN Optimization appliances, even with the new file format Vendors who use de-duplication and claim to be unaffected by this are not providing complete or accurate information Individual network conditions, traffic, bandwidth and latency can have a dramatic impact on the performance of AutoCAD 2007 when used with a WAN Optimization product Arun Taneja, founder and principal of Taneja Group commented Taneja Group conferred with Riverbed and validated their test methodology. Riverbed has been to the point and transparently forthcoming about these issues to help their customers understand the impact. Clearly, complex file format and application behavior issues present in AutoCAD 2007 impact all vendors in the same way when test conditions are equal, there simply is no magic pill that makes any one vendor s de-dupe or compression work better with this file format. Moreover, since Riverbed has a broad array of tools to apply to the issue, including mature protocol optimization as well as de-duplication, Riverbed actually performed better under more conditions than the WAN Optimization competition, even while AutoCAD s 2007 file format thwarted all approaches to de-duplication. Without a doubt, this issue emphasizes the complexity of storage in the new enterprise, and goes to show that even application vendors must now understand the big picture, including how and where data is accessed and stored in customer enterprises. 11BVendor claims about this issue Several vendors in the WAN Optimization market have claimed in recent press releases, campaigns, blog posts, and magazine articles that they are not affected by the file format change. Others have claimed that their customers have not reported any problems, and, therefore, their products are not affected. Some have also claimed that the problem is a Riverbed-specific issue because Riverbed is application-centric while the other vendors are data-centric. Another argument is that Riverbed deduplication is token-based while theirs is instruction-based, and that the difference in de-duplication architecture was the reason that they are not affected. As this new report clearly shows, Riverbed, Cisco, Silver Peak and Blue Coat are all affected by this issue. Statements to the contrary appear to be based on a misunderstanding of the issue, a misrepresentation of the facts, or a lack of actual customer experience. 12B What is the Truth To make sense of the tests, it s important to be familiar with the underlying issues. One is that bandwidth matters. That might seem obvious, but in some cases, vendors have used test results from 10 Mbps links and compared it to data on 1 Mbps links. As we show in this paper, the results vary significantly based on link bandwidth because the problem caused by reduced levels of deduplication is most severe on thin links; on fatter links (like 10 Mbps links) WAN Optimization products can deliver significant savings in time even if the bandwidth reductions are modest. Another important issue is the Incremental Save Percentage (ISP) setting. ISP (also known as ISAVEPERCENT) is an AutoCAD 2010 Riverbed Technology. All rights reserved. 1

3 feature that governs when a complete save is done vs. a partial save. Setting ISP to 0 forces a complete save and results in a completely scrambled file every time the file is saved. Some vendors have used an ISP setting other than 0 and compared that against files where the ISP setting had been 0. Lastly, some vendors have blurred the line between time savings and bandwidth savings by using a time savings number and calling that performance, and then comparing that against a bandwidth savings result from another vendor and implying they are comparable measures of performance. It would be like comparing acceleration with fuel economy for a car both are measures of different aspects of performance for automobiles, but it s meaningless to compare them directly. As we show here, time and bandwidth are two critical metrics but, they are different and cannot be directly compared. Riverbed has gone out of its way to bring this issue to the attention of its customers and to Autodesk. We are working closely with Autodesk and they are actively working on a hot fix that may mitigate the issue for all WAN Optimization vendors. Whether it will completely solve it or not remains to be seen. Complete test results, equipment used, software versions, network configuration and test methodology are included in appendices at the end of this paper. 2BSetting the Stage WAN Optimization Results with AutoCAD 2004 We tested the saving of an 8MB file in AutoCAD 2004 with ISP=0 (complete rewrite) across a 1Mb/s link with 100ms latency. All four vendors tested (Riverbed, Cisco, Blue Coat and Silver Peak) were able to de-duplicate the 2004 format, but all are affected in a very similar way by the file format changes in AutoCAD When comparing the performance of writing (saving) AutoCAD 2007 against the performance of AutoCAD 2004 in identical scenarios, all vendors had trouble deduplicating the AutoCAD 2007 files because of the byte scrambling (there is simply very little duplicate data). Figure 1 all four vendors were able to successfully deduplicate AutoCAD 2004 files as compared to the base case where no appliances are used (labeled base in the charts to follow). The height of the bar here shows how much data was sent across the WAN with each WAN Optimization vendor s products deployed, so a smaller bar means that less data was sent because of the effectiveness of the de-duplication algorithm. As shown in the chart, Riverbed was able to provide the most de-duplication in this case Riverbed Technology. All rights reserved. 2

4 While bandwidth savings are very important, from the end user s point of view the critical measurement is time: how long does it take to save a file across the WAN? Figure 2 all four vendors also reduce the time it takes to save AutoCAD 2004 files across a WAN as compared to the base case. It is important to notice that this measurement reflects the complete save of the file to the remote server. If we just wanted a really fast save of a particular file, it would almost always be faster for a user to save a local copy on their PC. Particularly in the AEC settings where AutoCAD is used, we assume that there are enterprise goals for collaboration, correctness, and data integrity that override a naïve emphasis on save time. Again, Riverbed provided the most benefit in this case among the four products tested. 3BAutoCAD 2007 Performance Goes Down In mid-2007, as significant numbers of AEC customers started to upgrade from AutoCAD 2004 to AutoCAD 2007, some users at firms who had deployed Riverbed s WAN Optimization solutions started experiencing longer save times than they had become accustomed to. Since many AEC firms had made significant changes to their IT infrastructure, business practices and engineering processes based on the higher performance of our products, we (and they) were concerned. 4BWhy Does AutoCAD 2007 Cause Reduced Performance? In mid-2006, Autodesk made a change in the.dwg file format used by their flagship architecture and engineering product AutoCAD. They make changes in the.dwg file format every few years to keep up with new feature demands and to improve performance. In the newest version of the file format that is used by AutoCAD 2007, 2008 (and soon 2009), some of the changes impact the performance of WAN Optimization solutions. The reason the new format causes difficulty for WAN Optimization solutions is that data in AutoCAD 2007 files can be almost completely scrambled even if a file is opened and saved with absolutely no changes being made. 5BIncremental Save Percentage (ISP) in AutoCAD The most severe issues arise on low bandwidth links (which are typical for most branch offices), but another factor that affects the results is an internal setting in AutoCAD called the Incremental Save Percentage (ISP). When ISP is changed from its default setting of 50 to 0, it forces a complete save of the file every time the file is saved, regardless of how much of the file was changed. Using higher numbers for the ISP setting reduce, but do not eliminate, the scrambling of the file data. That means that when an AutoCAD 2007 user saves a file across a WAN with ISP set to 0, it will always exhibit cold 2010 Riverbed Technology. All rights reserved. 3

5 performance, even if no changes have been made. In contrast, an AutoCAD 2004 user will generally get warm performance because as he or she makes changes to the file, very few of the underlying bytes are actually changed in most cases. Reads are not affected in the same manner. Multiple users of AutoCAD 2007 who read an unchanged.dwg file can all get warm read performance that looks similar to AutoCAD 2004 results. If a user uses ISP set to its default of 50, and makes a very minor change (e.g. adding some text to the drawing), then the impact might be minimal in terms of performance to the end user (time to save the file). However, as the data show, even with ISP set to 50, major changes (e.g. adding a new layout to the drawing) to files can still have a major impact on time to save. The following figures clearly illustrate the problem. First, let s look at the bandwidth savings achieved by all four vendors for AutoCAD 2007 vs. the base case. Again the base case is without any WAN Optimization solutions deployed. Figure 3 shows clearly that none of the vendors were able to significantly reduce the WAN traffic as compared to the base case. In fact, two of the vendors products (Silver Peak and Blue Coat) actually sent more bytes across the WAN than the base case. The lack of ability to deduplicate the files can translate into a degraded user experience when saving the file across a narrow link. For example, the time to save the AutoCAD 2004 file across a WAN in the base case (without WAN Optimization appliances, see Figure 2) was about 156 seconds. Saving the same file across a Riverbed enabled WAN only took 19 seconds, a decrease of about 88% (and other vendors also showed significant time savings). Saving a similar AutoCAD 2007 file across a WAN without WAN Optimization solutions took 145 seconds (See Figure 4) we would have expected a similar reduction in time with Riverbed appliances deployed (reducing 145 seconds by 88% would be about 17.4 seconds), but instead it took 70 seconds, only about a 51% reduction. That type of result is also shown across the board for all vendors the time to save an AutoCAD 2007 file is significantly longer than an AutoCAD 2004 file under identical conditions Riverbed Technology. All rights reserved. 4

6 Figure 4 shows the time required to save AutoCAD 2007 files (note that while the bandwidth required didn t change much from the base case with AutoCAD 2007, the time required is lower. While saving AutoCAD 2007 files across a WAN is faster with all four vendors products, it isn t nearly as fast as it had been with AutoCAD Figure 5 shows the difference between the two file formats in the time required to save the file Riverbed Technology. All rights reserved. 5

7 Figure 6 below shows a comparison of the bandwidth required (Note that some solutions actually required more bandwidth to transmit the AutoCAD 2007 file, not less, indicated by a AutoCAD 2007 bars being higher than the dotted line showing the base case). As you can clearly see, while all the WAN Optimization alternatives did reduce the time required to save the file, there were differences across them, and the time savings were not nearly as dramatic as they were for AutoCAD BDe-duplication Not just for WAN Optimization Solutions De-duplication is used not only by most WAN Optimization alternatives, but also by many storage vendors, backup applications and secondary storage products. In WAN Optimization and backup solutions, de-duplication is used to save bandwidth, while in storage it is used to save disk space. Any product that relies on de-duplication will see reduced effectiveness when handling AutoCAD 2007 files. 14BSummary of the Tests and Findings All four vendors were tested in identical test beds, and all were affected in somewhat similar ways when comparing AutoCAD 2004 (which uses the older file format) to AutoCAD 2007, which uses the new one. Across the board, for all vendors tested, the time to write a file across a WAN link is longer with AutoCAD 2007, and the bandwidth consumed is greater than with AutoCAD The tests were done in a lab with a simulated WAN with a latency of 100 ms (round trip) using three variables, doing each experiment for both AutoCAD 2004 and AutoCAD WAN Bandwidth: 1 Mbps and 10 Mbps Incremental save percentage (ISP): 0 and 50 File changes: Major and Minor (definitions in the appendix) Sixteen sets of tests were completed for each of the four vendors Riverbed, Cisco, Blue Coat and Silver Peak: 2010 Riverbed Technology. All rights reserved. 6

8 Bandwidth 1 Mbps 10 Mbps AutoCAD ISP Setting Change to file: Minor Major Minor Major Minor Major Minor Major AutoCAD 2004 AutoAD BMeasuring the Results Since virtually all WAN Optimization solutions rely on either disk or memory to store data and avoid subsequent data transmission, there is always a notion of a cold or first pass result, and a warm or subsequent pass result. The relevant result for most customers is to compare the warm result against the base case (when no WAN Optimization appliance is present). The reason is that as the WAN Optimization appliances are used, their disks fill up with useful bytes (or files in the case of cache-based solutions), which reduce future traffic and therefore help increase performance. There are two key metrics when evaluating the performance of any WAN Optimization solution over a base case (with no WAN Optimization appliances deployed): How much time was saved? How much bandwidth was saved? These numbers are not necessarily directly related they must be measured and reported separately. In other words, it makes no sense to compare the time savings from one solution against the bandwidth savings of another, which some other vendors have done in an effort to confuse customers. A fair test compares apples to apples. However, the bandwidth savings results for those same experiments tell a different story. For all four vendors, the AutoCAD 2007 file was difficult to deduplicate very little reduction in WAN traffic was seen. 7BThe Effect of WAN Bandwidth on the Test Results Bandwidth does matter it s not always the primary factor in improving application performance (response time, or end-user perception) but it can be very important. For large files that normally can be deduplicated, like AutoCAD, sent over thin links, WAN bandwidth is important. Most (but not all) branch offices at AEC firms are connected by relatively thin links. In these experiments we did the testing using two different bandwidth links: 1 Mbps and 10 Mbps. The former is typical for a branch office, especially when you take into account that a T1 link (1.544 Mbps) will be shared by more than one user and will also be carrying non-autocad traffic ( , backup, VoIP, SharePoint, MS Office, and so forth). The latter may be more representative of a data center (DC) to disaster recovery (DR) site. When comparing results for AutoCAD 2007on 1 Mbps and 10 Mbps links, for example in Table 1 below, the reduction in time for all vendors is much more impressive than the bandwidth savings. AutoCAD 2007 Time Savings Base Riverbed Cisco Blue Coat Silver Peak ISP 0; 1 Mbps 52% 45% 41% 32% ISP 0; 10 Mbps 85% 72% 82% 85% Bandwidth Savings Base Riverbed Cisco Blue Coat Silver Peak ISP 0; 1 Mbps 9% 5% -12% -3% ISP 0; 10 Mbps 11% 10% -6% 4% Some vendors use test results for time savings from a 10 Mbps link and compare that to bandwidth savings over a 1 Mbps link clearly a nonsensical and meaningless comparison. You need to evaluate the same test under the same conditions across vendors to have anything meaningful Riverbed Technology. All rights reserved. 7

9 8BCache-based Alternatives and AutoCAD 2007 Performance Several vendors in the WAN Optimization space have claimed that they are not affected by the AutoCAD 2007 file format issue. Two of the vendors who make those claims, Cisco F1F and Packeteer, have file caches built into their products (the file caches in their products are due to prior acquisitions of other companies: Actona in the case of Cisco and Tacit Networks in the case of Packeteer). File caches can make it difficult to make valid comparisons because it is possible to configure some cache-based solutions to prematurely acknowledge writes locally before the file save has actually been completed on the other side of the WAN. If a file cache is set up that way, it can make the end user feel like his or her AutoCAD 2007 file has been written very quickly, much like performing a save on the user s PC but with similar problems in terms of consistency of file data. All the testing in this paper was done with the files written back to the origin server, which is the preferred mode of operation for most firms. The next section explores this issue in more detail. 9BWriting to a Local Cache vs. to an Origin Server in Cache-based Solutions It is important to understand the distinction between these local and global modes (they may be called different names by different vendors). In a global mode, when a file is written back to a data center storage device across a WAN, the client (Windows for example) does not receive the final acknowledgement that the file is completely and safely written until all the data has actually been written to the data center file server. This mode can be a bit slower, but it s the only safe mode that ensures that a second user cannot open the same file while it s still being written. A cache configured in what we d call local mode would have to trade off safety in favor of end user performance by spoofing the normal server acknowledgement so that the user experience is better. In that case, there are, in essence, two writes for every file saved -- one locally in the branch office and the second on the origin server in the data center. The first save can be very fast because the file is saved locally on the cache with a premature acknowledgement of the save being completed in the data center. Despite appearances, the cache continues to write the data over the WAN to the origin server without any benefit of de-duplication for AutoCAD However, it s important for customers to realize that there is no free lunch. F2F Writing to a local file cache and acknowledging it without truly writing it to the server is less safe, and can be proven to be so. An obvious case where this can be a problem is in a multi-site collaborative environment. There is no way to reconcile local independence of action (multiple copies), global singlecopy correctness, and failures of networks or devices This is a well known problem in architecture and engineering circles which can cause massive mistakes and cost overruns. Vendors who have local cache mode settings in their product can make it seem as though they are not affected by the AutoCAD 2007 file format when in fact all they are doing is prematurely acknowledging saves at the expense of file consistency. They are exposed to the AutoCAD 2007 format change in a more dangerous way. As the data show, the new-format files will still take longer to travel across a narrow network link. If the write is acknowledged immediately but the file takes longer to actually be written at the server, a cache-based system is increasing the size of the window of vulnerability during which a failure can cause a problem. Users of a cache-based system that has worked fine with AutoCAD 2004 should not be comfortable with the fact that their system appears to work fine with AutoCAD 2007 since these measurements show they actually have the same underlying performance challenges. They will see similar increases in bandwidth consumption because they cannot deduplicate any more effectively than Riverbed, and they will also be exposed to a hard-to-characterize higher risk of data corruption from failures during writes. 10BSummary The collision of an important software application with storage and networking products is an interesting one. Riverbed has tried to provide complete and accurate information about this issue from the time we first became aware of it. We are working closely and collaboratively with Autodesk, and hope that they can issue a hot fix as quickly as possible. In the meantime, we hope that this paper and the accompanying data have clearly explained the issue. 1 In an April 21, 2008 article on SearchStorage.com, a Cisco customer is quoted as having done the specific tests of AutoCAD 2007 with his WAAS appliances, and the reduction in de-duplication was confirmed. See HUhttp://searchstorage.techtarget.com/news/article/0,289142,sid5_gci ,00.html UH 2 For a more complete exploration of this topic, see Riverbed s white paper No Free Lunch on the CAP theorem (Consistency, Availability and Partition-tolerance) Riverbed Technology. All rights reserved. 8

10 The key points to remember are: All products that use de-duplication are affected by the new file format Customers can still see significant time savings by using WAN Optimization appliances, even with the new file format Vendors who use de-duplication and claim to be unaffected by this are not providing complete or accurate information Your own network conditions, traffic, bandwidth and latency can have a dramatic impact on the performance Appendix A WAN Optimization Solution Details Hardware and Software Definitions of Minor and Major change used in these tests: Minor change: 1) Open drawing 2) Click on Draw Text Single Line Text 3) Added text 4) Save AS 5) Enter new name of file Major change: 1) Open drawing 2) Click on Insert Layout New Layout 3) Click on new layout tab 4) Save As 5) Enter new name of file 2010 Riverbed Technology. All rights reserved. 9

11 AutoCAD 2004 vs and WAN Optimization Appliances Appendix B Summary of all test results 2010 Riverbed Technology. All rights reserved. 10

12 1B AutoCAD 2004 vs and WAN Optimization Appliances The complete data sets and files used in the tests are available here: About Riverbed Riverbed Technology is the IT performance company. The Riverbed family of wide area network (WAN) optimization solutions liberates businesses from common IT constraints by increasing application performance, enabling consolidation, and providing enterprise-wide network and application visibility all while eliminating the need to increase bandwidth, storage or servers. Thousands of companies with distributed operations use Riverbed to make their IT infrastructure faster, less expensive and more responsive. Additional information about Riverbed (NASDAQ: RVBD) is available at Riverbed Technology, Inc. 199 Fremont Street San Francisco, CA Tel: (415) Riverbed Technology Ltd. Farley Hall, London Road, Level 2 Binfield, Bracknell Berks RG42 4EU Tel: Riverbed Technology Pte. Ltd. 391A Orchard Road #22-06/10 Ngee Ann City Tower A Singapore Tel: Riverbed Technology K.K. Shiba-Koen Plaza Building 9F 3-6-9, Shiba, Minato-ku Tokyo, Japan Tel: Riverbed Technology. All rights reserved. 11