An Evaluation of Using Deduplication in Swappers

Size: px
Start display at page:

Download "An Evaluation of Using Deduplication in Swappers"

Transcription

1 An Evaluation of Using in Swappers Weiyan Wang, Chen Zeng Computer Sciences Department University of Wisconsin, Madison Abstract Data deduplication is a specialized data compression technique for eliminating coarse-grained redundant data, typically to improve storage utilization. In this paper, we explore another direction of using deduplication. That is, we use deduplication in the swapper. Before swapping out any page from memory to the swap area in disk, deduplication checks that whether a page of the same contents has been written to the swap area. If so, we can avoid one I/O. We implement that idea in the linux kernel. Our experimental results indicate that using deduplication is able to reduce the overhead of swappers by orders of magnitude when there are many duplicate pages comparing to that of not using deduplication. However, we also notice that deduplication also incurs an overhead when few duplicate pages are present. Introduction Data deduplication is a specialized data compression technique for eliminating coarse-grained redundant data, typically to improve storage utilization. In the deduplication process, duplicate data is deleted, leaving only one copy of the data to be stored, along with references to the unique copy of data. is able to reduce the required storage capacity since only the unique data is stored. Data deduplication is important in saving storage space, which is able to save up to 9% of the storage space []. However, the majority of previous works have only focused on how to increase the compression ratio as well as the throughput of disk. One missing part is, is the deduplication technique useful in memory as well as in disk? To make it concrete, suppose we need to run a program which requires a lot of memory, and the physical memory is smaller than the working set of that program. Therefore, can we utilize the deduplication technique to compress those pages in memory, thus make the working set fit into memory? One example of the applications which requires a lot of memory resource is the sparse matrix. Sparse matrix is a matrix that the majority of the elements is. Therefore, if we can compress that matrix using deduplication technique, then that matrix could fit into memory, and many I/Os can be avoided if we need to operate on that matrix. In this paper, we explore one aspect to utilize deduplication techniques in memory. That is, we are attacking the problem of how to integrate deduplication with the swapper. The swapper is a process that periodically writes pages out to the swap area and reads in other pages. The swapper also manages swap areas on disk, and keeps track of mappings between per-process virtual address and disk block address. In general, our idea is to speed up swapping by first checking to see if a particular page is already in the swap region; if so, the swap can be avoided, thus speeding up the program. Therefore, when the program needs to swap out a page, and that page has already had a duplicate page in the swap area, there is no need to swap that page out. As a result, the program does not feel much overhead of that swap out operation. Hence, another way to interpret our approach of utilizing deduplication in swappers is that our approach actually transparently provides additional physical memory to the program in that the working set of a program seems to fit in memory using deduplication now. Our contribution lies in two folds. First, we implement the idea of integrating deduplication and the swapper together in the Linux kernel.6.6. Second, we empirically evaluate the performance of the deduplication swapper. We test our swapper with a program that sequentially scans an array. Our experimental results indicate that using deduplication in the swapper is able to reduce the accessing time by orders of magnitude when % of the pages in that array are duplicates. Of course, using deduplication in the swapper also incurs an overhead, and we find out that when there is no duplicate pages, our swapper is 7.4% slower than the original swaper. We have also run a benchmark program, gmake, which compiles the Linux kernel, and find out that the performence of our swapper is comparable to the original Linux swapper. The rest of this paper is organized as follows: Section presents our design of how to integrate deduplication with the swapper. Section discusses the details of our

2 Figure : page.! procedure when swapping out a implementation, and several optimizations we have made to improve the efficiency of the swapper. Section 4 empirically evaluate the performance of using deduplication in the swapper, and Section 5 discuss related works. We conclude our work in Section 6. Design A high-level architecture of our design consists of the following components. When a page is going to swap in or swap out, a checksum engine is used to compute a cryptographic hash digest of its content, as described in Section.. A dedup cache keeps track of swap entries of pages on the disk, as described in Section.. A swap cache is maintained to temporarily keep swap-out pages in the memory to avoid disk consistency problem, which is explained in Section.4. Our system uses these components to provide deduplication feature in the swapper. The following section explains the deduplication procedure during swapping out.. Procedure Figure shows the deduplication procedure for swapping out. When the system determines to swap out a page, checksum engine is called to calculate the checksum based on page s content, and return that checksum to the swapper. Then swapper uses the checksum as a key to search the dedup cache to check if some pages with the same content have been swapped out to the disk. If so, dedup cache will return the swap entry to the swapper, and increase the counter of that swap entry to indicate that one more page is deduplicated. The mapping between the page and swap entry is then added to the swap cache. After that, swapper will replace the page table entries of all processes mapped to the page with its swap entry. That process is called unmapping. When unmapping is finished, we remove the mapping of the page from the swap cache. Because the page s content has already been on the disk, we do not need to write that page to the disk again. Therefore, we avoid one disk access. However, if the lookup of the dedup cache fails, swapper must search in the swap area to find a free swap entry for the page. When the page s swap entry is determined, similar to the above case, the swapper will update the swap cache, and unmap the page from all related processes. Then a disk write will be issued to write the page to the swap area. When the pageout succeeds, the page s swap entry will be added to the dedup cache for further deduplication, the counter of the swap entry will be initialized to. And finally, the mapping of the page in the swap cache is removed. " #$% &'()* " #$% +$+, & 5, & '8'& 9 4 6% : &.6% +$+, & <=> : &+)&$& +6'(&)?@ AB C=D E F - &;& /) : &.6% +$+, & - &$. /). 4 G.. ( " #$% +$+, & Figure : procedure when swapping in a page. Figure demonstrates the update of the dedup cache for swap in process. When swapper wants to swap in a page with a swap entry, it will first search the swap cache to check whether that page is in the memory. If not, swapper must issue a disk read to load the page from disk to the memory. Then it will add the mapping between swap entry and page to the swap cache to avoid swapping in from disk again. When that page is loaded, checksum engine will compute its checksum, and dedup cache is indexed with checksum. If the swap entry found in the dedup cache is the same as the swap in one, we will decrease its counter. If the counter drops to zero, then all duplicate pages have been swapped in, we can remove it from the dedup cache. The following subsections explain the detail of each component.

3 . Checksum Engine In the checksum engine, we compute SHA- hash of a page s content. According to [8], a 6-bit SHA- hash has a collision probability of one in 8. This probability is fairly small that we can assume two pages are identical if they have the same checksum. Therefore, unlike [], we don t keep the whole page associated with checksum in the dedup cache to verify the content of page. Another reason for this trade-off is that for each swapout page, we need one page frame to keep the copy of it in the dedup cache, which will increase the memory pressure. Because of the implementation of dedup cache, dedup cache can only index a -bit key. Therefore, instead of using all bits of SHA- hash, we use its first -bit as the checksum for each page. The major reason for that is because of our implementation of the dedup cache, which we will address later. As a result, the collision probability drops to one in 6. We does observe that two different pages may be mapped to the same checksum, and leave the problem of how to improve the consistency of the swapper for future works.. Dedup Cache Figure : Structure of dedup entry. For each swap entry, we create a new structure called dedup entry to keep track of its status in the dedup cache. Figure shows the contents of dedup entry. Variable base stores the data of swap entry. Variable count is the counter of the swap entry, which maintains the number of duplicate pages for this entry. Count will be increased(decreased) by when the page is swapped out(swapped in). When count drops to zero, it means that all duplicate pages have been swapped in, we can remove the dedup entry from the dedup cache, and delete that dedup entry. Variable ref is similar to pagecount, it maintains the number of processes currently using the dedup entry. We can only delete the dedup entry when ref is equal to one. ref is necessary by considering the following scenario: Swapout process A is using dedup entry to update the page table entries, while at the same time swapin process B finds count is dropped to zero, it then deletes the dedup entry. Error happens when process A accesses it later. With ref, when process A is using the dedup entry, its ref will be increased by. Then process B can t delete dedup entry, which preserves dedup entry until process A finishes processing. The spin lock lock is used to support atomic updates of ref and count. Dedup cache stores the mappings from page checksums to dedup entries. For each swapout page, dedup cache is used to check if any page with the same content has been swapped out to the disk. If so, dedup cache will return the dedup entry for that page pretending the page was swapped out, one disk write can be avoided. We use a radix tree to implement dedup cache for two reasons. First, the time complexity of lookup and update for radix tree is O( key ) where key is the length of the key, which is as efficient as hash table. Second, radix tree is a well-implemented data structure in the linux kernel, we can directly use it instead of creating a bug-free hash table by ourselves. The only disadvantage of using built-in radix tree is that it can only handle a -bit key, which forces us to use the first -bit of SHA- hash as the checksum and this may hurt the consistency of system..4 Swap cache When swapping out pages to the swap area, Linux avoids writing pages if it does not have to. There are times when a page is both in a swap area and in physical memory. This happens when a page that was swapped out of memory was then brought back into memory when it was again accessed by a process. So long as the page in memory is not written to, the copy in the swap area remains valid. Linux uses the swap cache to track these pages. The swap cache is a list of page table entries, one per physical page in the system. That is a page table entry for a swapped out page and describes which swap file the page is being held in together with its location in the swap area. If a swap cache entry is non-zero, it represents a page which is being held in a swap file that has not been modified. If the page is subsequently modified (by being written to), its entry is removed from the swap cache. When Linux needs to swap a physical page out to the swap area, it consults the swap cache and, if there is a valid entry for this page, it does not need to write the page out to the swap area. This is because the page in memory has not been modified since it was last read from the swap area. The entries in the swap cache are page table entries for swapped out pages. They are marked as invalid but contain information which allow Linux to find the right slot within that swap area. The implementation of the swap cache is a radix tree, where the key is the index of the swap entry in swap area However, using deduplication in the swapper poses a possible inconsistency problem in the swap cache. Consider the following steps:. A process T needs to swap out page P, and install the mappinge P into swap cache

4 . Another process T requires to swap out another pagep, whose content is identical top. Therefore, we do not need to swap out P because the content of P is in the swap entry E but we can not install the mappinge P.. The pagep is swapped out to the swap area, and the mappinge P is deleted. 4. When T needs to access P again, T can not find the swap entry ofp, which causes inconsistency. To remedy that problem, we change the structure of the swap cache. That is, instead of storing the mapping between a swap entry and a single page, we store the mapping between a swap entry and a list of pages, whose contents are identical. The rationale of that implementation is that deduplication actually maps multiple pages to one page, which has the same content. Therefore, different pages could have different physical addresses but share the same slot in swap area. After that modification, the inconsistency does not happen.. A processt needs to swap out page P, and install the mappinge P into swap cache. Another process T requires to swap out another page P, whose content is identical to P. We add P to the list, and the mapping is E P,P > now.. The page P is swapped out to the swap area, and the mappinge P is deleted,and the mapping is E P 4. When T needs to access P again, T can find the swap entry ofp. We utilize the implementation of list in kernel to change the structure of the swap cache. Implementations In this section we discuss several implementation issues and optimizations of our design.. Using Built-in Swap Counter During the implementation, we find out that swap counter built in the kernel is a better choice than count variables in the dedup entry to reflect the usage of swap entry. That is because, every swap entry has its own swap counter to maintain the number of references to it in the memory in the kernel. Like the count variable, swap counter is updated when the corresponding page is swapped out or swapped in. Besides swapping, swap counter is updated when the swap entry is inserted into the swap cache or the page table entry containing the swap entry is copied from one process to another process. The latter happens when the system call fork is called to create a child process. The count variable can t be updated for that situation because only page table entry is involved. Without a page, we can t locate the dedup entry in the dedup cache to update count variable. As a result, when the count is dropped to zero, the corresponding swap entry may still be referenced by some processes. Because it is removed from dedup cache, it cannot be used for further deduplication to avoid more pageouts. In contrast, when swap counter is dropped to one, the corresponding swap entry is only referenced by the swap cache. We can safely remove it from the dedup cache at this time. To use swap counter in deduplication, we need to make a small change: besides the original updates, swap counter will be updated whenever the swap entry is inserted into the dedup cache. That is because we want to make sure the swap entry is not released by the swapper during its time in the dedup cache. Therefore, when a page is swapped in and the swap counter drops to two, we know that the swap entry is only referenced by the swap cache and dedup cache, then we can safely remove it from the dedup cache and swap cache. However, we should still keep ref in the dedup entry. Consider the scenario discussed in Section. again. If the process A is still using the dedup entry while process B notices that all pages in the swap entry have been swapped in. If the process B removes that entry from both the dedup cache and swap cache, then that swap entry will be released and may be used by other swapper process. Inconsistency will occur because the process A still considers that swap entry has the same content with the page that is swapping out.. Changes to swap cache Because the swap cache is implemented in radix tree, we optimize the structure of the list by having a header of the list. The reason of that is to reduce the overhead to maintain the structure in the radix tree. To make it concrete, suppose a swap entry E is mapped to a list P,P where P is the first element in that list. When we need to delete the mappinge P, we need to update the mapping to E P. Unfortunately, the implementation of radix tree does not allow us to explicitly do that update. Instead, that update consists of two parts: delete the mapping E P,P, and insert the mapping E P, which incurs two additional calls. Our optimization, which always have a header for that list, is able to eliminate that additional overhead. The reason is that the entry in the swap cache is changed to E head,p,p, and after deletingp, the mapping 4

5 ise head,p, where thee still points to the head. Hence, the structure is consistent, and we can avoid to delete the entry and reinsert a new entry. Another problem we find out is that the list can not increase infinitely. That is because the cost of search in the list increases significantly. Moreover, we find out that when the number of pages in that list is too long, the swapper just stops. Unfortunately, we can not find a good explanation for that phenomena. We solve that problem by limiting the number of pages in a list, and if that number exceeds a threshold, then we explicitly shrink that list.. Policies of Swap Cache Page List In the swap cache, we maintain a page list to store all pages refer to it for each swap entry. There are three operations related to the page list: insert a page during swapout, remove a page when swapout is finished, and return a page during swapin. Considering the page list is organized as a cyclic double linked list, we use the following policies to optimize those operations: To remove a page, search the list in the same direction with insertion. For example, if we add the page as the next page of the head each time, to remove a page, we will go through the list with next page pointer each time. This policy is based on one observation: most of time, the page added to the list is a duplicate page. It is likely to be removed from the list very soon after the insertion since it does not need to be swapped out to the disk. Therefore, start searching from the most recently added page is more likely to find the page to be removed. Return the oldest page in the list to the caller during swapin. The swapin process lookups the swap cache to find a usable page in the memory, it doesn t matter which page to be returned. However, when a page is returned to the caller, it will be kept in the page table of some process until the page is no long used, it can t be released after the swapout. If we return the newest page to the caller, since the list is growing, it s highly possible that we return different pages when lookup function is called at different time. As a result, these duplicate pages would be kept in the memory for a relatively long time and can not be released, which reduces the effect of swapper. In contrast, returning the oldest page enables other pages to be released after swapout as we expect. It s also memory efficient because a page is shared by processes as many as possible..4 Checksum Inside Dedup Entry Because using -bit checksum may have potential consistency problem, we try to solve this by including the whole SHA- checksum in the dedup entry. In this way, to retrieve the dedup entry for a page, we first use the first -bit of its SHA- hash to index the radix tree. If a nonnull dedup entry is returned, we then compare the whole checksum inside the dedup entry with the page s SHA- hash to do the further verification. Only when two SHA- hash are the same we return the dedup entry to the swapper. However, our experiments show that this small change hurts the performance of swapper especially when all pages are different. It s mainly because each dedup entry now costs much more space than before, which will increase memory pressure a lot if all dedup entries are kept in the dedup cache for a relatively long time. What s more, we need more time to compare the whole checksum. For this reason, we must make a trade-off between consistency at some point for the performance. For example, only the second -bit of SHA- hash is stored in the dedup entry. 4 Experimental evaluations We implement our algorithm in the linux kernel.6.6 of Fedora running on VMWare 7. workstation. We set the memory of of the virtual machine to be 8 MB. We run several workloads to compare the pros and cons of our approach. 4. Sequentially scanning an array The first one is to sequentially access the elements in an array, each of which is of the same size of a page, 4KB. We sequentially scan that array for ten times, and report the average accessing time of that array. We change the size of the array and the percentage of duplicate pages in that array to check the impact of those two parameters in the swappers. We sequentially scan that array for ten times, and report the average accessing time of that array. 4.. Under memory contention First, we set that all of the pages in an array are of the same contents. The experimental result is shown in Figure 4. We observe that using deduplication in the swapper is able to reduce the overhead of swapping by orders of magnitude. The reason why the access time increases significantly in original linux swapper when the size of the array is is that the operating system thrashes at that point. That is, the total size of the array is MB. 5

6 7 x x Figure 4: Access time when all of the pages are duplicates Figure 6: Access time when 4% of the pages are duplicates x Figure 5: Access time when % of the pages are duplicates 4.5 x Therefore, the memory contention is quite high. As a result, when all of the pages in memory are used, and the program needs to access another page, the original linux swapper has to swap out one page in memory to the swap area in disk. However, our swapper is able to detect that the contents of those pages are identical. Therefore, we can save that I/O. As a result, using deduplication in swapper is able to reduce the sequential access time by orders of magnitude when the array is large, and a lot of duplicate pages are present in that array. We also present the result when % and 4% of the pages in the array are duplicates in Figure 5 and Figure 6, respetively. We can still observe that using deduplication is able to significantly reduce the overhead of accessing the array for orders of magnitude when the array does not fit into memory. However, one question is, is deduplication always ben- Figure 7: Access time when none of the pages are duplicates eficial? In our first experiment, because all of the pages are duplicates, using deduplication in the swappers is able to significantly reduce the access time of sequential scanning. In the second experiment, we also push to another limit, where there are no duplicate pages in that array. Figure 7 shows that result. We find out that using deduplication also incurs a significant overhead. That is, when the size of the array is, the access time of scanning using deduplication is 7.4% longer than that of no deduplication. Therefore, using deduplication comes with a cost. We will analyze the cost of using deduplication later. 6

7 Figure 8: No memory contention, all duplicate pages Figure 9: Access time when 6% of the pages are duplicates 4.. No memory contention We are also interested in the overheads incurred by using deduplication. Therefore, we also present the result when there is no memory contention. Figure 8 illustrates the result when all of the pages are duplicates. We clearly observe that the overhead of using deduplication when there is no memory contention is negligible. That is because the average access time is comparable to the original Linux kernel. The reason is that the working set of the array is able to fit into memory. Therefore, the swapper is not invoked to relieve the memory pressure. Hence, no I/O occurs. As a result, the behavior of using deduplication in swapper is similar to the original swapper without deduplication. We observe similar results when we set the percentage of duplicate pages to 6% and 8%. Those results are show in Figure 9 and Figure, respectively Figure : Access time when 8% of the pages are duplicates 4. Overheads of using deduplication In Figure 7, we observe that when there is no duplicate pages, the accessing time of using deduplication in a swapper is actually worse than the original swapper. We further investigate the overheads of each function we have changed, and list the overheads of each function in Table. We find out there are two major overheads. The first one is to calculate the checksum of a page. We find out that the average overhead of calculating the checksum is 6. µs. Furthermore, we need to calculate the checksum value for each page whenever that page is swapped out or swapped in. Therefore, if there is no duplicate pages, then our swapper needs to swap in/out pages as original swappers, and calculate the checksums for each swap in/out. Moreover, we also observe that the function decrease dentry ref is 98.87µs. The reason for that is that Function Overhead checksum 6.µs can be dedup.7 µs add to dedup cache.5 µs decrease dentry ref µs Table : Overhead of modified functions in addition to calculating the checksum in that function, we also use locks to maintain the consistency of dentry ref field of a dedup entry. Callers have to wait on those locks to update the dentry ref field of a dedup entry, which becomes a bottleneck. Those two functions are the reasons why our swapper is slower than the original swapper when there are no duplicate pages in that array. 7

8 Using deduplication No deduplication.8 s.689 s Table : Overhead of modified functions 4. Real world applications To further investigate the pros and cons of using deduplication in swappers, we run a gmake benchmark to test the efficiency of our approach. We compile Linux kernel.6.6 with the following command for ten times, make -j >/dev/null and report the average compiling time. Table shows the experimental result. We find out that the average compile time of using deduplication in swappers is comparable to that of not using deduplication. Therefore, using deduplication does not pose a significant overhead in the gmake benchmark. One of the reasons is that the memory usage is not that intense in that gmake benchmark. Therefore, the experimental result is similar to Figure 8 that using deduplication does not incur a significant overhead when there is no memory contention. 5 Related works Much work on deduplication focused on basic methods to save the storage space. Early deduplication storage systems use file-level hashing to detect duplicate files and reclaim their storage space [,, 4]. Since such systems also use file hashes to address files. Some call such systems content addressed storage or CAS. Since their deduplication is at file level, such systems can achieve only limited global compression. Modern systems detect deduplication at a finer granularity than files. Removing duplications at content-based data segment level has been applied to network protocols and applications [, 9, 7, 5] and has reduced network traffic for distributed file systems [6, ]. However, all of those efforts aim to save storage space by detecting duplications. Therefore, our work is orthogonal to their in that we focus on using deduplication in memory instead of in disk. So far to our knowledge, our work is the first effort to integrate the deduplication technique into the swapper. Although [] also detect deduplication, their focus is to reduce the memory usage when two virtual machines require the same page. Our work is different to them in two ways. The first one is that our work focuses on a different aspect than theirs in that we do not consider page sharing among different virtual machines. Instead, we focus on increasing the efficiency of the swapper. Second, we trade consistency for performance in our implementation, which only uses checksum to identify two pages with same contents while their approach will still perform contents based comparison. 6 Conclusions In this paper, we propose to use deduplication techniques in swappers, and implement that idea in Linux kernel.6.. Our experimental results indicate that using deduplication is able to significantly improve the efficiency of the swapper when many pages are duplicates, and the operating system is under heavy memory contention. Furthermore, we also show that using deduplication is able to provide comparable performance when few pages are duplicates. Although we do find out that using swappers could incur a significant overhead when there are no duplicate pages comparing to the original swapper, we think that problem could be alleviated by better implementation. So far to our knowledge, our paper is the first effort to integrate the deduplication with the swappers. In addition to improving the efficiency of the original swapper, one fundamental benefit of using deduplication is to reduce the size of the working set of a program. That is, the pages of the same contents are considered to be one page by our swapper. Therefore, another perspective to understand deduplication in the swapper is to think that the swapper transparently enlarge the size of the memory. As a result, the working set of a program could fit into the memory using deduplication in swapper, whose size is even larger than the memory. Therefore, the execution of that program with our swapper is more efficient than using the original swapper. Of course, there are many future directions to improve our implementations. Currently, we calculate the checksum of a page whenever that page is swapped in or out. Therefore, we have to pay a performance penalty of 7 µs whenever that page is swapped in and out. However, we do observe that we can store the checksum with that page to avoid those redundant computations of checksum values of a same page if the content of that page does not change. As described in our experiments, another bottleneck is to update the dentry ref field. One possible way is to employ the idea of sloppy counter [] to reduce the overhead of concurrency access. We have made a trade-off in our implementation: we trade consistency for performance. That is, our implementation only uses checksums to compare two pages. Therefore, it is possible that two pages are not identical but have the same checksum. That problem is exaggerated by our implementation, which only uses the first bits of the 6 bits of a checksum value. Therefore, it could be beneficial to implement a 6 bits dedup cache to reduce the probability of collision. How to improve the consistency 8

9 while still improve performance of using deduplication in swappers is also an interesting topic for future research. References [] A. Adya, W. J. Bolosky, M. Castro, G. Cermak, R. Chaiken, J. R. Douceur, Jon, J. Howell, J. R. Lorch, M. Theimer, and R. P. Wattenhofer. Farsite: Federated, available, and reliable storage for an incompletely trusted environment. In In Proceedings of the 5th Symposium on Operating Systems Design and Implementation (OSDI. [] S. Boyd-wickizer, A. T. Clements, O. Mao, A. Pesterev, M. F. Kaashoek, R. Morris, and N. Zeldovich. An analysis of linux scalability to many cores. In OSDI,. [] N. Jain, M. Dahlin, and R. Tewari. Taper: Tiered approach for eliminating redundancy in replica synchronization. In In USENIX Conference on File and Storage Technologies, pages 8 94, 5. [4] P. Kulkarni, F. Douglis, J. LaVoie, and J. M. Tracey. Redundancy elimination within large collections of files. In Proceedings of the annual conference on USENIX Annual Technical Conference. [5] J. C. Mogul, Y. M. Chan, and T. Kelly. Design, implementation, and evaluation of duplicate transfer detection in http. In Proceedings of the st conference on Symposium on Networked Systems Design and Implementation - Volume, 4. [6] A. Muthitacharoen, B. Chen, and D. Mazières. A low-bandwidth network file system. SIGOPS Oper. Syst. Rev. [7] S. C. Rhea, K. Liang, and E. Brewer. Value-based web caching. In Proceedings of the th international conference on World Wide Web, WWW,. [8] V. Rijmen and E. Oswald. Update on sha-. In In Lecture Notes in Computer Science, pages Springer, 5. [9] C. P. Sapuntzakis, R. Chandra, B. Pfaff, J. Chow, M. S. Lam, and M. Rosenblum. Optimizing the migration of virtual computers. SIGOPS Oper. Syst. Rev.,. [] N. T. Spring and D. Wetherall. A protocol-independent technique for eliminating redundant network traffic. In In Proceedings of ACM SIGCOMM, pages 87 95,. [] N. Tolia, M. Kozuch, M. Satyanarayanan, B. Karp, T. Bressoud, and A. Perrig. Opportunistic use of content addressable storage for distributed file systems. In IN PROCEEDINGS OF THE USENIX ANNUAL TECHNICAL CONFERENCE. [] C. A. Waldspurger. Memory resource management in vmware esx server. SIGOPS Oper. Syst. Rev. [] B. Zhu, K. Li, and H. Patterson. Avoiding the disk bottleneck in the data domain deduplication file system. In Proceedings of the 6th USENIX Conference on File and Storage Technologies, FAST 8. 9

ECE 669 Parallel Computer Architecture

ECE 669 Parallel Computer Architecture ECE 669 Parallel Computer Architecture Lecture 9 Workload Evaluation Outline Evaluation of applications is important Simulation of sample data sets provides important information Working sets indicate

More information

Caching for NASD. Department of Computer Science University of Wisconsin-Madison Madison, WI 53706

Caching for NASD. Department of Computer Science University of Wisconsin-Madison Madison, WI 53706 Caching for NASD Chen Zhou Wanli Yang {chenzhou, wanli}@cs.wisc.edu Department of Computer Science University of Wisconsin-Madison Madison, WI 53706 Abstract NASD is a totally new storage system architecture,

More information

Portable Storage Support for Cyber Foraging

Portable Storage Support for Cyber Foraging Portable Storage Support for Cyber Foraging Ya-Yunn Su and Jason Flinn Department of Electrical Engineering and Computer Science University of Michigan Abstract The ubiquitous computing community has focused

More information

SHHC: A Scalable Hybrid Hash Cluster for Cloud Backup Services in Data Centers

SHHC: A Scalable Hybrid Hash Cluster for Cloud Backup Services in Data Centers 2011 31st International Conference on Distributed Computing Systems Workshops SHHC: A Scalable Hybrid Hash Cluster for Cloud Backup Services in Data Centers Lei Xu, Jian Hu, Stephen Mkandawire and Hong

More information

An Analysis of Linux Scalability to Many Cores

An Analysis of Linux Scalability to Many Cores An Analysis of Linux Scalability to Many Cores 1 What are we going to talk about? Scalability analysis of 7 system applications running on Linux on a 48 core computer Exim, memcached, Apache, PostgreSQL,

More information

Parallelizing Inline Data Reduction Operations for Primary Storage Systems

Parallelizing Inline Data Reduction Operations for Primary Storage Systems Parallelizing Inline Data Reduction Operations for Primary Storage Systems Jeonghyeon Ma ( ) and Chanik Park Department of Computer Science and Engineering, POSTECH, Pohang, South Korea {doitnow0415,cipark}@postech.ac.kr

More information

Memory Management. Virtual Memory. By : Kaushik Vaghani. Prepared By : Kaushik Vaghani

Memory Management. Virtual Memory. By : Kaushik Vaghani. Prepared By : Kaushik Vaghani Memory Management Virtual Memory By : Kaushik Vaghani Virtual Memory Background Page Fault Dirty Page / Dirty Bit Demand Paging Copy-on-Write Page Replacement Objectives To describe the benefits of a virtual

More information

The What, Why and How of the Pure Storage Enterprise Flash Array. Ethan L. Miller (and a cast of dozens at Pure Storage)

The What, Why and How of the Pure Storage Enterprise Flash Array. Ethan L. Miller (and a cast of dozens at Pure Storage) The What, Why and How of the Pure Storage Enterprise Flash Array Ethan L. Miller (and a cast of dozens at Pure Storage) Enterprise storage: $30B market built on disk Key players: EMC, NetApp, HP, etc.

More information

Scalable Correct Memory Ordering via Relativistic Programming

Scalable Correct Memory Ordering via Relativistic Programming Scalable Correct Memory Ordering via Relativistic Programming Josh Triplett Portland State University josh@joshtriplett.org Philip W. Howard Portland State University pwh@cs.pdx.edu Paul E. McKenney IBM

More information

Design Tradeoffs for Data Deduplication Performance in Backup Workloads

Design Tradeoffs for Data Deduplication Performance in Backup Workloads Design Tradeoffs for Data Deduplication Performance in Backup Workloads Min Fu,DanFeng,YuHua,XubinHe, Zuoning Chen *, Wen Xia,YuchengZhang,YujuanTan Huazhong University of Science and Technology Virginia

More information

GFS Overview. Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures

GFS Overview. Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures GFS Overview Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures Interface: non-posix New op: record appends (atomicity matters,

More information

Opportunistic Use of Content Addressable Storage for Distributed File Systems

Opportunistic Use of Content Addressable Storage for Distributed File Systems Opportunistic Use of Content Addressable Storage for Distributed File Systems Niraj Tolia *, Michael Kozuch, M. Satyanarayanan *, Brad Karp, Thomas Bressoud, and Adrian Perrig * * Carnegie Mellon University,

More information

Reducing The De-linearization of Data Placement to Improve Deduplication Performance

Reducing The De-linearization of Data Placement to Improve Deduplication Performance Reducing The De-linearization of Data Placement to Improve Deduplication Performance Yujuan Tan 1, Zhichao Yan 2, Dan Feng 2, E. H.-M. Sha 1,3 1 School of Computer Science & Technology, Chongqing University

More information

FAWN as a Service. 1 Introduction. Jintian Liang CS244B December 13, 2017

FAWN as a Service. 1 Introduction. Jintian Liang CS244B December 13, 2017 Liang 1 Jintian Liang CS244B December 13, 2017 1 Introduction FAWN as a Service FAWN, an acronym for Fast Array of Wimpy Nodes, is a distributed cluster of inexpensive nodes designed to give users a view

More information

A New Approach to Determining the Time-Stamping Counter's Overhead on the Pentium Pro Processors *

A New Approach to Determining the Time-Stamping Counter's Overhead on the Pentium Pro Processors * A New Approach to Determining the Time-Stamping Counter's Overhead on the Pentium Pro Processors * Hsin-Ta Chiao and Shyan-Ming Yuan Department of Computer and Information Science National Chiao Tung University

More information

Deduplication Storage System

Deduplication Storage System Deduplication Storage System Kai Li Charles Fitzmorris Professor, Princeton University & Chief Scientist and Co-Founder, Data Domain, Inc. 03/11/09 The World Is Becoming Data-Centric CERN Tier 0 Business

More information

Distributed Systems. Lec 10: Distributed File Systems GFS. Slide acks: Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung

Distributed Systems. Lec 10: Distributed File Systems GFS. Slide acks: Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Distributed Systems Lec 10: Distributed File Systems GFS Slide acks: Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung 1 Distributed File Systems NFS AFS GFS Some themes in these classes: Workload-oriented

More information

The Google File System

The Google File System October 13, 2010 Based on: S. Ghemawat, H. Gobioff, and S.-T. Leung: The Google file system, in Proceedings ACM SOSP 2003, Lake George, NY, USA, October 2003. 1 Assumptions Interface Architecture Single

More information

Design and Implementation of a Random Access File System for NVRAM

Design and Implementation of a Random Access File System for NVRAM This article has been accepted and published on J-STAGE in advance of copyediting. Content is final as presented. IEICE Electronics Express, Vol.* No.*,*-* Design and Implementation of a Random Access

More information

DELL EMC DATA DOMAIN SISL SCALING ARCHITECTURE

DELL EMC DATA DOMAIN SISL SCALING ARCHITECTURE WHITEPAPER DELL EMC DATA DOMAIN SISL SCALING ARCHITECTURE A Detailed Review ABSTRACT While tape has been the dominant storage medium for data protection for decades because of its low cost, it is steadily

More information

WAN Optimized Replication of Backup Datasets Using Stream-Informed Delta Compression

WAN Optimized Replication of Backup Datasets Using Stream-Informed Delta Compression WAN Optimized Replication of Backup Datasets Using Stream-Informed Delta Compression Philip Shilane, Mark Huang, Grant Wallace, & Windsor Hsu Backup Recovery Systems Division EMC Corporation Introduction

More information

COS 318: Operating Systems. NSF, Snapshot, Dedup and Review

COS 318: Operating Systems. NSF, Snapshot, Dedup and Review COS 318: Operating Systems NSF, Snapshot, Dedup and Review Topics! NFS! Case Study: NetApp File System! Deduplication storage system! Course review 2 Network File System! Sun introduced NFS v2 in early

More information

Adaptive Compressed Caching: Embracing and extending for the Linux 2.6 kernel

Adaptive Compressed Caching: Embracing and extending for the Linux 2.6 kernel UNIVERSITY OF OSLO Department of Informatics Adaptive Compressed Caching: Embracing and extending for the Linux 2.6 kernel Master Thesis Asbjørn Sannes asbjorsa@ifi.uio.no May 15, 2008 Abstract Applications

More information

Comparison of Storage Protocol Performance ESX Server 3.5

Comparison of Storage Protocol Performance ESX Server 3.5 Performance Study Comparison of Storage Protocol Performance ESX Server 3.5 This study provides performance comparisons of various storage connection options available to VMware ESX Server. We used the

More information

The Google File System

The Google File System The Google File System Sanjay Ghemawat, Howard Gobioff and Shun Tak Leung Google* Shivesh Kumar Sharma fl4164@wayne.edu Fall 2015 004395771 Overview Google file system is a scalable distributed file system

More information

ENCRYPTED DATA MANAGEMENT WITH DEDUPLICATION IN CLOUD COMPUTING

ENCRYPTED DATA MANAGEMENT WITH DEDUPLICATION IN CLOUD COMPUTING ENCRYPTED DATA MANAGEMENT WITH DEDUPLICATION IN CLOUD COMPUTING S KEERTHI 1*, MADHAVA REDDY A 2* 1. II.M.Tech, Dept of CSE, AM Reddy Memorial College of Engineering & Technology, Petlurivaripalem. 2. Assoc.

More information

Flash Drive Emulation

Flash Drive Emulation Flash Drive Emulation Eric Aderhold & Blayne Field aderhold@cs.wisc.edu & bfield@cs.wisc.edu Computer Sciences Department University of Wisconsin, Madison Abstract Flash drives are becoming increasingly

More information

GPUfs: Integrating a File System with GPUs. Yishuai Li & Shreyas Skandan

GPUfs: Integrating a File System with GPUs. Yishuai Li & Shreyas Skandan GPUfs: Integrating a File System with GPUs Yishuai Li & Shreyas Skandan Von Neumann Architecture Mem CPU I/O Von Neumann Architecture Mem CPU I/O slow fast slower Direct Memory Access Mem CPU I/O slow

More information

!! What is virtual memory and when is it useful? !! What is demand paging? !! When should pages in memory be replaced?

!! What is virtual memory and when is it useful? !! What is demand paging? !! When should pages in memory be replaced? Chapter 10: Virtual Memory Questions? CSCI [4 6] 730 Operating Systems Virtual Memory!! What is virtual memory and when is it useful?!! What is demand paging?!! When should pages in memory be replaced?!!

More information

Accelerating Restore and Garbage Collection in Deduplication-based Backup Systems via Exploiting Historical Information

Accelerating Restore and Garbage Collection in Deduplication-based Backup Systems via Exploiting Historical Information Accelerating Restore and Garbage Collection in Deduplication-based Backup Systems via Exploiting Historical Information Min Fu, Dan Feng, Yu Hua, Xubin He, Zuoning Chen *, Wen Xia, Fangting Huang, Qing

More information

Enhancing the Performance of High Availability Lightweight Live Migration

Enhancing the Performance of High Availability Lightweight Live Migration Enhancing the Performance of High Availability Lightweight Live Migration Peng Lu 1, Binoy Ravindran 1, and Changsoo Kim 2 1 ECE Dept., Virginia Tech {lvpeng,binoy}@vt.edu 2 ETRI, Daejeon, South Korea

More information

Final Examination CS 111, Fall 2016 UCLA. Name:

Final Examination CS 111, Fall 2016 UCLA. Name: Final Examination CS 111, Fall 2016 UCLA Name: This is an open book, open note test. You may use electronic devices to take the test, but may not access the network during the test. You have three hours

More information

CPHash: A Cache-Partitioned Hash Table Zviad Metreveli, Nickolai Zeldovich, and M. Frans Kaashoek

CPHash: A Cache-Partitioned Hash Table Zviad Metreveli, Nickolai Zeldovich, and M. Frans Kaashoek Computer Science and Artificial Intelligence Laboratory Technical Report MIT-CSAIL-TR-211-51 November 26, 211 : A Cache-Partitioned Hash Table Zviad Metreveli, Nickolai Zeldovich, and M. Frans Kaashoek

More information

OPERATING SYSTEM. Chapter 9: Virtual Memory

OPERATING SYSTEM. Chapter 9: Virtual Memory OPERATING SYSTEM Chapter 9: Virtual Memory Chapter 9: Virtual Memory Background Demand Paging Copy-on-Write Page Replacement Allocation of Frames Thrashing Memory-Mapped Files Allocating Kernel Memory

More information

Reducing Disk Latency through Replication

Reducing Disk Latency through Replication Gordon B. Bell Morris Marden Abstract Today s disks are inexpensive and have a large amount of capacity. As a result, most disks have a significant amount of excess capacity. At the same time, the performance

More information

COS 318: Operating Systems. File Systems. Topics. Evolved Data Center Storage Hierarchy. Traditional Data Center Storage Hierarchy

COS 318: Operating Systems. File Systems. Topics. Evolved Data Center Storage Hierarchy. Traditional Data Center Storage Hierarchy Topics COS 318: Operating Systems File Systems hierarchy File system abstraction File system operations File system protection 2 Traditional Data Center Hierarchy Evolved Data Center Hierarchy Clients

More information

CSE 410 Final Exam Sample Solution 6/08/10

CSE 410 Final Exam Sample Solution 6/08/10 Question 1. (12 points) (caches) (a) One choice in designing cache memories is to pick a block size. Which of the following do you think would be the most reasonable size for cache blocks on a computer

More information

Memory. Objectives. Introduction. 6.2 Types of Memory

Memory. Objectives. Introduction. 6.2 Types of Memory Memory Objectives Master the concepts of hierarchical memory organization. Understand how each level of memory contributes to system performance, and how the performance is measured. Master the concepts

More information

15 418/618 Project Final Report Concurrent Lock free BST

15 418/618 Project Final Report Concurrent Lock free BST 15 418/618 Project Final Report Concurrent Lock free BST Names: Swapnil Pimpale, Romit Kudtarkar AndrewID: spimpale, rkudtark 1.0 SUMMARY We implemented two concurrent binary search trees (BSTs): a fine

More information

Chapter 8 & Chapter 9 Main Memory & Virtual Memory

Chapter 8 & Chapter 9 Main Memory & Virtual Memory Chapter 8 & Chapter 9 Main Memory & Virtual Memory 1. Various ways of organizing memory hardware. 2. Memory-management techniques: 1. Paging 2. Segmentation. Introduction Memory consists of a large array

More information

Near Memory Key/Value Lookup Acceleration MemSys 2017

Near Memory Key/Value Lookup Acceleration MemSys 2017 Near Key/Value Lookup Acceleration MemSys 2017 October 3, 2017 Scott Lloyd, Maya Gokhale Center for Applied Scientific Computing This work was performed under the auspices of the U.S. Department of Energy

More information

ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective. Part I: Operating system overview: Memory Management

ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective. Part I: Operating system overview: Memory Management ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective Part I: Operating system overview: Memory Management 1 Hardware background The role of primary memory Program

More information

Dynamic Translator-Based Virtualization

Dynamic Translator-Based Virtualization Dynamic Translator-Based Virtualization Yuki Kinebuchi 1,HidenariKoshimae 1,ShuichiOikawa 2, and Tatsuo Nakajima 1 1 Department of Computer Science, Waseda University {yukikine, hide, tatsuo}@dcl.info.waseda.ac.jp

More information

Storage and File System

Storage and File System COS 318: Operating Systems Storage and File System Andy Bavier Computer Science Department Princeton University http://www.cs.princeton.edu/courses/archive/fall10/cos318/ Topics Storage hierarchy File

More information

Chapter 8: Virtual Memory. Operating System Concepts

Chapter 8: Virtual Memory. Operating System Concepts Chapter 8: Virtual Memory Silberschatz, Galvin and Gagne 2009 Chapter 8: Virtual Memory Background Demand Paging Copy-on-Write Page Replacement Allocation of Frames Thrashing Memory-Mapped Files Allocating

More information

Alternative Approaches for Deduplication in Cloud Storage Environment

Alternative Approaches for Deduplication in Cloud Storage Environment International Journal of Computational Intelligence Research ISSN 0973-1873 Volume 13, Number 10 (2017), pp. 2357-2363 Research India Publications http://www.ripublication.com Alternative Approaches for

More information

Deduplication and Incremental Accelleration in Bacula with NetApp Technologies. Peter Buschman EMEA PS Consultant September 25th, 2012

Deduplication and Incremental Accelleration in Bacula with NetApp Technologies. Peter Buschman EMEA PS Consultant September 25th, 2012 Deduplication and Incremental Accelleration in Bacula with NetApp Technologies Peter Buschman EMEA PS Consultant September 25th, 2012 1 NetApp and Bacula Systems Bacula Systems became a NetApp Developer

More information

Delta Compressed and Deduplicated Storage Using Stream-Informed Locality

Delta Compressed and Deduplicated Storage Using Stream-Informed Locality Delta Compressed and Deduplicated Storage Using Stream-Informed Locality Philip Shilane, Grant Wallace, Mark Huang, and Windsor Hsu Backup Recovery Systems Division EMC Corporation Abstract For backup

More information

PAC094 Performance Tips for New Features in Workstation 5. Anne Holler Irfan Ahmad Aravind Pavuluri

PAC094 Performance Tips for New Features in Workstation 5. Anne Holler Irfan Ahmad Aravind Pavuluri PAC094 Performance Tips for New Features in Workstation 5 Anne Holler Irfan Ahmad Aravind Pavuluri Overview of Talk Virtual machine teams 64-bit guests SMP guests e1000 NIC support Fast snapshots Virtual

More information

Memory Management. Disclaimer: some slides are adopted from book authors slides with permission 1

Memory Management. Disclaimer: some slides are adopted from book authors slides with permission 1 Memory Management Disclaimer: some slides are adopted from book authors slides with permission 1 CPU management Roadmap Process, thread, synchronization, scheduling Memory management Virtual memory Disk

More information

Deduplication File System & Course Review

Deduplication File System & Course Review Deduplication File System & Course Review Kai Li 12/13/13 Topics u Deduplication File System u Review 12/13/13 2 Storage Tiers of A Tradi/onal Data Center $$$$ Mirrored storage $$$ Dedicated Fibre Clients

More information

Pay Migration Tax to Homeland: Anchor-based Scalable Reference Counting for Multicores

Pay Migration Tax to Homeland: Anchor-based Scalable Reference Counting for Multicores Pay Migration Tax to Homeland: Anchor-based Scalable Reference Counting for Multicores Seokyong Jung, Jongbin Kim, Minsoo Ryu, Sooyong Kang, Hyungsoo Jung Hanyang University 1 Reference counting It is

More information

3/3/2014! Anthony D. Joseph!!CS162! UCB Spring 2014!

3/3/2014! Anthony D. Joseph!!CS162! UCB Spring 2014! Post Project 1 Class Format" CS162 Operating Systems and Systems Programming Lecture 11 Page Allocation and Replacement" Mini quizzes after each topic Not graded Simple True/False Immediate feedback for

More information

Online Version Only. Book made by this file is ILLEGAL. Design and Implementation of Binary File Similarity Evaluation System. 1.

Online Version Only. Book made by this file is ILLEGAL. Design and Implementation of Binary File Similarity Evaluation System. 1. , pp.1-10 http://dx.doi.org/10.14257/ijmue.2014.9.1.01 Design and Implementation of Binary File Similarity Evaluation System Sun-Jung Kim 2, Young Jun Yoo, Jungmin So 1, Jeong Gun Lee 1, Jin Kim 1 and

More information

An Application Awareness Local Source and Global Source De-Duplication with Security in resource constraint based Cloud backup services

An Application Awareness Local Source and Global Source De-Duplication with Security in resource constraint based Cloud backup services An Application Awareness Local Source and Global Source De-Duplication with Security in resource constraint based Cloud backup services S.Meghana Assistant Professor, Dept. of IT, Vignana Bharathi Institute

More information

Outline. V Computer Systems Organization II (Honors) (Introductory Operating Systems) Advantages of Multi-level Page Tables

Outline. V Computer Systems Organization II (Honors) (Introductory Operating Systems) Advantages of Multi-level Page Tables Outline V22.0202-001 Computer Systems Organization II (Honors) (Introductory Operating Systems) Lecture 15 Memory Management (cont d) Virtual Memory March 30, 2005 Announcements Lab 4 due next Monday (April

More information

Chapter 9: Virtual Memory

Chapter 9: Virtual Memory Chapter 9: Virtual Memory Silberschatz, Galvin and Gagne 2013 Chapter 9: Virtual Memory Background Demand Paging Copy-on-Write Page Replacement Allocation of Frames Thrashing Memory-Mapped Files Allocating

More information

VIRTUAL MEMORY READING: CHAPTER 9

VIRTUAL MEMORY READING: CHAPTER 9 VIRTUAL MEMORY READING: CHAPTER 9 9 MEMORY HIERARCHY Core! Processor! Core! Caching! Main! Memory! (DRAM)!! Caching!! Secondary Storage (SSD)!!!! Secondary Storage (Disk)! L cache exclusive to a single

More information

There are 10 questions in total. Please write your SID on each page.

There are 10 questions in total. Please write your SID on each page. Name: SID: Department of EECS - University of California at Berkeley EECS122 - Introduction to Communication Networks - Spring 2005 to the Final: 5/20/2005 There are 10 questions in total. Please write

More information

A Comparison of Relativistic and Reader-Writer Locking Approaches to Shared Data Access

A Comparison of Relativistic and Reader-Writer Locking Approaches to Shared Data Access A Comparison of Relativistic and Reader-Writer Locking Approaches to Shared Data Access Philip W. Howard, Josh Triplett, and Jonathan Walpole Portland State University Abstract. This paper explores the

More information

COSC3330 Computer Architecture Lecture 20. Virtual Memory

COSC3330 Computer Architecture Lecture 20. Virtual Memory COSC3330 Computer Architecture Lecture 20. Virtual Memory Instructor: Weidong Shi (Larry), PhD Computer Science Department University of Houston Virtual Memory Topics Reducing Cache Miss Penalty (#2) Use

More information

Multi-level Translation. CS 537 Lecture 9 Paging. Example two-level page table. Multi-level Translation Analysis

Multi-level Translation. CS 537 Lecture 9 Paging. Example two-level page table. Multi-level Translation Analysis Multi-level Translation CS 57 Lecture 9 Paging Michael Swift Problem: what if you have a sparse address space e.g. out of GB, you use MB spread out need one PTE per page in virtual address space bit AS

More information

Ext3/4 file systems. Don Porter CSE 506

Ext3/4 file systems. Don Porter CSE 506 Ext3/4 file systems Don Porter CSE 506 Logical Diagram Binary Formats Memory Allocators System Calls Threads User Today s Lecture Kernel RCU File System Networking Sync Memory Management Device Drivers

More information

Scalable Concurrent Hash Tables via Relativistic Programming

Scalable Concurrent Hash Tables via Relativistic Programming Scalable Concurrent Hash Tables via Relativistic Programming Josh Triplett September 24, 2009 Speed of data < Speed of light Speed of light: 3e8 meters/second Processor speed: 3 GHz, 3e9 cycles/second

More information

Program Counter Based Pattern Classification in Pattern Based Buffer Caching

Program Counter Based Pattern Classification in Pattern Based Buffer Caching Purdue University Purdue e-pubs ECE Technical Reports Electrical and Computer Engineering 1-12-2004 Program Counter Based Pattern Classification in Pattern Based Buffer Caching Chris Gniady Y. Charlie

More information

OS Virtualization. Why Virtualize? Introduction. Virtualization Basics 12/10/2012. Motivation. Types of Virtualization.

OS Virtualization. Why Virtualize? Introduction. Virtualization Basics 12/10/2012. Motivation. Types of Virtualization. Virtualization Basics Motivation OS Virtualization CSC 456 Final Presentation Brandon D. Shroyer Types of Virtualization Process virtualization (Java) System virtualization (classic, hosted) Emulation

More information

Improving Http-Server Performance by Adapted Multithreading

Improving Http-Server Performance by Adapted Multithreading Improving Http-Server Performance by Adapted Multithreading Jörg Keller LG Technische Informatik II FernUniversität Hagen 58084 Hagen, Germany email: joerg.keller@fernuni-hagen.de Olaf Monien Thilo Lardon

More information

Heckaton. SQL Server's Memory Optimized OLTP Engine

Heckaton. SQL Server's Memory Optimized OLTP Engine Heckaton SQL Server's Memory Optimized OLTP Engine Agenda Introduction to Hekaton Design Consideration High Level Architecture Storage and Indexing Query Processing Transaction Management Transaction Durability

More information

Lecture #15: Translation, protection, sharing

Lecture #15: Translation, protection, sharing Lecture #15: Translation, protection, sharing Review -- 1 min Goals of virtual memory: protection relocation sharing illusion of infinite memory minimal overhead o space o time Last time: we ended with

More information

Content Addressable Storage Provider in Linux. Vesselin Dimitrov. Project Advisor: Jessen Havill Department of Mathematics and Computer Science

Content Addressable Storage Provider in Linux. Vesselin Dimitrov. Project Advisor: Jessen Havill Department of Mathematics and Computer Science Content Addressable Storage Provider in Linux Vesselin Dimitrov Project Advisor: Jessen Havill Department of Mathematics and Computer Science Permission to make digital/hard copy of part or all of this

More information

FaRM: Fast Remote Memory

FaRM: Fast Remote Memory FaRM: Fast Remote Memory Problem Context DRAM prices have decreased significantly Cost effective to build commodity servers w/hundreds of GBs E.g. - cluster with 100 machines can hold tens of TBs of main

More information

CHAPTER 5 PROPAGATION DELAY

CHAPTER 5 PROPAGATION DELAY 98 CHAPTER 5 PROPAGATION DELAY Underwater wireless sensor networks deployed of sensor nodes with sensing, forwarding and processing abilities that operate in underwater. In this environment brought challenges,

More information

ChunkStash: Speeding Up Storage Deduplication using Flash Memory

ChunkStash: Speeding Up Storage Deduplication using Flash Memory ChunkStash: Speeding Up Storage Deduplication using Flash Memory Biplob Debnath +, Sudipta Sengupta *, Jin Li * * Microsoft Research, Redmond (USA) + Univ. of Minnesota, Twin Cities (USA) Deduplication

More information

CIS Operating Systems Memory Management Address Translation for Paging. Professor Qiang Zeng Spring 2018

CIS Operating Systems Memory Management Address Translation for Paging. Professor Qiang Zeng Spring 2018 CIS 3207 - Operating Systems Memory Management Address Translation for Paging Professor Qiang Zeng Spring 2018 Previous class What is logical address? Who use it? Describes a location in the logical memory

More information

Domain Level Page Sharing in Xen Virtual Machine Systems

Domain Level Page Sharing in Xen Virtual Machine Systems Domain Level Page Sharing in Xen Virtual Machine Systems Myeongjae Jeon, Euiseong Seo, Junghyun Kim, and Joonwon Lee CS Division, Korea Advanced Institute of Science and Technology {mjjeon,ses,joon}@calabkaistackr

More information

LOAD BALANCING AND DEDUPLICATION

LOAD BALANCING AND DEDUPLICATION LOAD BALANCING AND DEDUPLICATION Mr.Chinmay Chikode Mr.Mehadi Badri Mr.Mohit Sarai Ms.Kshitija Ubhe ABSTRACT Load Balancing is a method of distributing workload across multiple computing resources such

More information

ARC: An Approach to Flexible and Robust RAID Systems

ARC: An Approach to Flexible and Robust RAID Systems ARC: An Approach to Flexible and Robust RAID Systems Ba-Quy Vuong and Yiying Zhang Computer Sciences Department, University of Wisconsin-Madison Abstract RAID systems increase data storage reliability

More information

Parallel Processing for Data Deduplication

Parallel Processing for Data Deduplication Parallel Processing for Data Deduplication Peter Sobe, Denny Pazak, Martin Stiehr Faculty of Computer Science and Mathematics Dresden University of Applied Sciences Dresden, Germany Corresponding Author

More information

Operating System Performance and Large Servers 1

Operating System Performance and Large Servers 1 Operating System Performance and Large Servers 1 Hyuck Yoo and Keng-Tai Ko Sun Microsystems, Inc. Mountain View, CA 94043 Abstract Servers are an essential part of today's computing environments. High

More information

Data Modeling and Databases Ch 14: Data Replication. Gustavo Alonso, Ce Zhang Systems Group Department of Computer Science ETH Zürich

Data Modeling and Databases Ch 14: Data Replication. Gustavo Alonso, Ce Zhang Systems Group Department of Computer Science ETH Zürich Data Modeling and Databases Ch 14: Data Replication Gustavo Alonso, Ce Zhang Systems Group Department of Computer Science ETH Zürich Database Replication What is database replication The advantages of

More information

CS307: Operating Systems

CS307: Operating Systems CS307: Operating Systems Chentao Wu 吴晨涛 Associate Professor Dept. of Computer Science and Engineering Shanghai Jiao Tong University SEIEE Building 3-513 wuct@cs.sjtu.edu.cn Download Lectures ftp://public.sjtu.edu.cn

More information

CS370 Operating Systems

CS370 Operating Systems CS370 Operating Systems Colorado State University Yashwant K Malaiya Fall 2017 Lecture 21 Main Memory Slides based on Text by Silberschatz, Galvin, Gagne Various sources 1 1 FAQ Why not increase page size

More information

CS370: Operating Systems [Spring 2017] Dept. Of Computer Science, Colorado State University

CS370: Operating Systems [Spring 2017] Dept. Of Computer Science, Colorado State University Frequently asked questions from the previous class survey CS 370: OPERATING SYSTEMS [MEMORY MANAGEMENT] Matrices in Banker s algorithm Max, need, allocated Shrideep Pallickara Computer Science Colorado

More information

Memory Management. Disclaimer: some slides are adopted from book authors slides with permission 1

Memory Management. Disclaimer: some slides are adopted from book authors slides with permission 1 Memory Management Disclaimer: some slides are adopted from book authors slides with permission 1 Recap Paged MMU: Two main Issues Translation speed can be slow TLB Table size is big Multi-level page table

More information

An Integrated Synchronization and Consistency Protocol for the Implementation of a High-Level Parallel Programming Language

An Integrated Synchronization and Consistency Protocol for the Implementation of a High-Level Parallel Programming Language An Integrated Synchronization and Consistency Protocol for the Implementation of a High-Level Parallel Programming Language Martin C. Rinard (martin@cs.ucsb.edu) Department of Computer Science University

More information

Multi-level Byte Index Chunking Mechanism for File Synchronization

Multi-level Byte Index Chunking Mechanism for File Synchronization , pp.339-350 http://dx.doi.org/10.14257/ijseia.2014.8.3.31 Multi-level Byte Index Chunking Mechanism for File Synchronization Ider Lkhagvasuren, Jung Min So, Jeong Gun Lee, Jin Kim and Young Woong Ko *

More information

Optimizing the use of the Hard Disk in MapReduce Frameworks for Multi-core Architectures*

Optimizing the use of the Hard Disk in MapReduce Frameworks for Multi-core Architectures* Optimizing the use of the Hard Disk in MapReduce Frameworks for Multi-core Architectures* Tharso Ferreira 1, Antonio Espinosa 1, Juan Carlos Moure 2 and Porfidio Hernández 2 Computer Architecture and Operating

More information

dedupv1: Improving Deduplication Throughput using Solid State Drives (SSD)

dedupv1: Improving Deduplication Throughput using Solid State Drives (SSD) University Paderborn Paderborn Center for Parallel Computing Technical Report dedupv1: Improving Deduplication Throughput using Solid State Drives (SSD) Dirk Meister Paderborn Center for Parallel Computing

More information

NUMA replicated pagecache for Linux

NUMA replicated pagecache for Linux NUMA replicated pagecache for Linux Nick Piggin SuSE Labs January 27, 2008 0-0 Talk outline I will cover the following areas: Give some NUMA background information Introduce some of Linux s NUMA optimisations

More information

Process size is independent of the main memory present in the system.

Process size is independent of the main memory present in the system. Hardware control structure Two characteristics are key to paging and segmentation: 1. All memory references are logical addresses within a process which are dynamically converted into physical at run time.

More information

Basic Memory Management. Basic Memory Management. Address Binding. Running a user program. Operating Systems 10/14/2018 CSC 256/456 1

Basic Memory Management. Basic Memory Management. Address Binding. Running a user program. Operating Systems 10/14/2018 CSC 256/456 1 Basic Memory Management Program must be brought into memory and placed within a process for it to be run Basic Memory Management CS 256/456 Dept. of Computer Science, University of Rochester Mono-programming

More information

Lecture 6 Consistency and Replication

Lecture 6 Consistency and Replication Lecture 6 Consistency and Replication Prof. Wilson Rivera University of Puerto Rico at Mayaguez Electrical and Computer Engineering Department Outline Data-centric consistency Client-centric consistency

More information

CS 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring Lecture 15: Caching: Demand Paged Virtual Memory

CS 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring Lecture 15: Caching: Demand Paged Virtual Memory CS 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring 2003 Lecture 15: Caching: Demand Paged Virtual Memory 15.0 Main Points: Concept of paging to disk Replacement policies

More information

A Connectionless Approach to Intra- and Inter-Domain Traffic Engineering

A Connectionless Approach to Intra- and Inter-Domain Traffic Engineering A Connectionless Approach to Intra- and Inter-Domain Traffic Engineering Hema T. Kaur, Shivkumar Kalyanaraman ECSE Department, Rensselaer Polytechnic Institute, Troy, NY-12180 {hema,shivkuma}@networks.ecse.rpi.edu

More information

Virtual Memory COMPSCI 386

Virtual Memory COMPSCI 386 Virtual Memory COMPSCI 386 Motivation An instruction to be executed must be in physical memory, but there may not be enough space for all ready processes. Typically the entire program is not needed. Exception

More information

LINUX OPERATING SYSTEM Submitted in partial fulfillment of the requirement for the award of degree of Bachelor of Technology in Computer Science

LINUX OPERATING SYSTEM Submitted in partial fulfillment of the requirement for the award of degree of Bachelor of Technology in Computer Science A Seminar report On LINUX OPERATING SYSTEM Submitted in partial fulfillment of the requirement for the award of degree of Bachelor of Technology in Computer Science SUBMITTED TO: www.studymafia.org SUBMITTED

More information

Copyright 2013 Thomas W. Doeppner. IX 1

Copyright 2013 Thomas W. Doeppner. IX 1 Copyright 2013 Thomas W. Doeppner. IX 1 If we have only one thread, then, no matter how many processors we have, we can do only one thing at a time. Thus multiple threads allow us to multiplex the handling

More information

Distributed File Systems II

Distributed File Systems II Distributed File Systems II To do q Very-large scale: Google FS, Hadoop FS, BigTable q Next time: Naming things GFS A radically new environment NFS, etc. Independence Small Scale Variety of workloads Cooperation

More information

Assignment 5. Georgia Koloniari

Assignment 5. Georgia Koloniari Assignment 5 Georgia Koloniari 2. "Peer-to-Peer Computing" 1. What is the definition of a p2p system given by the authors in sec 1? Compare it with at least one of the definitions surveyed in the last

More information

A DEDUPLICATION-INSPIRED FAST DELTA COMPRESSION APPROACH W EN XIA, HONG JIANG, DA N FENG, LEI T I A N, M I N FU, YUKUN Z HOU

A DEDUPLICATION-INSPIRED FAST DELTA COMPRESSION APPROACH W EN XIA, HONG JIANG, DA N FENG, LEI T I A N, M I N FU, YUKUN Z HOU A DEDUPLICATION-INSPIRED FAST DELTA COMPRESSION APPROACH W EN XIA, HONG JIANG, DA N FENG, LEI T I A N, M I N FU, YUKUN Z HOU PRESENTED BY ROMAN SHOR Overview Technics of data reduction in storage systems:

More information