School of Computer and Information Science

Transcription

1 School of Computer and Information Science CIS Research Placement Report Multiple threads in floating-point sort operations Name: Quang Do Date: 8/6/2012 Supervisor: Grant Wigley Abstract Despite the vast improvements in CPU design and the inclusion of multiple cores in even the low end processors, little has been done to improve the performance of sorting algorithms in this respect. This paper aims to justify the benefits of the use of multiple cores in these operations. It will also show how to implement these sorting operations.

2 1. Introduction Multi-threaded applications are starting to become a necessity as hardware development shifts from speed to multiple cores. Even though they have been around for quite a while, many applications and operating systems do not yet efficiently use them. The purpose of this paper is to show the possible performance increases by multi-threading different sorting algorithms. This is achieved by showing uses of different sorting algorithms and to display the differences between single threaded and multi-threaded versions of the same algorithms. An overview of the steps taken to achieve the data is given along with the resulting data. ing algorithms make up the basis of many applications used in the real world. Since their uses are so abundant, it is extremely useful to be able to speed up their processing time. This results in more efficient applications that respond and output data at an increased rate and throughput. Although there are many advantages to multi-threading an algorithm, which is the reason why so many technologies and hardware platforms are moving to multiple core processors, there are also several disadvantages. 1.1 Floating Point Numbers Floating-point numbers, in this paper, are 32 bit single precision numbers. All sorting operations performed are on these 4 byte values. 1.2 ing Algorithms There are several sorting algorithms that are used within this document. Most of these are commonly used to display the differences between sorting efficiency instead of real world uses. Insertion sort is used to display the performance changes for an algorithm which does not have an average processing time of but instead. Quick sort is an algorithm which is used in Java for its sorting operations. As such it is often used in real world applications and games. It has an average case performance of. Shell sort is yet another sorting algorithm with a different average processing time. It has an average performance case of.

3 All of these algorithms, though, are naturally sequential and single threaded. They rely on the data within the list to be sorted to be in one place and be unchanged as the processing is done to sort the list. In order to allow for comparisons of different single threaded sorting algorithms when they are multi-threaded, the use of a parallel sorting algorithm is required. In this paper, bucket sort is used to allow a useful comparison. Bucket sort works by first splitting the array of objects to be sorted, in this case float point numbers, by specified criteria. The aim is to have each of the buckets containing a similar number of objects. For a random set of data, this may be difficult to achieve. For a completely random set of data, the data could be split by numbers between certain ranges. Once the splitting of data is done, a different sorting algorithm would be used of each of the buckets until fully sorted. Since the buckets contain within them fully sorted data of the appropriate order, they can be simply joined together to return the completed sorted array. Naturally, the first division of data into buckets incurs a loss in performance, so the algorithm does not scale the individual sorting algorithms used perfectly. This is the downside of using a parallel sorting algorithm such as bucket sort. But since each of the algorithms incurs the same loss in performance, the difference is negligible. Figure 1.1 Bucket of Percentages

4 Figure 1.1 shows an example of bucket sort on percentages. The data has been arbitrarily chosen to be split equally between 0 and 100. The number of buckets has been decided as 4. This may be because the system doing the processing is a quad core, quad thread processor. Each of the buckets can then be sorted on each of the threads. The circles represent where the individual buckets data is sorted. This can consist of any single or even multi-thread sorting algorithm. This area is where each of the sorting algorithms in Figure 1.2 is used. The placing of data into buckets in a bucket sort takes a constant O(n) time. ing Algorithm Best Case Performance Worst Case Performance Quick Shell ( ) Insertion Average Performance Figure 1.2 ing Algorithms Memory Usage

5 1.3 Insertion Insertion sort is a very simple sorting algorithm that uses comparisons to sort an array. Its speed depends on the size of the array; larger arrays take a significantly larger amount of time to sort. It is also efficient on arrays that are already partially sorted and very easy to implement. Insertion sort works by methodically going along an array, starting at position 0, comparing the element with the rest of the elements before that position, and putting it in the right position. For example: [ ] Starting array [ ] Start at first element [ ] Move to next element. It is larger so no sorting is required. [ ] Move to next element. Element is smaller so it is inserted at the correct position. [ ] Move to next element. It is also smaller so it is inserted at the correct position. [ ] Completed sorted array. The best case for insertion sort is when the array is already sorted. It then has a running time of O(n). The worst case for insertion sort is when it is used on array already sorted in reverse order. This has a running time of O(n 2 ).

6 1.4 Quick Quick sort is said to be the fastest sorting algorithm for sorting random sets of data. This makes it ideal to be used for testing of floating point sorting. Quick sort works by: A pivot point is chosen, in the case of our algorithm, we use the middle element. It can have any value, even if it is not within the array. Any values in the array that are greater than the pivot value are put on the right of the pivot point. Values that are smaller are put in the left side and values that are equal are left in either side. The algorithm then recursively sorts both sides of the pivot. [ ] Starting array [ ] Choose pivot point [ ] Arrange elements [[2 1] [4 3]] Recursively quick sort [[1 2] [3 4]] [ ] ed array The best case for quick sort is when each partition is the same size after each step. This is still considered O( ), which is the same as its average case. The worst case scenario is O(n 2 ) for when the array is when the last element in the array is selected as the pivot and the array is already sorted.

7 1.5 Shell Shell sort is a sorting algorithm that, like the other algorithms discussed in this paper, uses comparison based sorting. It is an improvement upon insertion sort that allows elements to be exchanged even if they are not adjacently located. A gap size is chosen. For this paper, a gap size of half of the length of the array is used. Each element at the gap position is added to a sub array and sorted. The gap is then reduced, in our case, it is halved. This is continuously done until the gap size is set to 1, effectively turning into an insertion sort on an almost completely sorted array. [ ] Gap size is 3. [ ] the array of [3 5] [ ] the array of [1 2] [ ] the array of [6 4] = [4 6] = [ ] Gap size is 2 [ ] the array of [3 4 2] = [2 3 4] [ ] the array of [1 5 6] = [ ] Gap size is 1 [ ] = [ ] Finished array The best case for shell sort is when the elements are already sorted. This is O(n). The worst case scenario for this version of shell sort is the same as insertion sort. That is when the data set is sorted in the reverse order. It is also O(n 2 ).

8 2. Results To be able to get a fair comparison of sorting algorithms when multi-threaded, comparisons need to be made to their single threaded processing times. The sorting algorithms being tested are: Insertion, Quick and Shell. The system used to perform all these sorting operations is a 64-bit Windows 7 Machine with an Intel Xeon 3.47GHz. This CPU is a hyper-threaded quad core, meaning it has a total of 8 threads. The machine uses a WD Black 1TB hard drive as the primary drive and has 6 GB of DDR3 RAM. The sorting algorithms are programmed in Java and run using the Java Virtual Machine. This incurs only a minor performance loss in these trials. Threads are used in Java to perform the sorting of the float arrays. Each thread is considered a bucket in the bucket sort algorithm. On each of these threads, one of the following sorting algorithms is performed. The resulting time taken is then compared with a sequential single threaded form of the algorithm. All times are first measured in nanoseconds for the greatest accuracy and then converted to the appropriate value. The results are organized from 8 threads to one thread in order to easily show the performance changes. The floating point numbers are generated using a pseudo-random number generator.

9 No. of Objects No. of Objects Using 8 Threads: ing Algorithms (8 Threads) Objects 1000 Objects 100 Objects Shell Quick Insertion Time (Seconds) ing Algorithms (8 Threads) Objects Objects Objects Shell Quick Insertion Time (Seconds) With fewer objects, insertion sort performs the fastest with 8 threads on the quad core machine. At 1000 objects, insertion sort already starts to fall to the other sorting algorithms. It regains its place at objects, remaining similar with shell sort in speed while quick sort is significantly slower. Shell sort begins to show its speed when objects number and over. Insertion sort rapidly becomes slower than the other algorithms, eventually at 10 million objects; it is unable to sort the float array under a minute. Quick sort is the fastest algorithm at 10 million objects.

10 No. of Objects No. of Objects Using 4 Threads ing Algorithms (4 Threads) Objects 1000 Objects 100 Objects Shell Quick Insertion Time (Seconds) ing Algorithms (4 Threads) Objects Objects Objects Shell Quick Insertion Time (Seconds) Unlike in the results above, where insertion sort is able to best some of the other sorting algorithms with the lower number of elements; it manages to come last in every trial. This is due to the use of 4 threads instead of 8, severely limiting its speed as it is an O(n 2 ) algorithm. With a million objects, insertion sort takes 28 seconds compared to a bit over 5 seconds with 8 threads. This is a reduction in speed of about 511%. Quick sort, once again, is the leader in speed when a large number of elements are required to be sorted.

11 No. of Objects No. of Objects Using 2 Threads ing Algorithms (2 Threads) Objects 1000 Objects 100 Objects Shell Quick Insertion Time (Seconds) ing Algorithms (2 Threads) Objects Objects Objects Shell Quick Insertion Time (Seconds) Using just 2 threads, insertion sort, once again, suffers even more performance loss. It is unable to sort even a million objects in under a minute. The time it takes insertion sort to sort objects is comparable to that of shell sort sorting 10 million objects, a speed difference of 100 times. Shell sort is the fastest using 2 threads at 1000 objects and under. Quick sort then quickly takes the lead from or more elements.

12 No. of Objects No. of Objects Using 1 Thread ing Algorithms (1 Thread) Objects 1000 Objects 100 Objects Shell Quick Insertion Time (Seconds) ing Algorithms (1 Thread) Objects Objects Objects Shell Quick Insertion Time (Seconds) By sorting without using bucket sort, each of the algorithms performs much faster at 1000 and less objects. At objects, insertion sort already shows a massive increase in processing time. At one thread, this effect is more pronounced and noticeable at an earlier stage. The time it takes to sort objects is twice as long as that of shell sort sorting a 10 million object array, a performance decrease of around 200 times. Quick sort is once again the clear victor in terms of speed. Shell sort solidly maintains its second place position.

13 3. Conclusion Based on the results (viewable in appendix A), it can be concluded that: - With around 1000 objects or less to be sorted, the overhead from multithreaded sorting makes it inefficient to multi-thread these sorts; the speed of a single-threaded sorting algorithm is much faster than a multithreaded one at this stage. o Shell sort and quick sort both have similar sorting times at this stage, so their use would be recommended. Although insertion sort is slower than these two, it is not significantly slower at this stage. - At numbers greater than 1000, multi-threading starts to have an effect. o Insertion sort is clearly the slowest by far of these three sorting algorithms, achieving times that are many times greater than those of the other two. o Even though insertion sort gains significant speed increases every time extra threads are added, this is still not enough for it to compete with the other more efficient sorting algorithms. o Single threaded sorting is still mainly faster, as with each thread increase, the processing overhead also increases. - Once there are elements are more in the floating point array, the power of multi-threading really starts to show. o Insertion sort, from 1 thread to 2 threads shows a performance increase of 400%. Quick sort and shell sort also show an improvement along the lines of 20 to 50%. o At 4 threads, insertion sort is over 1200% faster than the single threaded variant. At 8 threads, this improvement is over 4800%. o With 4 threads, an improvement of 188% is noticed with Quick sort. Shell sort is sped up by 203%. o At 8 threads, the performance of Quick sort and Shell sort is actually decreased from 4 threads.

14 - With elements, a single threaded insertion sort is unable to sort this in under a minute. o Both Quick sort and Shell sort suffer from a loss of performance when compared to their 4 thread counterparts. Quick sort suffers most heavily, becoming even slower than its single core version. - At extremely high numbers of : o Quick sort at 4 threads is the fastest overall, taking an average of a bit over half a second to sort the array. o The fastest form of shell sort is also with 4 threads, taking about 50% longer than Quick sort. It seems that 4 threads are the ideal method for multi-threading sorting algorithms on CPUs. This gave the best performance based on the results. The trend seems to be that more threads are better for more elements. This means that for arrays of data over the that was tested, it may be more efficient to increase the number of threads. This also means that insertion sort may even be a viable solution, given enough threads; although this does mean a great deal of overhead due to the large amount of threads. The improvement may even be more pronounced if implemented on a GPU as they are able to have hundreds of cores.

15 4. Bibliography ing Algorithms Compared - Cprogramming.com ing Algorithms Compared - Cprogramming.com. Available at: [Accessed 21 May 2012]. What different sorting algorithms sound like - YouTube What different sorting algorithms sound like - YouTube. Available at: [Accessed 1 June 2012]. Bucket and radix sorting Bucket and radix sorting Available at: [Accessed 21 May 2012].

16 5. Appendix A Floating point numbers compared with Integers The sorting of floating point numbers may at first seem like a simple task, but the truth is that there is much more processing required to sort a set of floating point numbers when compared to integers. Due to each number having up to 32 bits of information, comparisons take a significant amount of time and logical processing. This also means that the algorithm chosen to perform this task needs to be suitable and efficient. Results in graph form These results are reduced to 5 decimal places to ensure clarity. The original results were taken with an accuracy of 9 decimal places. 8 Threads Insertion Quick Shell Objects Objects Objects Objects Objects Objects Threads Insertion Quick Shell Objects Objects Objects Objects Objects Objects

17 2 Threads Insertion Quick Shell Objects Objects Objects Objects Objects Objects Thread Insertion Quick Shell Objects Objects Objects Objects Objects Objects