Multi-core Parallelization in Clojure - a Case Study

Transcription

1 Multi-core Parallelization in Clojure - a Case Study Johann M. Kraus and Hans A. Kestler AG Bioinformatics and Systems Biology Institute of Neural Information Processing University of Ulm

2 Outline 1. Concepts of parallel programming 2. Short introduction to Clojure 3. Multi-core parallel K-means - the case study 4. Analysis and Results 5. Summary

3 Parallel Programming Definition: Parallel programming is a form of programming where many calculations are performed simultaneously. Physical constraints prevent frequency scaling of processors This led to an increasing interest in parallel hardware and parallel programming Multi-core hardware is standard on desktop computers Parallel software can use this hardware to the full capacity

4 Large problems are divided into smaller ones and the subproblems are solved simultaneously Speedup S is limited by the fraction of parallelizable code P Amdahl s law: S = 1 1 P + P N Amdahl's law Speedup Fraction of parallelizable code 0.95 % 0.90 % 0.75 % 0.50 % Number of processors

5 Concepts of Parallel Programming Explicit vs. implicit parallelization Explicitly define communication and synchronization details for each task: MPI Java Threads Functional programming allows implicit parallelization: Parallel processing of functions Functions are free of side-effects Data is immutable

6 Distributed vs. local hardware Master - Slave parallelization (e.g. Message Passing Interface) Shared memory parallelization (e.g. Open Multi-Processing) Master CPU 0 Slave 0 Slave 1 Slave 2 CPU 4 Shared Memory CPU 1 Slave 3 Slave 4 CPU 3 CPU2 send data send result read write

7 Thread programming Threads are refinements of a process that share the same memory and can be processed separately and simultaneously Available in many languages, e.g. PThreads (C), Java Threads (Java), OpenMP Threads (C, Fortran) Execution of threads is handled by a scheduler that manages the available processing time Communication between threads is faster than communication between processes new start runnable schedule awake waiting Invoking threads is also faster than fork/join processes terminated end running block

8 Concurrency control via locking and synchronizing Concurrency control ensures that threads can access shared memory without violating data integrity The most popular approach to concurrency is locking and synchronizing public class Counter { private int value = 0 ; public synchronized void incr{ value = value + 1 ; } } Counter counter = new Counter ( ) ; counter. i n c r ( ) ; Problems might occur when using too many locks, too few locks, wrong locks, or locks in the wrong order Using locks can be fatally error-prone, e.g. dead-locks

9 Concurrency control via transactional memory Transactional memory offers a flexible alternative to lock-based concurrency control Functionality is analogous to controlling simultaneous access to database management systems Transactions ensure properties: Atomicity: Either all changes of a transaction occur or none do Consistency: Only valid changes are committed Isolation: No transaction sees the effect of other transactions Durability: Changes from transactions will be persistent

10 Software transactional memory maps transactional memory to concurrency control in parallel programming TIME :Transaction 0 :Data :Transaction 1 get data get data [consistent data] send modified data [consistent data] send modified data get data [consistent data] send modified data

11 Clojure Functional programming language hosted on the JVM Extends the code-as-data paradigm to maps and vectors Based on immutable data structures Provides built-in concurrency support via software transactional memory Completely symbiotic to Java, e.g. easy access to Java libraries Platform independent

12 Java interaction ( import ( c e r n. j e t. random. sampling RandomSamplingAssistant ) ) ( defn sample [ n k ] ( seq (. RandomSamplingAssistant ( samplearray k ( i n t a r r a y ( range n ) ) ) ) ) ) Dynamic typing and multi-methods An object is defined as the sum of what it can do (methods), rather than the sum of what it is (type hierarchy) Add type hints to speed up code ( defn da+ [#ˆ doubles a s #ˆdoubles bs ] (amap as i r e t (+ ( aget as i ) ( aget bs i ) ) ) )

13 Transactional references and STM Transactional references ensure safe coordinated synchronous changes to mutable storage locations Are bound to a single storage location for their lifetime Only allow mutation of that location to occur within transactions Available operations are ref-set, alter, and commute No explicit locking is required ( def counter ( ref 0 ) ) ( dosync ( alter counter inc ))

14 Agents Agents allow independent asynchronous change of mutable locations Are bound to a single storage location for their lifetime Only allow mutation of that location to a new state to occur as a result of an action Actions are functions that are asynchronously applied to the state of an Agent The return value of an action becomes new state of the Agent Agents are integrated with the STM ( def counter ( agent 0 ) ) ( send counter inc )

15 Cluster analysis Given a data set X compute a partition of X into k disjoint clusters C, such that: (1) How many clusters are in the data set? k i=1 C i = X (2) C i and C i C j = 3 cluster 9 cluster

16 Cluster algorithms For all possible partitions evaluate the objective function f and search the optimum. The cardinality of the set of all possible partitions is given by: Stirling numbers of the second kind S k N = 1 k! k ( ) ( 1) k i k i i=0 Runtime (nanosecond) i N Number of data points Number of clusters Cluster algorithms provide a heuristic for this search: Partitional clustering (K-means, Neuralgas, SOM, Fuzzy C-means,...) Hierarchical clustering (Divisive/agglomerative, Complete linkage,...) Graph-based clustering (Spectral clustering, NMF, Affinity propagation,...) Model-based clustering, Biclustering, Semi-supervised clustering

17 K-means algorithm Function KMeans Input : X = { x 1,..., x n } ( Data to be c l u s t e r e d ) k ( Number o f c l u s t e r s ) Output : C = { c 1,..., c k } ( C l u s t e r c e n t r o i d s ) m: X > C ( C l u s t e r a s s i g n m e n t s ) I n i t i a l i z e C ( e. g. random s e l e c t i o n from X) While C has changed For each x i i n X m( x i ) = a r g m i n j d i s t a n c e ( x i, c j) End For each c j i n C c j = c e n t r o i d ({ x i m( x i ) = j }) End End

18 Cluster Validation Evaluation requires repeated runs of clustering, e.g.: Resampled data sets Different parameters MCA-index: mean proportion of samples being consistent over different clusterings MCA = 1 n max π k i=1 A i B j

19 Estimation of the expected value of a validation index Random label: randomly assign each item to a cluster k Random partition: choose a random partition Random prototype: assign each item to its next prototype mean mca index Mean value from 100 runs cluster

20 Multi-core K-means with Clojure Split the data set into smaller pieces that are handled by agents Each cluster is represented by an agent Add a commutative list of cluster members within a transactional reference to accelerate the centroid update step Data Agent 0 Data Agent 1 Data Agent 2 Data Agent 3 Data Agent n Cluster Agent 0 Member Ref 0 Cluster Agent 1 Member Ref 1 Cluster Agent k Member Ref k read write

21 simultaneous read Cluster Agent 0 Data Agent 0 Cluster Agent 1 Data Agent 1 Cluster Agent k Data Agent n simultaneous write Member Ref 0 Data Agent 0 Member Ref 1 Data Agent 1 Member Ref 2 Data Agent n

22 read: (nearest-cluster) write: (commute) (assoc) ( defn assignment [ ] (map #(send % update dataagent ) DataAgents ) ( defn update dataagent [ d a t a p o i n t s ] (map update d a t a p o i n t d a t a p o i n t s ) ) ( defn update d a t a p o i n t [ d a t a p o i n t ] ( l e t [ newass ( n e a r e s t c l u s t e r d a t a p o i n t ) ] ( dosync (commute ( nth MemberRefs newass ) conj ( : data d a t a p o i n t ) ) ) ( assoc d a t a p o i n t : assignment newass ) ) )

23 Benchmark results Large data sets (artificial): Each data point is sampled from N(0,1) Summary for 10 runs of K-means cases, 100 dimensions 20 Cluster cases, 200 dimensions 20 Cluster runtime (seconds) runtime (minutes) ParaKMeans K-means R McKmeans K-means R McKmeans

24 Number of computer cores used Number of data agents used x cluster x cluster runtime (seconds) runtime (seconds) number of computer cores number of data agents

25 Large data sets with cluster structure Data sampled from a multi-variate normal distribution samples, 200/500 dimensions, 10/20 cluster runtime (seconds) K-means R McKmeans 200 / / / / / / / / 20 Number of samples / Number of clusters

26 Accuracy compared to the known grouping of data Measured with the MCA index Red bars indicate the random-prototype baseline x cluster x cluster x cluster x cluster MCA index McKmeans K-means R McKmeans K-means R McKmeans K-means R McKmeans K-means R

27 Real world data set Microarray data (Radiation-induced changes in human gene expression) samples (genes) and 465 features (profiles) K-means R McKmeans runtime (seconds) Cluster 5 Cluster 10 Cluster 20 Cluster 2 Cluster 5 Cluster 10 Cluster 20 Cluster Number of clusters Smirnov D, Morley M, Shin E, Spielman R, Cheung V: Genetic analysis of radiation-induced changes in human gene expression. Nature 2009, 459:

28 Application to Cluster Number Estimation Repeated clustering with different subsets of data Repeated for different number of clusters k Most stable clustering is produced for the real cluster number Jackknife resampling Evaluation with MCA index Data set: samples, 100 features, 3 cluster 10 runs per cluster number minutes on dual-quad core 3.2 GHz MCA index number of clusters

29 Java GUI ( import ( javax. swing JFrame JLabel JTextField JButton ) ( j a v a. awt. event A c t i o n L i s t e n e r ) ( j a v a. awt GridLayout ) ) ( l e t [ frame ( new JFrame Hello, World! ) h e l l o button ( new JButton Say h e l l o ) h e l l o l a b e l ( new JLabel ) ] (. h e l l o button ( a d d A c t i o n L i s t e n e r ( proxy [ A c t i o n L i s t e n e r ] [ ] ( a c t i o n P e r f o r m e d [ evt ] (. h e l l o l a b e l ( s e t T ext Hello, World! ) ) ) ) ) ) ( doto frame (. setlayout ( new GridLayout ) ) (. add h e l l o button ) (. add h e l l o l a b e l ) (. s e t S i z e ) (. s e t V i s i b l e t r u e ) ) )

30

31 Summary Writing parallel programs usually requires a careful software design and a deep knowledge about thread-safe programming Concurrency control via transactional memory circumvents problems of lock-based concurrency strategies Immutable data structures play a key role to software transactional memory Clojure combines Lisp, Java and a powerful STM system This enables fast parallelization of algorithms, even for rapid prototyping Our simulations show a good performance of the parallelized code

32 Thank you for your attention.

33 Statistical computing library Clojure-based statistical computing R-like semantics COLT library for numerical computation JFreeChart library for graphics