Machine Learning & Science Data Processing

Transcription

1 Machine Learning & Science Data Processing Rob Lyon SKA Group University of Manchester

2 Machine Learning (1) Collective term for branch of A.I. Uses statistical tools to make decisions over data intelligently. Appearance of intelligence is an illusion backed up by functions. So how does it work?

3 Machine Learning (1) Collective term for branch of A.I. Uses statistical tools to make decisions over data intelligently. Appearance of intelligence is an illusion backed up by functions. So how does it work? f(x) f(x) 0 f(x) 00 Observe Collect Data Build Model

4 Machine Learning (2) Redesign Intensity Phase Training Validation MAP Design Evaluate Build

5 Machine Learning & SDP

6 Machine Learning & SDP SDP converts / filters CSP data in to products useful for science.

7 Machine Learning & SDP SDP converts / filters CSP data in to products useful for science.

8 Machine Learning & SDP SDP converts / filters CSP data in to products useful for science.

9 Machine Learning & SDP SDP converts / filters CSP data in to products useful for science. Includes pulsar timing, single pulse search (transients signals, FRBs) and periodicity search (pulsars).

10 Machine Learning & SDP SDP converts / filters CSP data in to products useful for science. Includes pulsar timing, single pulse search (transients signals, FRBs) and periodicity search (pulsars). For single pulse and periodicity search, CSP data products describe potential observations of astrophysical phenomena - new discoveries?

11 Existing Approaches Applied to candidate selection for single pulse and periodicity search. Supervised machine learning algorithms. Learn from fixed-size training sets of examples. Variety of algorithms used, with varying computational requirements Sub-Band Number Sub-Int Number 4.0e-032.0e-03 0 Amplitude Sub-Bands Bin Number Sub-Integrations Bin Number Profile A B C Phase DM SNR Period-DM Plane -4.0e e e e e+10 Topo Period Offset from ms DM Curve DM Bary Period Bary Freq Topo Period Topo Freq Acc Jerk DM width SNR Initial N/A D E Optimised ms Hz ms Hz m/s/s m/s/s/s cm/pc periods periods

12 Existing Approaches Applied to candidate selection for single pulse and periodicity search. Supervised machine learning algorithms. Learn from fixed-size training sets of examples. Variety of algorithms used, with varying computational requirements.

13 Which method?

14 Issues With ML at Scale

15 Issues with ML at SKA Scale ML typically very accurate if training data is good. Problems: 1. Not optimised to minimise resource use. 2. Non-adaptive, and retraining with more examples can be expensive (depending on the algorithm). Other issues: training data hard to obtain, classifier decisions often hard to audit.

16 Practical Issues with ML at SKA Scale (1) Adapting to distributional change advantageous. Rapidly adapting to new training examples important for discovery. 1 Period SNR Mean value (normalised) Feature 1 Feature 2 Feature 12 Feature 13 Feature 14 0 DM Sample number

17 Structural Issues with ML at SKA Scale (2) Performance issues 3000 Small sample problem Small disjuncts due to imbalance. How to acquire training examples? How to incorporate expert feedback? How to audit Chi squared value from fitting sine-squared curve to pulse profile Class overlap classifications? Pulsar Non-pulsar Chi squared value from fitting sine curve to pulse profile. 1500

18 Exploring solutions

19 Possible SDP Approach Data stream learning methods. Very low resource requirements. Able to adapt to concept drift. Able to learn from new training examples observed over time. Classifier Predict Store a) Incremental Model Train Test Report Performance b) Batch Model Report Performance

20 Incremental Stream Prototype Process OCLDs Pre-process data ML prediction Source matching Archive OCLD Spouts Key n 0 n 1 2 n 2 2 n 3 2 n 4 2 n 5 2 n 6 2 n 7 n 8 Telescope Manager / Science Teams Bolts Sift per beam Feature extraction Second sift Alerts Spouts Auditing data Primary data Auditing nodes (deactivated during normal execution)

21 Stream Classifier: GH-VFDT

22 Algorithm Performance See Fifty Years of Pulsar Candidate Selection: From simple filters to a new principled real-time classification approach, Lyon et al, accepted for publication in MNRAS, Other results in: Hellinger Distance Trees for Imbalanced Streams, Lyon et al., ICPR, 2014.

23 Prototype Performance Local tests on a single machine (Quad Core i7) 1,000 candidates per second Approx. 2 seconds for a candidate to move through the system. Relatively easy to configure & program. Possible problems: you may want to send more than a tuple. you may want data to go backwards.

24 Open Questions How to acquire unlimited supply of accurately labelled data? To what extent does pulsar data drift? Will a multi-class approach improve classification performance? What will the final compute environment look like? How do we keep track of training data and validate our approaches? Are our features good enough?

25 Summary Structural issues with ML at scale Practical issues with ML at scale Success depends on understanding the data Prototype pipelines under development Thanks for listening!

26 Candidate Numbers

27 Tree Classification Example x 1 x 2 x 3 x 4 x 5 x n h(cm) w(kg) l(cm) h > 50? Predictions x 1 x 2 x No Yes x 1 x 2 x 3 A B A No w > 5? l > 49? Yes No Yes A B A B

28 Apache Storm Cluster Master node Nimbus Zookeeper Zookeeper cluster Zookeeper Zookeeper JVM JVM Worker Process Worker Process Executor Executor Spout Spout Bolt Bolt JVM Worker Process Executor Spout Bolt Executor Executor Executor Bolt Bolt Bolt Worker node Worker node Worker node Supervisor Supervisor Supervisor JVM JVM Worker Process Worker Process Executor Executor Spout Spout Bolt Bolt JVM Worker Process Executor Spout Bolt Executor Executor Executor Bolt Bolt Bolt 1 n