Cell Tracking via Proposal Generation & Selection

Transcription

1 Cell Tracking via Proposal Generation & Selection Saad Ullah Akram, Juho Kannala, Lauri Eklund and Janne Heikkilä 2

2 Overview Introduction Importance Challenges: detection & tracking Current state of cell tracking field Our method Cell proposal generation (CPN) Proposal graph Cell tracking via proposal selection Summary

3 Overview Introduction Importance Challenges: detection & tracking Current state of cell tracking field 4

4 Cell Tracking Process of: Locating a moving cell over time & Detecting cellular events (division, death, entry, exit) : highlights links between parent & daughter cells Time Lineage of a cell saad.akram@oulu.fi 5

5 Applications Understanding dynamic cellular behavior Spread of diseases (e.g. cancer) Effectiveness/safety of a drug Organ/embryo development Gene profiling saad.akram@oulu.fi 6

6 Automation Huge datasets 1,000s of D images/day 10,000s cells in each image > 10 TBs of image data/day Subtle patterns Accurate/objective/repeatable Video from: F. Amat et al., Fast, accurate reconstruction of cell lineages from large-scale fluorescence microscopy data., Nat. Methods, saad.akram@oulu.fi 7

7 Challenges: Tracking Difficult to model motion Similar cell appearances and shapes Similar shapes & appearances 8

8 Challenges: Tracking Low frame rate Large movements Large shape and appearance changes Large movement Difficult to detect cell division Parent cells in last frame before division 9

9 DIC Microscopy Phase Contrast Bright Field Challenges: Detection Detection failures account for most tracking errors Large variation in cell shapes & appearances Microscopy modality Labeled structure Resolution Cell density Fluorescence Dyes & Proteins Image from: E. Meijering, Cell Segmentation: 50 Years Down the Road, IEEE Signal Process. Mag.,

10 Current cell tracking methods Tracking by detection Detect cells in each frame Associate detections in nearby frames + Low frame rate/resolution Simple and modular Pros Tracking by model evolution Detect cells in 1 st frame and represent it using some model Segment current frame using the model from last frame Pros + Can handle noisy images ImageJ (TrackMate, MTrack2), CellProfiler (TrackObjects), etc saad.akram@oulu.fi 11

11 Limitations of current cell tracking methods Cell detection methods assume Cell appearances High intensity near the center Characteristic intensity profile at cell boundaries Cell shapes Round or elliptical shapes Cell size within some narrow range Assumptions are often not satisfied for new sequences Lack robustness Require tuning multiple parameters 12

12 Overview Our method Cell proposal generation (CPN) Proposal graph Cell tracking via proposal selection 1

13 Tracking by proposal selection 1. Generate cell proposals Use CNNs to learn the cell shapes and appearances 2. Link cell proposals using edges (represent cellular events). Select subset of cell proposals & their associations (provides cell tracks) 14

14 Cell proposals/candidates Cell detection can benefit from Temporal information Machine learning to decide what a cell looks like A cell proposal is a candidate segmentation of a cell Enables representation of multiple segmentation hypothesis in ambiguous regions Image Cell Detection Cell Proposals: 2 Cell Proposals: Cell Proposals: 4 saad.akram@oulu.fi 15

15 ROIPooling bbox score Candidate Bounding Boxes Cell proposal network (CPN) Bounding box network Segmentation network k 2k Nx k NMS NMS Proposals saad.akram@oulu.fi 16 1 NMS Proposal Masks Fig: CPN: Convolutional, max-pooling, fully connected, and deconvolutional layers are shown. Proposed bounding boxes and segmentation masks after non-maxima suppression (NMS) are shown for a selected area from Fluo-N2DL-HeLa dataset.

16 Time Tracking graph (g): Ground truth Cell tracks can be represented using a graph S Nodes represent cells D 1 Special nodes (S & T): Initiate/terminate cell lineage trees D 2 D Edges represent cell events Cell entering field of view Cell exiting field of view Cell division (mitosis) Cell moving a) D 4 T D 5 b) saad.akram@oulu.fi 17

17 Time Graphical model (G) A graphical model (G) is used to reason about which cell tracks should be selected Nodes represent cell proposal probabilities Black edges represent proposal constraints Special nodes (S & T): Initiate/terminate cell lineage tress Only few edges of each type are shown to avoid clutter Edge weights represent cellular event probabilities Cell entering field of view Cell exiting field of view Mitosis Cell moving a) S T b) saad.akram@oulu.fi 18

18 Time Proposal selection Each sub-graph of our model (G) represents a tracking hypothesis Optimal tracks can be obtained by finding the sub-graph with the highest probability S a) 0.07 T 0.0 b) c) saad.akram@oulu.fi 19

19 Results Same model structure is used for all datasets 4 datasets from ISBI Cell Tracking Challenge [1] TRA: Tracking score (0-1 (perfect)) penalizes errors in the tracks SEG: Segmentation score Instance level intersection over union overlap. Video [1] Method TRA SEG CPN Fluo-N2DL-HeLa-02 EPFL [] 0.97 KTH [4] CPN Fluo-N2DH-GOWT1-02 HEID [2] 0.95 EPFL [] 0.95 PhC-C2DH-U7-02 CPN U-Net [5] PhC-C2DL-PSC-01 CPN [1] M. Maška et al., A Benchmark for Comparison of Cell Tracking Algorithms, Bioinformatics, [2] M. Schiegg, et al., Graphical Model for Joint Segmentation and Tracking of Multiple Dividing Cells, Bioinformatics, [] E. Turetken, et al., Network Flow Integer Programming to Track Elliptical Cells in Time-Lapse Sequences, in T-MI, [4] K. E. G. Magnusson, et al., A Batch Algorithm using Iterative Application of the Viterbi Algorithm to Track Cells and Construct Cell Lineages, in ISBI, [5] O. Ronneberger, et al., U-Net: Convolutional Networks for Biomedical Image Segmentation, in MICCAI, saad.akram@oulu.fi 20

20 Time Fluo-N2DL-HeLa-02 21

21 Time PhC-C2DH-U

22 Time PhC-C2DL-PSC-01 2

23 Overview Summary 24

24 Summary + (Potentially) can be applied to other sequences Has better performance than existing methods Requires annotated data Our tracking method More Information: Paper: Code: saad.akram@oulu.fi 25

25 Thanks. Questions? 26