Jersey Number Recognition using Convolutional Neural Networks

Transcription

1 Image Processing Jersey Number Recognition using Convolutional Neural Networks, Einsteinufer 37, Berlin

2 Outline Motivation Previous work Jersey Number Dataset Convolutional Neural Network Architecture Results Summary and Outlook 2

3 Motivation Goal: Player identification in broadcast soccer videos???? 3

4 Motivation Goal: Player identification in broadcast soccer videos Usage Scenarios: Interactive second screen applications Personalized video summary (grandma can watch her grandson playing) Match analysis, training Video search engines Media archives Kadlec Player scouting Gentner Sakai Harnik Kvist Tasci Holzhauser Bender Sam 4

5 Application Example: UHD Zoom UHD receiver WLAN Müller Kroos Özil Kaká Reus 4k display 5

6 Player Annotation 6

7 Tracking of Selected Player 7

8 Approaches for Player Identification Features for player identification: Face recognition (not feasible in overview shots too low resolution) Hair/skin/shoe color Tactical positions of players Jersey numbers 8

9 Analysis Components Player Identification <50% recognition rate tactical player positions Jersey number recognition Player Tracking 9

10 Dataset Manually labelled 8,281 * images with 36 classes (jersey numbers 1-44, some numbers not in the corpus) Quite small in comparison to other datasets used for evaluating similar tasks Deep Learning feasible for such a small dataset? * ) 8281 out of 10,000 player images had visible jersey numbers. These 10,000 players were chosen out of 70,000 player images with the highest number in visibility probability. 10

11 Distribution of Data Dataset is very challenging, especially for deep learning approaches: Imbalanced class distribution (many players with number 10) might decrease accuracy small dataset with high number of classes compared to other databases # Samples per class Dataset Classes Resolution Training Test Handwritten digits (MNIST*) x 28 x 1 60,000 10,000 Street View House Number (SVHN) x 32 x 3 73,257 26,032 Traffic Sign (TS) x 32 x 3 39,209 12,630 Soccer Jersey Number (SJN) x 128 x 3 5,760 2,521 * ) MNIST: Mixed National Institute of Standards and Technology 11

12 Approach: Convolutional Neural Network Neural Network Architecture: 3 convolutional layers, each followed by a max-pooling layer and rectified linear unit activations (stages 1-3) 3 fully connected hidden layers with optional dropout layers (stages 4-6) 1 final softmax activation layer (stage 6) Stage 1 Stage 2 Stage 3 Stage 4 Stage 5 Stage 6 16 feature maps 40 x feature maps 20 x feature maps 6 x 6 34 nodes 34 nodes 45 ouput nodes convolution fully connected fully connected + softmax max pooling + convolution max pooling + convolution fully connected 12

13 Convolutional Neural Network Learned convolutional filters (1st layer) Filtered output 13

14 Neural Network Performance Reference: HOG features + SVM classifier (RBF kernel) accuracy: 0.40 Baseline Convolutional Neural Network accuracy: 0.61 Improving Accuracy: Dropout to prevent overfitting Using grey-scale representations instead of RGB to reduce influence of color information (should also avoid color bias for certain numbers) Augmenting training data by applying modifications: Cropping Translation Scaling Color inversion (HOG: Histogram of Oriented Gradients) (SVM: Support Vector Machine) (RBF: Radial Basis Function) 14

15 Results Number recognition results:

16 Verification of Results Class confusion matrix Confusion matrices show correctly classified entries at the main diagonal and wrongly classified entries at other positions Most confusions occur when true number and recognized number share at least one digit, (e.g. 19 recognized as 29 ) 16

17 Comparison of Different Methods Method HOG 0.40 ConvNet 0.61 ConvNet dropout 0.71 ConvNet grey dropout 0.72 ConvNet inverted grey dropout 0.76 ConvNet augmented grey digit-wise 0.62 ConvNet augmented 0.68 ConvNet augmented dropout 0.71 ConvNet augmented grey 0.73 ConvNet augmented grey inverted 0.82 ConvNet augmented grey inverted dropout 0.83 Accuracy Best result with 83% accuracy for convolutional neural network with dropout layers, using grey-scale data, and augmented (including inverted) training data 17

18 Summary Relatively small dataset for jersey number recognition Deep learning based method for jersey number recognition Augmentation strategies to account for small dataset, thus giving higher accuracy in deep learning approaches Results are promising: Accuracy of up to 0.83, but there still room for improvement 18

19 Outlook Pre-Training with larger datasets, fine-tuning with jersey numbers Combination with tactical role as identification criterion: + 19

20 Thank You! The Einsteinufer Berlin, Germany Phone Contact: 20