An Open Source Framework for Tracking and State Estimation ('Stone Soup')

Transcription

1 CAN UNCLASSIFIED An Open Source Framework for Tracking and State Estimation ('Stone Soup') Paul A. Thomas, Jordi Barr Defence Science and Technology Laboratory (UK) Bhashyam Balaji DRDC Ottawa Research Centre Kruger White Defence Science and Technology Group (Australia) Society of Photo-Optical Instrumentation Engineers (SPIE), Vol , Date of Publication from Ext Publisher: May 2017 Terms of Release: This document is approved for Public release. Defence Research and Development Canada External Literature (N) DRDC-RDDC-2017-N031P00 September 2017 CAN UNCLASSIFIED

2 CAN UNCLASSIFIED IMPORTANT INFORMATIVE STATEMENTS Disclaimer: This document is not published by the Editorial Office of Defence Research and Development Canada, an agency of the Department of National Defence of Canada, but is to be catalogued in the Canadian Defence Information System (CANDIS), the national repository for Defence S&T documents. Her Majesty the Queen in Right of Canada (Department of National Defence) makes no representations or warranties, express or implied, of any kind whatsoever, and assumes no liability for the accuracy, reliability, completeness, currency or usefulness of any information, product, process or material included in this document. Nothing in this document should be interpreted as an endorsement for the specific use of any tool, technique or process examined in it. Any reliance on, or use of, any information, product, process or material included in this document is at the sole risk of the person so using it or relying on it. Canada does not assume any liability in respect of any damages or losses arising out of or in connection with the use of, or reliance on, any information, product, process or material included in this document. This document was reviewed for Controlled Goods by Defence Research and Development Canada (DRDC) using the Schedule to the Defence Production Act. Template in use: (2012) CR EL1 Advanced Template_EN _21-V02_WW.dotm Her Majesty the Queen in Right of Canada (Department of National Defence), 2017 Sa Majesté la Reine en droit du Canada (Ministère de la Défense nationale), 2017 CAN UNCLASSIFIED

3 An Open Source framework for Tracking and State Estimation ( Stone Soup ) Paul A. Thomas (pathomas@dstl.gov.uk) a, Jordi Barr a, Bhashyam Balaji b, Kruger White c a Defence Science and Technology Laboratory (United Kingdom), b Defence Research and Development Canada (Canada), c Defence Science and Technology Group (Australia). ABSTRACT The ability to detect and unambiguously follow all moving entities in a state-space is important in multiple domains both in defence (e.g. air surveillance, maritime situational awareness, ground moving target indication) and the civil sphere (e.g. astronomy, biology, epidemiology, dispersion modelling). However, tracking and state estimation researchers and practitioners have difficulties recreating state-of-the-art algorithms in order to benchmark their own work. Furthermore, system developers need to assess which algorithms meet operational requirements objectively and exhaustively rather than intuitively or driven by personal favourites. We have therefore commenced the development of a collaborative initiative to create an open source framework for production, demonstration and evaluation of Tracking and State Estimation algorithms. The initiative will develop a (MIT-licensed) software platform for researchers and practitioners to test, verify and benchmark a variety of multi-sensor and multi-object state estimation algorithms. The initiative is supported by four defence laboratories, who will contribute to the development effort for the framework. The tracking and state estimation community will derive significant benefits from this work, including: access to repositories of verified and validated tracking and state estimation algorithms, a framework for the evaluation of multiple algorithms, standardisation of interfaces and access to challenging data sets. Keywords: Tracking, state estimation, open source, software development Content includes material subject to Crown copyright (2017), Dstl. This material is licensed under the terms of the Open Government Licence except where otherwise stated. To view this licence, visit or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or psi@nationalarchives.gsi.gov.uk. 1. INTRODUCTION Tracking & state estimation is a critical capability for military Situational Awareness (SA). The ability to detect and follow unambiguously all moving entities in a state-space is important for SA in multiple domains ranging from the traditional (e.g. air surveillance, maritime SA, ground moving target indication) to the less conventional (e.g. tracking internet transactions, monitoring changes in sentiment, CBRN dispersion prediction or following disease epidemics). However, researchers and practitioners find difficulties recreating state-of-the-art algorithms in order to benchmark their own work 1. Comparison of new algorithms with alternative solutions is time-consuming, involving recoding of the alternative algorithms from the academic literature. Industrial exploiters of algorithms also need to assess which algorithms meet operational requirements objectively and exhaustively rather than intuitively and driven by personal favourites. The best solution to these issues is to develop an open source tracking and state estimation framework and to populate it with open source components. This is in line with wider open source and open data government initiatives 2,3. The project described in this paper aims to provide a flexible and unified software platform for researchers and engineers to Signal Processing, Sensor/Information Fusion, and Target Recognition XXVI, edited by Ivan Kadar, Proc. of SPIE Vol , SPIE CCC code: X/17/$18 doi: / Proc. of SPIE Vol

4

5

6 3. ACCESSIBILITY The key to wide adoption of this framework is to make it as accessible as possible. 3.1 Licence Most of the code developed in the Consortium phase of the Stone Soup project will be subject to copyright. All such code will be licensed under the MIT licence 4, which is one of the freest forms of open source licence. Work done by US Government employees as part of their official duties is not subject to copyright and shall contain a disclaimer stating that it is not subject to copyright protection. All source code files shall contain a reproduction of the licence as a prominent boilerplate in the file. During the open phase of the project external contributions may either be subject to copyright or not subject to copyright protection. Contributions subject to copyright protection will be permissible if they are released under any of the following licences: The MIT License 4 ; The BSD 3-Clause license 5 ; The BSD 2-clause license 6 ; The International Astronomical Union's Standards of Fundamental Astronomy License 7. Contributions not subject to copyright shall contain a disclaimer stating that they are not subject to copyright protection. 3.2 Protective marking The open source version of the framework will not be protectively marked, though users are at liberty to download a copy and run it on protected data/scenarios as they wish. 3.3 Releases Ideally, in an open source project, the development team would code in the open 11. However, due to the legacy requirement for release permission from each government lab before each released software version, this would require a culture change. Such a culture change is long overdue and this project should prove the driver for this, but until this happens, Stone Soup will work under the model of shared coding in a private Git repository, coupled with frequent authorised releases. 4. REQUIREMENTS Open source software, unlike the traditional software development model, tends to develop amorphously according to the will and enthusiasm of members of the user/developer community. Therefore, upfront definition of requirements for the open phase of Stone Soup is less valid. However, in order to rapidly build the initial framework in the Consortium phase it is useful to have a set of requirements. These allow the funding stakeholders in the Consortium phase to shape the initial project to suit their objectives, which gives them an initial return on their investment. Of course, the Stone Soup consortium phase is more concerned with building a framework that is compatible with future algorithms, data, metrics, simulators and sensor models, than actually populating these elements. The requirements, therefore, relate to compatibility with specified classes of algorithm, types of data, types of model, etc., that might be added in the open phase; rather than requiring specific functions to be added immediately. These types of requirement will drive the data model inside the Stone Soup framework. For example, the funding stakeholders for the consortium phase might decide that, although (of course) they would prefer compatibility with all types of algorithm, compatibility with certain algorithm classes have a higher priority for the initial work on the framework. This would drive the data model to ensure compatibility with these classes first. Proc. of SPIE Vol

7 1 4.1 Use Cases In order to frame the requirements, the project team constructed three Use Cases. As Stone Soup is open source software, where the community is characterised as user/developers rather than simply users, the use cases contain activities associated with code development. The use cases are: Analyst A common analysis use case involves a task to evaluate the efficacy of a recently-developed novel state-estimation algorithm by a Tracking Evaluator. The evaluator has a technical knowledge of the field, but not a deep understanding of the algorithm, or its theoretical underpinnings. The evaluator is able to receive a Stone Soup compatible algorithm from the developer and download standard verified benchmarking algorithms, test data, and some relevant metrics. In this way they are able to form a judgement on the algorithm, impartially, and with pertinence to their problem. Algorithm developer Provides algo thm Downloads relevant metrics I Tracking Evaluator Uses standard test dataset Evaluates Stone Soup Component developer An algorithm developer is at work on a bleeding-edge multi-sensor, multi-target tracking (MTT) algorithm. The developer codes up the algorithm in Python and is busy writing up their results for publication in a top journal. They are able to access standard, well-used, MTT algorithms in the Stone Soup repository. At the same time they can access appropriate data and some useful metrics. With this repository, the developer is able to reproduce that others have run and to demonstrate impressive margin. The results are published no doubt to much acclaim. The developer may wish to upload their algorithm to the Stone Soup repository in order to benefit the wider tracking community. that the new algorithm does indeed outperform the current state of the art by an the same experiment Proc. of SPIE Vol

8 loads novel algorithm Downloads standard algorithm Alpo Elm De wnloads relavant me rics Writes up re:yl Produces novel otonmm 1(e..goh,Pap Tests am renne Stone Soup Framework developer New_Requiremenl User/Developer include» Upload to Stone Soup Repository Update Software documentation «include» «extend >4 Mod erator Authorise New Release The Framework developer, perhaps prompted by an external requirement, or their own requirement, makes a modification to the code base of the Stone Soup framework. This modification must be associated with an update to the software documentation. Both these elements are uploaded to the Stone Soup (Git) repository. Perhaps prompted by the upload or perhaps during a scheduled release, the moderator authorises the new release. The release process involves retrieval of code from version control and running a script that pulls dependencies, builds and tests the latest code. If the build is for release, the script will also collate the relevant information to populate a Release Note, including the commit ID, the test results, the changes to the code since last release and any outstanding and fixed issues. The master branch will be used and tagged in Git to ensure every release can be replicated. Proc. of SPIE Vol

9 4.2 Algorithms Priority for the initial phases of Stone Soup will be to ensure compatibility with the following generic algorithm types: Filtering algorithms: Discrete-Time State and Measurement Models o Standard Kalman Filter for the Linear State and Measurement Model o Extended Kalman Filter o Derivative-free Kalman Filters Particle Filter Class of algorithms o Random Particle Filters o Deterministic Particle Filters Multiple Model filtering algorithms for Kalman filter class of algorithms 4.3 Data Model These different algorithms, at the most basic level, are attempting to represent a dynamic system. A general dynamic system consists of state and measurement models, either of which may be linear or non-linear (noting that the system might be time-varying; i.e. the state model may change with time, the measurement model may change with time or the state and measurement models may explicitly depend on time). The fundamental requirement of Stone Soup is to enable the comparison of different algorithmic approaches against the same data or simulated scenario. Thus algorithms must be swappable at the state and measurement model level. Therefore, key to the success of the framework is the creation of an underlying data model which enables algorithm swappability across all the supported algorithm classes and can cope with all the supported data types. This is more complex than it seems because the algorithms represent uncertainty in different ways, e.g. algorithms based around the Kalman filter model a target state distribution as a mixture of one or more Gaussian distributions, versus particle filters which represent the distribution by a set of weighted samples. Furthermore, the data model must support manipulation of a measurement model via data from a variety of data sources, in effect data swappability, across range and bearing, bearings only, range only, and/or categorical (identity) data. 4.4 Data Priority for the initial phases of Stone Soup will be to enable the use of the following data types. This involves creation of an appropriate data model in the framework for handling the data and making available representative data sets in the repository. The initial data types will be: Airborne radar detection data. This could be in 2D or 3D, from a rotating radar or from a planar array. Coincident AIS data EO/IR data (which typically is reasonably accurate in bearing but has poor accuracy or no information about range) The intention is to build wider compatibility with further data types and provide more example data sets, as the project progresses. 4.5 Metrics Assessment of alternative Tracking and State Estimation algorithms requires a means of objective evaluation using a set of metrics which correspond to characteristics of interest to operational systems 16. In certain circumstances knowledge of the actual number of entities and their states may be available such as when simulations include the state of entities, sensor data and implementation of tracking and state estimation algorithms. In other circumstances knowledge of the actual number of entities and their states may not be available. Priority for the initial phases of Stone Soup will be to enable assessments in the presence of entity truth state information. In this case, an absolute evaluation of each alternative algorithm can be conducted by comparing the tracking and state estimation output against the truth data. Typically, a data association approach is first used to associate track data with entity truth data 16 Subsequently, performance metrics can be applied to quantify tracking characteristics such as the following: Proc. of SPIE Vol

10 Initiation of a track following the first appearance of an entity Accuracy of a track estimate compared to the state of an entity Continuity of a track during the time evolution of the state of an entity Existence of false tracks which are tracks that do not correspond to entities of interest Credibility of a track estimate and its estimation uncertainty compared to the state of an entity 17 As with other aspects of Stone Soup, the intention is to develop further performance metrics as the project progresses. 5. DEVELOPER GUIDE The Stone Soup project aims to provide an environment which is as open and inclusive as possible to contributions from the broad tracking community. However, some broad development guidance is required in order to ensure a minimum quality standard. The Stone Soup Consortium phase will implement a standard body of developer guidance that is defined in the Stone Soup Requirements Document 10. This provides guidance on the following: Documentation / comments Code Quality Interfaces to and dependencies on external code Extensibility and reuse of algorithms Unit Tests 6. JOINING IN The Stone Soup project would welcome additional commitments of developer resources during the Consortium Phase. The minimum meaningful commitment is 0.5 x developer year per year. This is open to all types of contributing agency; e.g. Government, Industry or Academia, in both the Defence and Civil sectors. Agencies willing to make this level of commitment would gain membership of the Requirements Panel, shaping the software to their needs. Benefits to being part of the Consortium Phase are: Influence on the development of what (the authors hope) will become an academic standard open architecture o This supports the most governments visions of making greater use of open systems. Significant gearing on algorithm and testbed development o Several pledges of support of manpower have so far been received from government agencies. Ability to influence the international tracking and state estimation community o Leading the community towards next-generation tracking and state estimation approaches, scenarios and solutions. Any interested party should contact the corresponding author of this paper. 7. WHY THE NAME? The name Stone Soup refers to an old folk story 13, 14, 15, and many more, in which a traveller persuades the local people of a village into collaborating to share their food. The story: A traveller arrives in a village, carrying nothing more than an empty cooking pot. Upon arrival, the discovers the village is undergoing a famine and everyone is hungry. So the traveller goes to a stream and fills the pot with water, drops a large stone in it, and places the pot over a fire. One of the villagers becomes curious and asks what is happening. The traveller answers that "stone soup" is being made and that, it is very nutritious, although it still needs a little bit of garnish to improve the flavour. The villager does not mind parting with an old cabbage so that is added to the soup. Another villager walks by, inquiring about the pot, and the traveller again mentions their stone soup which has not reached its full potential yet. The villager hands them some scraps to help them out. More and more villagers walk by, Proc. of SPIE Vol

11 each adding meat, potatoes, etc, etc. Finally, the stone (being inedible) is removed from the pot, and a delicious and nourishing pot of soup is enjoyed by all. Moral: By working together, with everyone contributing what they can, a greater good is achieved Of course, in the case of the original story the soup is consumed. However in the case of a software project, the output from the collaboration will remain to benefit everyone into the future. 8. SUMMARY Stone Soup is a collaborative initiative to create an open source framework for comparison of tracking and state estimation algorithms, data, metrics, simulators and sensor models. The initiative will develop a software platform for researchers and practitioners to test, verify and benchmark multi-sensor and multi-object state estimation algorithms. The initiative is split into two phases, Consortium and Open, with the Consortium phase being supported by a number of government defence laboratories, who will contribute to the development effort for the framework. The Open phase will build upon and existing repository which will be drawn from The Tracker Component Library 5, published on Git with the addition of algorithms from the academic community. The authors hope this will become an academic standard open architecture. Code developed in the Consortium phase of the Stone Soup project will be not protectively marked and released under the MIT license, where possible. Requirements are being developed currently and are driven by three use cases; Analyst, Component developer and Framework developer. Developer guidance is also being written to ensure a minimum software quality standard. We predict that the tracking and state estimation community will derive significant benefits from this work, including: access to a repository of tracking and state estimation algorithms, framework for evaluation of multiple algorithms and access to challenging data sets. The authors encourage contributions of developer time to assist in building the framework. Any interested party should contact the corresponding author. REFERENCES [1] P. Thomas, Open letter on the stone soup tracking framework, in Proceedings of the 10th Data Fusion and Target Tracking Conference, University of Liverpool, United Kingdom, 30 Apr [Online]. Available: [2] Defence White Paper: National Security Through Technology. pp [Online] Available [3] B. Obama. Presidential memorandum building a 21st century digital government. The White House, Office of the Press Secretary. (2012) [Online]. Available: [4] Raymond, Eric S. The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary. O'Reilly Media. ISBN (1999). [5] David Frederic Crouse. The Tracker Component Library: Free Routines for Rapid Prototyping. Submitted for publication to IEEE Aerospace and Electronic Systems Magazine. [6] [7] [8] [9] [10] Open Source Tracking and State Estimation "Stone Soup" Requirements Document. Thomas, PA, Crouse, DF, Balaji, B, White, K, Tagg, J, Barr, J, Brown, G, Green, R, Ablett, S, McKinlay, R. DSTL/PUB (2017). [11] Coding in the open. Blog - Government Digital Service. [online] Available Proc. of SPIE Vol

12 [12] Open Source Development Guidelines. Civic Commons. [Online] Available [13] Stone Soup. DLTK's Educational Activities: Fables. [Online] Available [14] Stone Soup: An Old Tale. Marcia Brown. Aladdin Picture Books. ISBN: [15] Stone Soup. Jon J Muth. Scholastic Press; 1st edition ISBN-10: X [16] A. Blackman, S. and Popoli, R. Design and Analysis of Modern Tracking Systems. Artech House, [17] B. Li, X.R., Zhao, Z., Li, X-B. Evaluation of Estimation Algorithms Credibility Tests. IEEE Trans on Systems, Man, and Cybernetics Part A: Systems and Humans, Vol. 42, No. 1, January Proc. of SPIE Vol

13 CAN UNCLASSIFIED DOCUMENT CONTROL DATA (Security markings for the title, abstract and indexing annotation must be entered when the document is Classified or Designated) 1. ORIGINATOR (The name and address of the organization preparing the document. Organizations for whom the document was prepared, e.g., Centre sponsoring a contractor's report, or tasking agency, are entered in Section 8.) 2a. SECURITY MARKING (Overall security marking of the document including special supplemental markings if applicable.) DRDC Ottawa Research Centre Defence Research and Development Canada 3701 Carling Avenue Ottawa, Ontario K1A 0Z4 Canada CAN UNCLASSIFIED 2b. CONTROLLED GOODS NON-CONTROLLED GOODS DMC A 3. TITLE (The complete document title as indicated on the title page. Its classification should be indicated by the appropriate abbreviation (S, C or U) in parentheses after the title.) An Open Source Framework for Tracking and State Estimation ('Stone Soup') 4. AUTHORS (last name, followed by initials ranks, titles, etc., not to be used) Thomas, P.A.; Barr, J.; Balaji, B.; White, K. 5. DATE OF PUBLICATION (Month and year of publication of document.) September a. NO. OF PAGES (Total containing information, including Annexes, Appendices, etc.) 11 6b. NO. OF REFS (Total cited in document.) DESCRIPTIVE NOTES (The category of the document, e.g., technical report, technical note or memorandum. If appropriate, enter the type of report, e.g., interim, progress, summary, annual or final. Give the inclusive dates when a specific reporting period is covered.) External Literature (P) 8. SPONSORING ACTIVITY (The name of the department project office or laboratory sponsoring the research and development include address.) DRDC Ottawa Research Centre Defence Research and Development Canada 3701 Carling Avenue Ottawa, Ontario K1A 0Z4 Canada 9a. PROJECT OR GRANT NO. (If appropriate, the applicable research and development project or grant number under which the document was written. Please specify whether project or grant.) 9b. CONTRACT NO. (If appropriate, the applicable number under which the document was written.) 10a. ORIGINATOR S DOCUMENT NUMBER (The official document number by which the document is identified by the originating activity. This number must be unique to this document.) 10b. OTHER DOCUMENT NO(s). (Any other numbers which may be assigned this document either by the originator or by the sponsor.) DRDC-RDDC-2017-N031 11a. FUTURE DISTRIBUTION (Any limitations on further dissemination of the document, other than those imposed by security classification.) Public release 11b. FUTURE DISTRIBUTION OUTSIDE CANADA (Any limitations on further dissemination of the document, other than those imposed by security classification.) Public release CAN UNCLASSIFIED

14 CAN UNCLASSIFIED 12. ABSTRACT (A brief and factual summary of the document. It may also appear elsewhere in the body of the document itself. It is highly desirable that the abstract of classified documents be unclassified. Each paragraph of the abstract shall begin with an indication of the security classification of the information in the paragraph (unless the document itself is unclassified) represented as (S), (C), (R), or (U). It is not necessary to include here abstracts in both official languages unless the text is bilingual.) The ability to detect and unambiguously follow all moving entities in a state-space is important in multiple domains both in defence (e.g. air surveillance, maritime situational awareness, ground moving target indication) and the civil sphere (e.g. astronomy, biology, epidemiology, dispersion modelling). However, tracking and state estimation researchers and practitioners have difficulties recreating state-of-the-art algorithms in order to benchmark their own work. Furthermore, system developers need to assess which algorithms meet operational requirements objectively and exhaustively rather than intuitively or driven by personal favourites. We have therefore commenced the development of a collaborative initiative to create an open source framework for production, demonstration and evaluation of Tracking and State Estimation algorithms. The initiative will develop a (MIT-licensed) software platform for researchers and practitioners to test, verify and benchmark a variety of multi-sensor and multiobject state estimation algorithms. The initiative is supported by four defence laboratories, who will contribute to the development effort for the framework. The tracking and state estimation community will derive significant benefits from this work, including: access to repositories of verified and validated tracking and state estimation algorithms, a framework for the evaluation of multiple algorithms, standardisation of interfaces and access to challenging data sets. 13. KEYWORDS, DESCRIPTORS or IDENTIFIERS (Technically meaningful terms or short phrases that characterize a document and could be helpful in cataloguing the document. They should be selected so that no security classification is required. Identifiers, such as equipment model designation, trade name, military project code name, geographic location may also be included. If possible keywords should be selected from a published thesaurus, e.g., Thesaurus of Engineering and Scientific Terms (TEST) and that thesaurus identified. If it is not possible to select indexing terms which are Unclassified, the classification of each should be indicated as with the title.) Fusion; Tracking; state estimation; open source; software development CAN UNCLASSIFIED