Gadgetron: An Open Source Framework for Medical Image Reconstruction

Transcription

1 COMPUTER PROCESSING AND MODELING - Full Papers Magnetic Resonance in Medicine 69: (2013) Gadgetron: An Open Source Framework for Medical Image Reconstruction Michael Schacht Hansen 1 * and Thomas Sangild Sørensen 2,3 This work presents a new open source framework for medical image reconstruction called the Gadgetron. The framework implements a flexible system for creating streaming data processing pipelines where data pass through a series of modules or Gadgets from raw data to reconstructed images. The data processing pipeline is configured dynamically at run-time based on an extensible markup language configuration description. The framework promotes reuse and sharing of reconstruction modules and new Gadgets can be added to the Gadgetron framework through a plugin-like architecture without recompiling the basic framework infrastructure. Gadgets are typically implemented in C/C++, but the framework includes wrapper Gadgets that allow the user to implement new modules in the Python scripting language for rapid prototyping. In addition to the streaming framework infrastructure, the Gadgetron comes with a set of dedicated toolboxes in shared libraries for medical image reconstruction. This includes generic toolboxes for data-parallel (e.g., GPU-based) execution of compute-intensive components. The basic framework architecture is independent of medical imaging modality, but this article focuses on its application to Cartesian and non-cartesian parallel magnetic resonance imaging. Magn Reson Med 69: , Wiley Periodicals, Inc. Key words: image reconstruction; open source; GPU INTRODUCTION Image reconstruction software is an integral part of all modern medical imaging devices, and medical image reconstruction research is a strong and active field with hundreds of articles published each year. In the field of magnetic resonance imaging (MRI), great advances have been driven by image reconstruction algorithms. Examples include parallel MRI reconstruction (1 3) and more recently compressive sensing (4,5). Most image reconstruction algorithms are published without a reference implementation (i.e., without source code). In some cases, the authors or the device vendors they collaborate with are reluctant to share their algorithms. In many other cases, there is simply no practical 1 Division of Intramural Research, National Heart Lung and Blood Institute, National Institutes of Health, Bethesda, Maryland, USA 2 Department of Computer Science, Aarhus University, Aarhus, Denmark 3 Department of Clinical Medicine, Aarhus University, Aarhus, Denmark *Correspondence to: Michael S. Hansen, Ph.D., National Heart, Lung, and Blood Institute, NIH, NIH Building 10/B1D-405, 10 Center Drive, Bethesda, MD michael.hansen@nih.gov Disclosure: The National Heart, Lung, and Blood Institute and Siemens Medical Systems have a Cooperative Research and Development Agreement (CRADA). Received 16 March 2012; revised 25 April 2012; accepted 2 June DOI /mrm Published online 12 July 2012 in Wiley Online Library (wileyonlinelibrary.com) Wiley Periodicals, Inc way of sharing the algorithms; they may rely on a great deal of accessory code, some of which could be vendor specific or even contain vendor-provided code that cannot be shared. Regardless of the reasons, it undermines the scientific process that readers and reviewers are prevented from reproducing the results of reconstruction research articles. It is exceedingly difficult for other researchers to evaluate how a given algorithm might perform given a different type of data or how it might interact with other algorithms. As a result, researchers who wish to build on previous work by other groups often have to reimplement previous work for comparison purposes, as there is no available common platform for sharing medical image reconstruction algorithms. Given the complexities of new reconstruction algorithms, it may be difficult or indeed impossible reimplement the methods described in research articles. This problem has been pointed out by other researchers, e.g. in the report by Buckheit and Donoho (6). The proof-of-concept implementations that form the basis of many publications have further limitations. They are often unsuitable for deployment in a clinical (or clinical research) environment where they could be tested in broader patient populations. More specifically, the reconstruction algorithm development is often done in offline applications that are difficult to integrate with medical devices in such a way that the reconstruction can be performed online. Consequently, proper evaluation of a given reconstruction algorithm is not often done until a medical device vendor implements it in a commercial product. Furthermore, when developers wish to deploy their algorithms clinically they could be limited by the device manufacturers reconstruction development environment. As manufacturers are under regulatory restrictions and consequently have long lead times on integration of the latest hardware and software in their platforms, delays in early testing of new algorithms can result. Moreover, a new algorithm may rely on more computational power than that is available from the manufacturers hardware; the algorithm may rely on dedicated high-performance computing equipment such as graphics processing units (GPUs) (7 9), or the algorithm may simply need some software libraries that are not available or easy to integrate into a commercial medical device. Thus, it would be desirable to have an open source platform for implementing medical image reconstruction algorithms in general and more specifically MRI reconstruction algorithms. To address the above-mentioned concerns, such a platform should have the following characteristics: Free. The platform should be freely available to all researchers. Specifically, it should be provided with

2 Gadgetron 1769 full source code at no cost. Users should be allowed to modify, redistribute, repackage, and deploy the platform with no restrictions. Modular. Except in the most trivial cases, medical image reconstruction algorithms consist of multiple steps that must be performed accurately and in a timely fashion to ensure adequate performance of the medical imaging device. Many of these steps (e.g., basic data filtering) are shared among many different reconstruction algorithms and the platform should facilitate reuse of such components. This would enable developers to focus on the new algorithm. Flexible. Although medical image reconstruction algorithms share many common components, there are significant differences between algorithms and data structures for different imaging devices. For instance, MRI and positron emission tomography would likely have different data structures and different data flows as dictated by the fact that MRI data are acquired (often line by line) in the spatial frequency domain (k-space) and positron emission tomography raw data would be a list of coincidence events at detectors surrounding the sample. A practical platform for image reconstruction should allow such differences in data types and algorithms. Facilitate cross platform Development. The platform should be operating-system independent such that the users can then deploy it on their platform of choice, which may be dictated by the medical imaging device or their local computing environment. Facilitate prototyping. It is often easier to prototype new reconstruction algorithms in high-level interactive environments such as Matlab and Python. The platform should support reconstruction algorithms partly implemented in a scripting language, so changes can be made to the algorithm quickly and without recompilation during the development process. Facilitate high-performance computing. GPUs and emerging many-core CPUs have added a powerful and affordable high-performance computing technology to the medical imaging reconstruction community. A suitable reconstruction framework must enable a seamless integration of such technologies. Moreover, developers should be able to integrate high-performance computing technology that was not defined at the time when the framework was developed, i.e., a new type of parallel processor. This can be achieved in part through a modular design where individual modules can be compiled and inserted in the framework using a plugin architecture. Facilitate deployment. It should be possible to deploy the newly developed reconstruction algorithms in a clinical research environment. This deployment must be transparent to the users; it should be online in the sense that reconstructed images are created either concurrently with or immediately after data acquisition such that the images can be used in the regular clinical or research workflow. This article describes a modular open source medical image reconstruction framework called Gadgetron, which has been designed to encompass all of the aforementioned requirements. The Gadgetron contains a streaming reconstruction pipeline in which data passes through a set of modules or Gadgets. The framework can be configured dynamically by assembling and connecting these Gadgets. In addition, the Gadgetron has a set of toolboxes that provide common image reconstruction tools, data structures, iterative solvers, and GPU components. The architecture is described here in detail and framework functionality is illustrated with MRI reconstruction examples. ARCHITECTURE AND IMPLEMENTATION The design of the Gadgetron is based on the observation that most image reconstruction algorithms consist of a set of well-defined tasks that can be modularized and reused in several different reconstruction scenarios. As a simple example, a k-space raw data Hamming window may be used in multiple different MRI reconstructions to reduce ringing artifacts. One can think of each of these reconstruction modules as a Gadget that data are fed through; it has an input and an output. Multiple Gadgets are wired together (connected) to assemble a reconstruction process (a Gadget stream). The Gadget stream is configured dynamically at run-time using a plugin architecture. More specifically, the Gadgets of a particular reconstruction stream can reside in multiple independently compiled binaries (shared libraries) and are loaded at run-time based on a Gadget stream configuration in an extensible markup language (XML) file. Data are passed from a given client into the Gadgetron streaming architecture through a Transmission Control Protocol/Internet Protocol (TCP/IP) connection through which the resulting images are eventually returned to the client application. Each Gadget may implement a given task entirely, or it may rely on a set of support libraries or toolboxes. The Gadgetron framework contains several toolboxes, which provide a wide range of functionality, e.g., basic data structures such as multidimensional arrays (for CPUs and GPUs), Cartesian and non-cartesian Fourier transforms, basic linear algebra subprograms (BLAS) (10) and LAPACK (11) interfaces, iterative solvers, etc. The toolboxes can of course also be used as shared libraries in independent stand-alone image reconstruction programs that do not use the Gadgetron streaming architecture but provide a more traditional reconstruction executable for reconstruction tasks. The Gadgetron is implemented in C++ using several existing open source libraries. Network communications, streaming architecture, and message queues are implemented using the ADAPTIVE Communication Environment (12), and some memory management features such as smart pointers and Python interfacing are derived from the Boost C++ Libraries. Some Gadgets are implemented on the GPU to achieve faster reconstructions. All current GPU examples in the Gadgetron framework are implemented using Compute Unified Device Architecture (CUDA, Nvidia, Santa Clara, CA). In the following sections, we provide design and implementation details for the individual components of the Gadgetron framework. When referring to specific

3 1770 Hansen and Sørensen FIG. 1. Gadgetron architecture. The Gadgetron is in communication with a client application through a TCP/IP connection. The client application sends data to the Gadgetron and associated with each data package is a Message ID. Based on the Message ID, control of the socket is handed over to a specific Reader, which is capable of deserializing the incoming data package. The data package is converted to message blocks that are added to the first Gadget s queue. Data are then passed down the Gadget stream where each Gadget can modify and transform the data. Any Gadget can return images (or partially processed data) to the Gadgetron framework. Based on the Message ID of this return data package, the control of the socket and the data are handed to a particular Writer, which is responsible for writing the return message to the client. source code components such as class and function names, a monospace font will be used, e.g., GadgetStreamController. Gadgets and the Streaming Reconstruction Pipeline A schematic outline of the Gadgetron framework is shown in Fig. 1. A Gadgetron reconstruction process (Gadget stream) consists of three main components: Readers, Writers, and Gadgets. These components are assembled and controlled by a GadgetStreamController, which is also responsible for maintaining the connection with the client. The role of the Reader is to receive and deserialize data from the client, and the Writer serves the (opposite) role of serializing and returning data to the client. Once the data has been received, it is passed down the chain of Gadgets. Each Gadget can modify or transform the data before passing it on to the subsequent Gadget. This sections describes the function and relationship of these components in more detail. The Gadgetron maintains communication with client applications through a TCP/IP socket. Connections with multiple simultaneous clients are supported and each individual connection is managed by a software module (a class) called GadgetStreamController. The communication protocol with the client application is as follows: 1. A connection request from a client is received and a new instance of a GadgetStreamController is created. 2. The client sends an XML configuration file defining the desired reconstruction pipeline to the Gadgetron. The GadgetStreamController loads the required data serialization and deserialization components (Writers and Readers, see later) and Gadgets from shared libraries as specified in the configuration file. 3. The client sends an XML file with parameters for the reconstruction. This XML document is passed to each individual Gadget, which can interpret this document in the context of the function it is responsible for. 4. The client starts streaming data to the GadgetStream- Controller, which uses its set of Readers to deserialize the data and pass it on to the Gadgets. 5. Images (or partially reconstructed datasets) are returned to the GadgetStreamController from the Gadgets. The GadgetStreamController uses Writers to serialize the reconstructed images and pass them back to the client. Associated with each Gadgetron stream is a set of Readers and Writers, which are responsible for the deserialization and serialization of data packages on the TCP/IP connection (see Fig. 1). As the XML configuration file is received from the client, the GadgetStreamController loads the corresponding set of Readers and Writers and associates each Reader and Writer with a unique Message ID. Each transmission of data is initiated with transmission of a Message ID such that the GadgetStreamController can pass control of the socket connection to the appropriate Reader for deserialization of the data package. When returning data packages (images) to the client, the packages are read from an output queue where they are again associated with a Message ID that allows the GadgetStreamController to pick the appropriate Writer for serialization of the package. This abstraction of serialization and deserialization allows flexible configuration of the Gadgetron framework where several different types of packages can be processed by the same Readers and Writers when applicable. Moreover, the behavior of the Gadgetron framework in response to a specific type of data can be changed by replacing the Reader/Writer associated with that type of data. For example, in one configuration the Gadgetron framework may pass a specific type of MRI raw data directly to the Gadget stream and in

4 Gadgetron 1771 FIG. 2. Each Gadget is an active object with at least one thread of execution. The thread is responsible for picking generic data packages of the input message queue and decoding the data packages into the data types that the Gadget is expecting. If the data types match the Gadgets signature, the data will be passed on to a data processing function where the data can be modified and/or transformed into new data packages and enqueued on the next Gadget s input queue. another configuration the same data type may be corrected or filtered in some way depending on the Reader associated with a specific Message ID. The Gadgetron framework provides abstract classes for Readers/Writers and a specific deserialization/serialization behavior can be achieved by implementing these abstract interfaces. One of the toolboxes (see later) contains a GadgetronConnector class that allows the user to implement communication with the Gadgetron framework by registering Readers and Writers with the GadgetronConnector. Specific examples on how to interface raw data and image data are contained in the distribution. The basic design of a Gadget is illustrated in Fig. 2. Each individual Gadget is an active object (13). It has its own thread (or threads) of execution and an input message queue, which holds data packages from upstream Gadgets. The data packages are processed in the order they arrive at the Gadget. Each Gadget has a built-in system for checking that the data on the input queue is of the type that the Gadget is expecting. In many practical applications, a Gadget would operate on more than one data argument (e.g., it may operate on a data header and a multidimensional array containing the actual image data). The Gadgetron framework provides a set of templated Gadget classes that process different number of arguments. Users of the Gadgetron framework can implement a Gadget that processes two arguments by simply inheriting from one of the templates and implementing just one function (the process function). Toolboxes Core functionality to be shared across multiple Gadgets, and standalone applications is provided in toolboxes. In this section, we provide a nonexhaustive list of some key toolbox functionality. Multidimensional arrays. The framework comes with a templated multidimensional array (NDArray). This array structure is used throughout the Gadgetron framework to store and process data. Both CPU- and GPU-based arrays are available with operators to transparently handle memory transfers between the available computational resources. Fast Fourier transforms. There are multiple toolbox components that provide access to various flavors of the Fast Fourier Transform (FFT). CPU-based FFT is available using the FFTW3 library (14) and GPU-based using the CUFFT library from Nvidia. The framework also contains a GPU implementation of the nonuniform FFT (8). Iterative solvers. The framework provides a set of templated classes for iterative reconstruction. A solver base class is provided and a number of iterative solver implementations are derived from this base class such as a Conjugate Gradient solver for linear least square problems (15) and Split Bregman solvers for total variation/compressed sensing based reconstructions (16). The solvers are completely generic and straightforward to configure for suitable reconstruction problems. For example, the conjugate gradient solver can be used to solve a minimization problem such as min ρ Eρ m 2 2 where ρ is the unknown image to be reconstructed, E is the image modality encoding matrix, and m is the measured data. The user must supply a matrix operator that implements multiplication with E and E H. This is done by filling in (in a derived class) two pure virtual functions of a linearoperator base class. The solvers allow multiple operators to be added such that multiple terms can be minimized simultaneously. The source code contains multiple examples for users to build on and modify for their own specific purposes. A specific example of using these solvers for non-cartesian parallel MRI is described later. In addition to image encoding operators the framework provides a set of commonly used regularization operators such as an identity operator, a partial derivative operator, a similarity image operator, and a convolution operator; all derived from the linearoperator base class. Python Prototyping To make the transition from prototyping to deployment easier, the Gadgetron is designed so that the functionality of individual Gadgets can be implemented using Python as a scripting language. Python was chosen over Matlab, as it can be obtained at no cost on all the supported platforms. The Python architecture is outlined in Fig. 3. The Python scripting functionality can be accessed by the user through a PythonGadget, and the framework provides certain hooks that the user can implement callback functions for in a Python module. The user specifies which Python module the Gadget should load and which functions in the Python module should be called in response to which events. The PythonGadget is not responsible for loading the Python modules. Due to the way that the Python interpreter is implemented, it is not safe for multiple Gadgets to access

5 1772 Hansen and Sørensen algorithms will be added in future releases and made available continuously on the Gadgetron web page (see later). It is beyond the scope of this article to explain the algorithms in detail, but references to some key reconstruction steps are provided in the following. We use MRI as the primary demonstration modality. FIG. 3. Python prototyping architecture. The Gadgetron framework provides a specialized Gadget (PythonGadget). The PythonGadget communicates with a Python interpreter instance through a PythonCommunicator object (a process wide singleton), which ensures that only one Gadget attempts to access the interpreter at any given time. At run-time, the PythonGadget will instruct the PythonCommunicator to load (import) a Python module in the interpreter instance as specified in the XML configuration file. When data arrive in the Gadget, they are passed on to the Python module (through the PythonCommunicator). Each loaded Python module receives a reference to the PythonGadget such that reconstruction results can be passed back to the calling Gadget and continue down the stream. As indicated, a given Gadget stream can contain a mixure of regular C/C++ Gadgets and Python Gadgets. the interpreter at the same time and consequently the communication with the Python interpreter is centralized in the PythonCommunicator (a process wide singleton for the Gadgetron). When the PythonGadget loads, it will request that the PythonCommunicator loads the Python module. When data need to be passed to the Python module, it will be passed first to the PythonCommunicator, which will pass it on to the appropriate Python module when the Python interpreter becomes available. As indicated in Fig. 3, it is possible to have an arbitrary number of PythonGadgets in the reconstruction pipeline. Moreover, the PythonGadgets can be mixed with standard Gadgets implemented purely in C/C++. This enables the user to reuse existing, efficient implementations of reconstruction steps while maintaining the capability of prototyping in Python. Standalone Compilation The core reconstruction algorithm components included in the framework (e.g., FFTs, iterative solvers, etc.) are contained in the various toolboxes (see earlier) and compiled into shared libraries. As previously mentioned, these toolboxes can either be linked into Gadgets or be used in standalone applications. To exemplify how to use these toolboxes in practice, we include a number of small standalone applications. These applications are compiled outside the streaming client/server architecture of the Gadgetron (Fig. 1) and demonstrate how to use the toolboxes in third-party applications. EXAMPLE APPLICATIONS This section outlines some of the example applications that are included in the initial Gadgetron distribution. More Real-Time Cartesian Parallel Imaging The Gadgetron includes a GPU-based real-time reconstruction engine for Cartesian parallel MRI. The implemented algorithm is an extension of the GRAPPA algorithm (3), which has been optimized for high-throughput reconstruction of real-time MRI using a large number of receive channels. To our knowledge, the closest implementation in the literature is the one by Saybasili et al. (17). In the present Gadgetron implementation, the GRAPPA convolution kernels are Fourier transformed to image space and applied directly to the aliased images using pixelwise multiplications. Additionally, the image space unmixing coefficients for individual channels are combined in image space using a B1-weighted coil combination procedure as per Walsh et al. (18). Data are assumed to be acquired in a timeinterleaved fashion, so that a number of neighboring frames can be averaged and used for calibration data (19). Figure 4 outlines the Gadgetron GRAPPA reconstruction chain. The core algorithm is implemented in the GrappaGadget and the GrappaUnmixingGadget. The GrappaGadget is responsible for monitoring the acquired raw data and will trigger an update of GRAPPA coefficients when required (and data are available). The GRAPPA unmixing coefficients and the required coil sensitivity maps for B1-weighted coil combination are calculated on the GPU by the GrappaCalculator. Once the GRAPPA coefficients have been calculated, they are downloaded to the CPU where they are stored in a memory structure that can be accessed by the downstream GrappaUnmixingGadget. The GrappaUnmixingGadget performs the Fourier transform of k-space data and linear combination of the available coils using the coefficients calculated by the GrappaCalculator. The GRAPPA calculation is done asynchronously with Fourier transform and image unmixing, which enables this configuration to have a high frame rate. The unmixing coefficients are updated as often as the hardware is capable and always when the slice orientation changes. The time that it takes to update the unmixing coefficients depends on the number of receiver channels, the image matrix size, and the available hardware, but for most applications it is on the order of 200 ms. More specifically, on a 16 CPU core (Intel Xeon 2.67 GHz) computer with 24 GB of RAM and an Nvidia GeForce GTX 590 graphics card, the GRAPPA unmixing coefficients for parallel imaging rate 4 with 16 input channels could be calculated in under 200 ms. As the coefficients are updated independently of the unmixing, the framerate is not limited by how quickly the coefficients can be calculated. The framerate is determined by how quickly the raw data can be Fourier transformed and the coefficients can be applied. With the previously described hardware, images could be displayed on the scanner with less than one frame (approximately 100 ms) latency.

6 Gadgetron 1773 number of target channels included in the GRAPPA reconstruction. For example, the GRAPPA coefficients may be calculated from 16 source channels to eight target channels. This reduces the computational load and memory requirements for the GRAPPA calculation without explicitly reducing the parallel imaging performance. It is beyond the scope of this article to do an analysis of appropriate settings for these compression factors, but in practice we have found that with a 32-channel receive array, a reduction to 16 channels upstream and eight channels downstream results in only marginal reduction of image quality. Non-Cartesian Parallel MRI As an example of parallel MRI using non-cartesian trajectories, the Gadgetron includes a GPU-based implementation of non-cartesian SENSE (23). The implementation included in the Gadgetron is based on previous work (9) but adapted for the Gadgetron architecture. The main functionality of the iterative solver is implemented in the toolboxes described earlier. The Gadget architecture is used to wrap the solver and data handling buffers in a reusable manner. The conjugate gradient SENSE Gadget pipeline is illustrated in Fig. 5. The noise adjustment and PCA virtual coil generation Gadgets have been reused from the GRAPPA reconstruction described earlier. Additionally, a series of other Gadgets for image scaling, magnitude extraction, etc. are also reused. The non-cartesian SENSE reconstruction uses a conjugate gradient iterative solver to solve: { ρ λ = arg min Eρ m 2 ρ 2 + λ Lρ 2 } 2, [1] FIG. 4. GRAPPA Gadget chain. NoiseAdjustGadget and PCAGadget create virtual channels in which the noise is white and the data have undergone a principal component decomposition. An optional coil reduction stage is then used to remove the PCA coils. The GRAPPA coefficients are calculated in k-space but Fourier transformed to image space where they are combined into one set of unmixing coefficients as described in Ref. 17. The GRAPPA coefficient calculation also includes an optional target channel reduction for further speed-up of the reconstruction. The calculation of unmixing coefficients is asynchronous with the stream of raw data, and the coefficients (weights) are updated as frequently as possible. Sample reconstruction images are shown on the bottom right. In addition to the high-throughput GRAPPA functionality described earlier, the implementation also allows for receiver channel compression based on Principal Component Analysis (PCA) (20,21). The implemented scheme is inspired by Huang et al. (22) in that it implements two stages of channel compression. After noise prewhitening in the NoiseAdjustGadget, the channels are converted to virtual channels using PCA coefficients in the PCAGadget. In the present implementation, the coefficients are calculated based on the k-space data from the first frame of real-time data. Subsequently, the data pass through a coil reduction Gadget, which simply discards data above a certain channel number (here the number of channels is reduced from 32 to 16). In the GRAPPA calculation itself, further channel compression is possible by limiting the where ρ is the unknown image to be reconstructed, m is the measured (possibly undersampled) data, and E is the encoding matrix describing (a) multiplication with coil sensitivities, (b) Fourier transform to k-space, and (c) sampling on the non-cartesian trajectory. The matrix L is a diagonal regularization matrix, and λ is a regularization weight that controls the balance between the data consistency and regularization terms in the optimization (9). In practice, information about coil sensitivity distributions is needed to apply E and E H during the iteration. To facilitate computation of an estimate hereof the Gadget holds a queue for buffering the data as they are received. The data are assumed to be acquired in such a way that a set of neighboring frames can be averaged to yield a fully sampled (but low temporal resolution) dataset. This can be achieved with a rotating radial acquisition scheme or with the golden ratio radial acquisition scheme (24), which was used in this case. Data from previous frames are buffered in the conjugate gradient SENSE Gadget and before reconstruction of a frame, new coil sensitivity maps are formed from the buffered data. Based on this unaliased dataset, it is also possible to extract a regularization mask that is used to populate the diagonal matrix L. The conjugate gradient SENSE Gadget has several tunable parameters that can be used to trade off image quality, signal-to-noise ratio, frame rate, and computational speed. In practice, the user can control how many k-space projections are used in each reconstructed frame (temporal resolution), how many data acquisitions that are shared among neighboring frames, the regularization factor λ, the

7 1774 Hansen and Sørensen latter often under the assumption that the image degradation was caused by convolution of the desired image with a known point spread function. The initial Gadgetron release provides three iterative solvers that can be used to restore noisy or blurred images; linear least squares by the conjugate gradient algorithm (15), and both an unconstraint and a constraint Split-Bregman solver (16) for total variation based minimization. The solvers are derived from variational problems minimizing, respectively min ρ ρ λ Eρ m 2 2 min ρ ρ TV + λ Eρ m 2 2 min ρ ρ TV s.t. Eρ m 2 2 < σ2 where ρ is the desired image to be restored, E is the linear degradation operator, and m is the acquired image. For deblurring, E is modeled as a ConvolutionOperator and for denoising as an IdentityOperator. The solvers can be set up with just a handful of lines of code and GPU-based operators are available for high-performance considerations. The framework also includes a small standalone application that applies the non-cartesian parallel MRI encoding operator (see earlier) on the latter two total variation (TV) based functionals demonstrating compressed sensing in combination with parallel MRI. FIG. 5. Non-Cartesian parallel MRI Gadget chain. Most of the reconstruction pipeline is reused from the Cartesian parallel MRI Gadget chain (Fig. 4). After forming virtual channels, the data pass into the conjugate gradient SENSE Gadget where they are buffered. Once enough data are available to reconstruct a new frame, coil sensitivities and regularization data are calculated from a time average of previously acquired data and used along with the undersampled data in a conjugate gradient iterative solver. The reconstructed images are passed on to subsequent Gadgets in the chain for further processing. number of iterations and stopping criterion in the conjugate gradient solver. Furthermore, the user can take advantage of the PCA channel compression to tune the reconstruction to the performance of the available reconstruction hardware. In the current implementation, we were able to perform non-cartesian parallel imaging reconstruction with over 10 frames per second using a single GPU for a resolution of It should be noted that PCA coil reduction from 32 to 8 virtual channels was used to achieve this framerate. Further Solver Examples A frequently occurring image reconstruction task is to reduce the degree of noise and/or blurring in an image. The Integration with a Clinical MRI System One of the key goals in designing the Gadgetron framework was to enable easy deployment in a clinical research environment. This is facilitated by the flexible TCP/IP interface that allow clients to connect from any device where it is possible to open a network socket. The exact steps involved in communication between a given device and the Gadgetron framework will vary greatly from device to device. Here an example of interfacing a clinical MRI scanner (Siemens Avanto 1.5T and Siemens Espree 1.5T, Siemens Medical Solutions, Erlangen, Germany) with the Gadgetron is demonstrated. The connection between scanner and Gadgetron is outlined in Fig. 6. In this example, the Siemens reconstruction pipeline was modified by added modules that communicate with the Gadgetron. Two new modules were added: a GadgetronEmitter and a GadgetronInjector. These modules can be inserted anywhere in the vendor reconstruction pipeline by specifying the insertion points in a configuration file. The GadgetronEmitter is responsible for opening a connection (TCP/IP) to the Gadgetron and sending parameters and raw data. When the images have been reconstructed, they will be returned to the scanner reconstruction pipeline using the GadgetronInjector. As it is possible to return reconstructed images to the scanner reconstruction pipeline before images are send to the image database on the MRI scanner host, seamless integration with the clinical scanner is achieved. Example videos can be found at

8 Gadgetron 1775 FIG. 6. Integration with a Clinical MRI Scanner (Siemens Avanto/Espree, Siemens Medical Solutions, Erlangen, Germany). The reconstruction software of the MRI system was modified by adding two additional modules to the reconstruction vendor reconstruction pipeline. One module (GadgetronEmitter) was responsible for opening a connection to the Gadgetron and transmitting data, another module was responsible for receiving reconstructed images from the Gadgetron. The modules for sending and receiving data were implemented in such a way that they can be added to any existing reconstruction program on the scanner without the need to recompile any MRI scanner software. The location of the GadgetronEmitter and GadgetronInjector in the vendor reconstruction chain is flexible and can be specified with a configuration file. The original vendor reconstruction modules (between GadgetronEmitter and GadgetronInjector) are bypassed by the Gadgetron reconstruction. AVAILABILITY AND PLATFORM SUPPORT The Gadgetron is available under a permissive, free software license based on the Berkeley Software Distribution license, which allows the users to use, modify, and redistribute the software in source or binary form with virtually no restrictions. The source code and documentation can be found on the Sourceforge Open Source distribution website ( The framework was designed to be cross platform and the source code compiles and runs on Linux, Mac OS X, and Microsoft Windows (Windows 7, 64-bit). Great care was taken during the implementation to ensure that the streaming architecture with dynamic run-time linking works on all platforms. Installation instructions are available on the Sourceforge website. DISCUSSION Limitations and Future Directions The Gadgetron is implemented mainly in C++ and takes advantage of many specific language features such as object inheritance and templates. Consequently, a certain proficiency with these features is an advantage when using the Gadgetron, and this may make it harder for programmers who are less familiar with C++ programming to use the Gadgetron. However, as a Gadget can be implemented by writing just a single function (the process class member function), users with limited C++ knowledge are expected to be able to use the Gadgetron framework. The documentation contains multiple examples of how to implement new Gadgets. There is a certain implementation overhead involved in working with the Gadgetron streaming architecture. A given algorithm has to be separated into components (Gadgets), and a client has to be written (if the included clients are not adequate). This can be more involved than writing a simple standalone C/C++ application. Furthermore, there is an inherent overhead associated with the TCP/IP communication that one would not incur in a standalone application. However, it is important to note that there are some specific advantages as well. First, the individual modules can be reused in other reconstruction programs, potentially reducing future development time. Second, the Gadgetron streaming framework is inherently multithreaded, because each Gadget has its own thread of execution, which can provide a performance increase to offset the TCP/IP overhead. In fact, in simple tests conducted to explore this, the Gadgetron streaming framework provided better performance for identical reconstructions when compared to a single threaded standalone application. It should also be noted that the toolboxes included with the Gadgetron can be used in standalone applications, which is illustrated with multiple examples in the source code. The framework is envisioned as a platform in continuous development. Functionality will be added through new additional Gadgets, but there are also infrastructural changes planned that will enable a wider range of applications. Currently, it is not possible to create branches in a Gadgetron stream, i.e., it is not possible to create streams where data pass through different sets of Gadgets based on a condition. For the applications we have focused on so far, this has not been an issue, but it is easy to envision other applications in which it would be advantageous to create multiple branches (e.g., to process different types of images). Such functionality is planned for an upcoming release. A related limitation is that there is currently no shared memory storage between different instances of Gadget chains. Such functionality would be useful, if multiple Gadget chains were active simultaneously and some data were shared between the chains (e.g., calibration data). Currently, this can be achieved using the file system or similar means of sharing data between processes, but in the future, we plan to have a more straight forward global data storage with life cycle management of the stored data objects. The GPU components in the current version of the Gadgetron are based on Nvidia s CUDA framework, which is limited to Nvidia s GPUs. To make the GPU components hardware vendor independent, it would be an advantage to use OpenCL instead. At present, however, the OpenCL architecture is less feature rich than CUDA and more cumbersome to use in templated class hierachies, and consequently the framework has limited its scope to CUDA. There are, on the other hand, no fundamental restrictions in the framework for implementing Gadgets and toolboxes using OpenCL, and future releases may be extended with OpenCL support.

9 1776 Hansen and Sørensen In the current implementation, there are some limitations in the Python scripting support. As described, the access to the Python interpreter is controlled such that two Gadgets cannot access the interpreter simultaneously. This could have performance implications if multiple Gadgets are implemented in Python and has to compete for the interpreter. With the current Python C/C++ API (version 2.7.3), it has not been possible to overcome this limitation in a way that would work robustly on all supported platforms. A future release will seek to improve the Python support and may also provide support for other scripting languages such as Matlab. Finally, the main application of the framework so far has been MRI reconstruction. Consequently, the toolboxes are focused on tasks related to this particular modality. We anticipate that the contents of the toolboxes will naturally evolve, as we and other developers embrace new applications and imaging modalities. For example, we are exploring using the framework for deblurring and denoising of light microscopy images using the iterative solvers in the toolboxes. CONCLUSIONS We have presented a new open source framework for medical image reconstruction and described several applications for which it can be used (MRI, image denoising, and image deblurring). The architecture is modular, and flexible and promotes reuse of existing reconstruction software modules in the form of Gadgets. It is possible to implement new Gadgets in C/C++ with integration of GPU acceleration for high frame rate, low latency reconstructions. It is also possible to prototype new reconstruction components using the high-level scripting language Python. The framework and all the example applications are made freely available to the medical image reconstruction community, and we hope that it can serve as a platform for researchers to share and deploy novel reconstruction algorithms. ACKNOWLEDGMENTS The authors thank Drs. Peter Kellman and Souheil Inati at the National Institutes of Health and David Hansen, Christian Christoffersen, and Allan Rasmusson at the Department of Computer Science, Aarhus University for valuable discussion, feedback, and suggestions. REFERENCES 1. Pruessmann KP, Weiger M, Scheidegger MB, Boesiger P. SENSE: sensitivity encoding for fast MRI. Magn Reson Med 1999;42: Sodickson DK, Manning WJ. Simultaneous acquisition of spatial harmonics (SMASH): fast imaging with radiofrequency coil arrays. Magn Reson Med 1997;38: Griswold MA, Jakob PM, Heidemann RM, Nittka M, Jellus V, Wang J, Kiefer B, Haase A. Generalized autocalibrating partially parallel acquisitions (GRAPPA). Magn Reson Med 2002;47: Donoho D. Compressed sensing. IEEE Trans Inform Theory 2006;52: Lustig M, Donoho D, Pauly JM. Sparse MRI: the application of compressed sensing for rapid MR imaging. Magn Reson Med 2007;58: Buckheit JB, Donoho DL. WaveLab and Reproducible Research. Technical Report 474. Department of Statistics, Stanford University, Hansen MS, Atkinson D, Sorensen TS. Cartesian SENSE and k-t SENSE reconstruction using commodity graphics hardware. Magn Reson Imaging 2008;59: Sorensen TS, Schaeffter T, Noe KO, Hansen MS. Accelerating the nonequispaced fast Fourier transform on commodity graphics hardware. IEEE Trans Med Imaging 2008;27: Sørensen TS, Atkinson D, Schaeffter T, Hansen MS. Real-time reconstruction of sensitivity encoded radial magnetic resonance imaging using a graphics processing unit. IEEE Trans Med Imaging 2009;28: Dongarra J, DuCroz J, Hammarling S, Duff I. A set of level 3 basic linear algebra subprograms. ACM Trans Math Softw 1990;16: Anderson E, Bai Z, Bischof C, Blackford S, Demmel J, Dongarra J, DuCroz J, Greenbaum A, Hammarling S, McKenney A, Sorensen D. LAPACK users guide, 3rd ed., Philadelphia, PA: Society for Industrial and Applied Mathematics; Schmidt DC. The ADAPTIVE communication environment: object- Oriented Network Programming Components for Developing Client/Server Applications. In: Proceedings of the 11th Annual Sun Users Group Conference, pp Lavender RG, Schmidt DC. Active object: an object behavioral pattern for concurrent programming. In: Vlissides JM, Coplien JO, Kerth NL, editors. Pattern languages of program design 2. Boston: Addison-Wesley, 1996, Chapter Frigo M, Johnson SG. The design and implementation of FFTW3. Proc IEEE 2005;93: Special issue on Program Generation, Optimization, and Platform Adaptation. 15. Golub G, van VanLoan C. Matrix computations, 3rd ed., Baltimore, MD: The Johns Hopkins University Press; 1996, Chapter 10, pp Goldstein T, Osher S. The split bregman method for L1-regularized problems. SIAM J Imaging Sci 2009;2: Saybasili H, Kellman P, Griswold MA, Derbyshire JA, Guttman MA. HTGRAPPA: real-time b1-weighted image domain tgrappa reconstruction. Magn Reson Med 2009;61: Walsh DO, Gmitro AF, Marcellin MW. Adaptive reconstruction of phased array MR imagery. Magn Reson Med 2000;43: Kellman P, Epstein FH, McVeigh ER. Adaptive sensitivity encoding incorporating temporal filtering (TSENSE). Magn Reson Med 2001;45: Huang F, Vijayakumar S, Li Y, Hertel S, Duensing GR. A software channel compression technique for faster reconstruction with many channels. Magn Reson Imaging 2008;26: Buehrer M, Pruessmann KP, Boesiger P, Kozerke S. Array compression for MRI with large coil arrays. Magn Reson Med 2007;57: Huang F, Lin W, Duensing GR, Reykowski A. A hybrid method for more efficient channel-by-channel reconstruction with many channels. Magn Reson Med 2012;67: Pruessmann KP, Weiger M, Börnert P, Boesiger P. Advances in sensitivity encoding with arbitrary k-space trajectories. Magn Reson Med 2001;46: Winkelmann S, Schaeffter T, Koehler T, Eggers H, Doessel O. An optimal radial profile order based on the golden ratio for time-resolved MRI. IEEE Trans Med Imaging 2007;26:68 76.