Kodo - Cross-platform Network Coding Software Library

Transcription

1 Kodo - Cross-platform Network Coding Software Library

2 Background Academia Network coding key enabler for efficient user cooperation (p2p). Kodo developed during a 3 year research project CONE (COoperation and NEtwork Coding). Concluded Industry On campus start-up Steinwurf ApS founded in Taking over the rights for Kodo and development. Library source code fully available. Licenses: a. Free for Research / Educational b. Paid Commercial

3 Where does Kodo fit? Many different requirements Deterministic vs. random, inter- vs. intra-flow, physical to application / transport layer. Current versions of Kodo implement Software & Digital Random Linear Network Coding (RLNC) Suitable for transport / application layer protocol implementations Focus on the coding Application Data Input Data partitioning Encoder Recoder Decoder Data output TCP / UDP

4 Kodo (the library) C++11 (staying compatible with major compilers). Designed to allow for easy experimentation and a high degree of code reuse. Very flexible design technique used called "mixin-layers" or "parameterized inheritance" using C++ templates. Low-level = ample ways of shooting yourself in the foot. With API specs. we try to mitigate this. High Performance - code generated by compiler comparable to single monolithic implementation.

5 Using Kodo Kodo depends on three external libraries All dependencies are header-only (we just set the include path). In total you need to download these four libraries to use Kodo in your application. 1. Sak is a set of generic C++ components. 2. Fifi contains finite field arithmetic. 3. Boost C++ libraries (slowly being replaced by C++11). 4. Kodo itself is also header-only Options for building/using Kodo: a. Using makefile (generic solution) b. Waf build script Visual Studio solution generator

6 Using Kodo From the kodo/examples/sample_makefile example: # The include path to the kodo sources kodo_dir =../../src # The include path to the sak sources sak_dir = ~/dev/bundle_dependencies/sak-master/src # The include path to the fifi sources fifi_dir = ~/dev/bundle_dependencies/fifi-master/src # The include path to the boost sources boost_dir = ~/dev/bundle_dependencies/boost-master/boost-lib # Invoke the compiler all: g++ main.cpp -o test --std=c++0x -I $(boost_dir) -I $(fifi_dir) -I $(kodo_dir) -I $(sak_dir)

7 Example: Simple encoding / decoding Encodes and decodes a single generation. Try changing the field size, symbol size and symbols and monitor the payload use and symbols needed to decode. Use-case example: Reliable multicast / broadcast Example basic.cpp

8 Review example // Set the number of symbols (i.e. the generation size in RLNC // terminology) and the size of a symbol in bytes uint32_t max_symbols = 16; uint32_t max_symbol_size = 1400; symbols Number of packets in a block / generation Increase -> Larger computational complexity Increase -> Larger per-packet overhead (coding coefficients) Increase -> Larger per-packet decoding delay Increase -> Less protocol complexity Increase -> Less dependency on field size symbol size The size of a symbol in bytes Depends on the transport used (TCP, UDP, other)

9 Review example Finite field (rule of thumb) Increase -> Larger computational complexity Increase -> Larger decoding probability typedef kodo::full_rlnc_encoder<fifi::binary8> rlnc_encoder; typedef kodo::full_rlnc_decoder<fifi::binary8> rlnc_decoder; For a field size of q we use log2(q) bits per coefficient The number of coefficients per packet equal the number of symbols Example fifi::binary = 1 bit per coefficient fifi::binary8 = 1 byte per coefficient fifi::binary16 = 2 bytes per coefficients

10 Review example rlnc_encoder::factory encoder_factory(max_symbols, max_symbol_size); auto encoder = encoder_factory.build(); rlnc_decoder::factory decoder_factory(max_symbols, max_symbol_size); auto decoder = decoder_factory.build(); In use: encoder encoder encoder factory pool Free: encoder encoder encoder encoder encoder Factories Pre-allocates resources for all configurations below the max values Initializes encoders / decoders Can recycle resources - internal pool of encoders / decoders When a encoder/decoders goes out of scope

11 Review Example payload Buffer containing coded data to be transmitted from encoder to decoder payload size The payload buffer size bytes std::vector<uint8_t> payload(encoder->payload_size()); std::vector<uint8_t> block_in(encoder->block_size()); uint32_t bytes_used = encoder->encode( &payload[0] ); block size The total number of application bytes we can encode and decode with this configuration bytes used The number of bytes that we have to transmit over the network (bytes used <= payload size)

12 std::vector<uint8_t> payload(encoder->payload_size()); std::vector<uint8_t> block_in(encoder->block_size()); Review Example uint32_t bytes_used = encoder->encode( &payload[0] ); Payload size header symbol id coded payload Optional information about the coded symbols Describes how the coded payload was produced (the coding coefficients) The coded payload (linear combination of original data)

13 Adding loss Encoder Decoder sent received Notice we finish anyway In fact we will always finish (if error < 1) This "feature" is called rate-less. Example add_loss.cpp Notice that the bytes used increases Happens after sending symbols payloads This is because we exit the "systematic" phase

14 Systematic What does it mean Send everything uncoded once first Avoid if We do not know the state of our receivers We have multiple sources sending at the same time Advantages If we have no errors we pay nothing for the coding If we do have errors we benefit from systematic symbols in the decoding Example turn_systematic_off.cpp

15 Seed based encoder / decoder symbols RNG(seed) 4 byte byte fifi::binary8 fifi::binary bit Purpose Reduce the overhead by the added coding coefficients Disadvantage Does not support recoding Example seed_codes.cpp

16 Example: Recoding Recoding is one of the key features of a RLNC code. In this example we will look at how to recode packets using Kodo. Use-case example: Relay network, p2p, multi-path. Example recoding.cpp

17 Customizing an Encoder / Decoder Kodo is a set of building blocks rather than one specific code. It is meant to be modified. Use-cases: You want to change which symbols are included based on receiver feedback. You want to track the time elapsed since a specific decoder last received a packet. You want to collect statistics about the decoding You want to encode and decode using a sliding window...

18 Mixin-Layers / Parameterized inheritance In this context a mixin is not a stand-alone class but represents a "slice" of functionality. The slices are then combined to compose the final functionality. In C++ we do this through templates. Kodo heavily relies on this technique. Flexibility Performance 1. template<class T> 2. class foo 3. { 4. public: 5. T m_t; 6. }; foo<int> f1; 10. foo<my_other_type> f2;

19 Mixin-Layers 1. template<class Super> 2. class add_layer : public Super 3. { 4. public: int add(int a, int b) 7. { 8. return a + b; 9. } 10. }; class final_layer 13. { }; class calculator 16. : public add_layer<final_layer> > 17. { }; 1. int main() 2. { 3. calculator calc; 4. std::cout << calc.add(4,2) << std::endl; 5. return 0; 6. } Output: 6

20 Mixin-Layers (adding functionality) 1. template<class Super> 2. class sub_layer : public Super 3. { 4. public: int subtract(int a, int b) 7. { 8. return a - b; 9. } 10. }; class calculator 14. : public sub_layer< 15. add_layer<final_layer> > 16. { }; 1. int main() 2. { 3. calculator calc; 4. std::cout << calc.add(4,2) << std::endl; 5. std::cout << calc.subtract(2,5) << std::endl; 6. return 0; 7. } Output: 6-3

21 Mixin-Layers (customize functionality) 1. template<class Super> 2. class modulo_layer : public Super 3. { 4. public: int add(int a, int b) 7. { 8. return Super::add(a,b) % 5; 9. } int subtract(int a, int b) 12. { 13. int res = 14. Super::subtract(a,b) % 5; 15. return res < 0? res + 5 : res; 16. } 17. }; class calculator 20. : public modulo_layer< 21. sub_layer< 22. add_layer<final_layer> > > 1. int main() 2. { 3. calculator calc; 4. std::cout << calc.add(2,2) << std::endl; 5. std::cout << calc.subtract(2,2) << std::endl; 6. return 0; 7. } Output: 1 2

22 Example: Layers used in the simple encoding / decoding 1. Open the full_vector_codes.hpp file in the kodo/src/kodo/rlnc folder. Each layer implements a specific API with a specific purpose. Read more in the "Hacking Kodo" section of the manual. Your custom functionality may be added as a layer Example add_layer.cpp

23 Kodo Layers Typical Codec Stack Payload Codec Layer Payload Codec API Codec Header API Codec Header Layer Symbol ID API Coefficient Generator API Codec Layer Construction API Codec API User API Utility Layers Symbol Storage API Coefficient Symbol Storage API Finite Field API Factory API

24

25

26

27 Kodo Documentation Manual API

28 Encoding real data Roughly two classes objects and streams. Objects have a fixed size and data is not useful until the entire object has been decoded. Streams are typically unbound and can provide useful data on-the-fly and can have delay constraints

29 Example: Storage Encoding / Decoding In this example we will look at how you encode a file or large buffer easily with Kodo Use-case: Reliable transmission of fixed buffers / files for e.g. distributed storage, software updates etc. disk 1 disk 2 disk n

30 Example: Sliding window Encoding / Decoding In this example we will look at how encode / decode content produced on the fly (variable bit-rates). Use-case: Implementation of stream based protocols e. g. for reliable video and audio streaming.

31 Kodo Testing Continuous Integration (build on every commit) Different platforms & compilers Core part of our release management

32 Kodo Performance Main thing: Measure raw coding speed. Catch performance regressions Research / Experimentation Memory access patterns Finite field operations Prove / test a clever algorithm

33 The End Contributions + bug fixes please Simple procedure with sign-off Feedback / comments /questions are all very welcome! Questions? mvp@es.aau.dk