pyfinity Documentation

Transcription

1 pyfinity Documentation Release Frank Singleton January 26, 2014

2

3 Contents 1 Fast Counters pyfinity.counter_manager_versioned Indices and tables 13 Python Module Index 15 i

4 ii

5 pyfinity is a fast, versioned, mmap counter framework for instrumenting code with low overhead (approximately 4 usec on my old laptop). These counters also exists outside the python vm both during and after the process writing counters has exited. This mmap framework can be used to to decouple counter mutation (writing) from counter reading. The latter is planned to run as a consumer of the counter data, via polling. Contents 1

6 2 Contents

7 CHAPTER 1 Fast Counters This API provides for memory mapped (typically < 4 usec) counter manipulation to support application instrumentation. By using mmap, clients can be written to read the counters periodically and forward them to other destinations as required. MMAP region indexing In order to avoid locking on a common mmap region, the following items are used as input into determining which mmap region should be used. counter group name counter type PID of invoking code TID of invoking code (actually id(tid)) 1.1 pyfinity.counter_manager_versioned pyfinity Copyright (C) , Frank Singleton Utilities for storing and mutating counters in memory mapped structure using mmap. Provides options to increment/decrement 8/16/32/64 bit counters This implementation avoids contention/overhead from mutex/locking. Single writer per mmap region based on f(pid,tid) without locking. Readers also do not use locks, but can be notified if a read was interleaved with a write. This is accomplished by using counter versions. In essence each counter has 2 copies of a version associated with it. 3

8 Counter Versioning Counter version [version_lhs] [counter] [version_rhs] The versions are stepped in the order shown as counters are mutated. If a read attempt reveals that the versions are same or different, the reader can respond appropriately. The assumption is that a read attempt interleaved write will reveal different counter versions and the reader can choose to count that read attempt as invalid. Note: There is one case of a false positive (false counter valid) and that is if the reader reads version_lhs, then the counter, but then 2^32 writes occur (for 32 bit counters) occur, and the writer version_rhs matches the version_lhs when the reader reads that value. If we assume a write takes 1 usec, that means a reader would have to stall during its read for 1e-6 * 2^32 = 4295 seconds in order to have this problem. Hmm, I encourage folks to use 32 and 64bit counters :-) NOTE: counter_index is NOT mmap offset, so do not treat it that way. Work in progress, testing execution times, typically < 4usec. The aim is to have a region of shared memory (via mmap) that threads and processes can access to provide counters for (near realtime) instrumentation capabilities. Currently there are regions dedicated for uint8, uint16, uint32, uint64 unsigned integers. 4 Chapter 1. Fast Counters

9 Supported counter types The following list indicates which counters are currently supported via mmap regions. Unsigned 8 bit counter Unsigned 16 bit counter Unsigned 32 bit counter Unsigned 64 bit counter Also, to avoid locking between process and threads when writing, each group/pid/tid/type combination will have its own mmap region. These files contain enough information for clients to attach via mmap and read counters in a robust manner. The writing of counters is independent of the reader(s) that consume this data. This provides low overhead and flexible consumer design. An example of this are writers updating counters in (near) realtime and readers polling this data on a slower interval to minimize impact on the running system. The counter framework is designed such that every writer (process/thread) writes without contention to their own mmap region, to avoid locking etc. The counter files remain on the file system until explicitly unregistered or reaped by a reaper process. This means processes outside the running Python VM s can also read counters. Timing: Laptop (Fedora 16 VM), typically < 4 usec for uint64 increment pyfinity.counter_manager_versioned.register_user(num_counters, counter_group) Register user and create corresponding mmap counter regions for requested counter type Returns num_counters The number of counters required for this user counter_type, counter_type (COUNTER_TYPE_UINT8, COUNTER_TYPE_UINT16, COUNTER_TYPE_UINT32, COUNTER_TYPE_UINT64) The counter type required for this counter group counter_group (string.) The counter group that this counter belongs to. boolean (True/False) Indicates if registration was successful or not 1.1. pyfinity.counter_manager_versioned 5

10 from pyfinity.counter_manager_versioned import COUNTER_TYPE_UINT32 register_user(10, COUNTER_TYPE_UINT32, counter_group= com.example.superapp ) This example creates space to hold 10 counters. The counters are 32 bit unsigned integers pyfinity.counter_manager_versioned.unregister_user(counter_type, counter_group, remove_file=true) Unregister user and delete corresponding mmap counter regions for requested counter type mmap is flushed first, then file is deleted if requested (default) By default, remove the related counter file. from pyfinity.counter_manager_versioned import COUNTER_TYPE_UINT32 unregister_user(counter_type_uint32, counter_group= com.example.superapp ) This example unregisters the existing 32 bit unsigned mmap region. from pyfinity.counter_manager_versioned import COUNTER_TYPE_UINT64 unregister_user(counter_type_uint64, counter_group= com.example.superapp, remove_file=false) This example unregisters the existing 64 bit unsigned mmap region. It leaves the counter file that was previously created and mmap d in place. This is sometimes useful for post analysis pyfinity.counter_manager_versioned.increment_counter_uint8(counter_index, counter_group, counter_increment=1) Increment an 8 bit unsigned integer counter for specified counter index counter_index (int) The index of the counter to increment ( 0 <= counter_index < num_counters). counter_increment The integer amount to increment the counter by. counter_group (string.) The counter group that this counter belongs to. Returns None increment_counter_uint8(86, counter_group= com.example.superapp ) This example increments an 8 bit unsigned integer belonging to the counter group com.example.superapp at index 86 by 1 (default increment) increment_counter_uint8(4, counter_group= com.example.superapp, counter_increment=3) This example increments an 8 bit unsigned integer belonging to the counter group com.example.superapp at index 4 by 3 pyfinity.counter_manager_versioned.increment_counter_uint16(counter_index, counter_group, counter_increment=1) Increment a 16 bit unsigned integer counter for specified counter index 6 Chapter 1. Fast Counters

11 counter_index (int) The index of the counter to increment ( 0 <= counter_index < num_counters). counter_increment The integer amount to increment the counter by. counter_group (string.) The counter group that this counter belongs to. Returns None increment_counter_uint16(42, counter_group= com.example.superapp ) This example increments a 16 bit unsigned integer belonging to the counter group com.example.superapp at index 42 by 1 (default increment) increment_counter_uint16(4, counter_group= com.example.superapp, counter_increment=3) This example increments a 16 bit unsigned integer belonging to the counter group com.example.superapp at index 4 by 3 pyfinity.counter_manager_versioned.increment_counter_uint32(counter_index, counter_group, counter_increment=1) Increment a 32 bit unsigned integer counter for specified counter index counter_index (int) The index of the counter to increment ( 0 <= counter_index < num_counters). counter_increment The integer amount to increment the counter by. counter_group (string.) The counter group that this counter belongs to. Returns None increment_counter_uint32(2, counter_group= com.example.superapp ) This example increments a 32 bit unsigned integer belonging to the counter group com.example.superapp at index 2 by 1 (default increment) increment_counter_uint32(4, counter_group= com.example.superapp, counter_increment=3) This example increments a 32 bit unsigned integer belonging to the counter group com.example.superapp at index 4 by 3 pyfinity.counter_manager_versioned.increment_counter_uint64(counter_index, counter_group, counter_increment=1) Increment a 64 bit unsigned integer counter for specified counter index 1.1. pyfinity.counter_manager_versioned 7

12 counter_index (int) The index of the counter to increment ( 0 <= counter_index < num_counters). counter_increment The integer amount to increment the counter by. counter_group (string.) The counter group that this counter belongs to. Returns None increment_counter_uint64(3, counter_group= com.example.superapp ) This example increments a 64 bit unsigned integer belonging to the counter group com.example.superapp at index 3 by 1 (default increment) increment_counter_uint64(4, counter_group= com.example.superapp, counter_increment=3) This example increments a 64 bit unsigned integer belonging to the counter group com.example.superapp at index 4 by 3 pyfinity.counter_manager_versioned.decrement_counter_uint8(counter_index, counter_group, counter_decrement=1) Decrement a uint8 counter for specified counter index counter_index (int) The index of the counter to decrement ( 0 <= counter_index < num_counters). counter_increment The integer amount to decrement the counter by. counter_group (string.) The counter group that this counter belongs to. Returns None decrement_counter_uint8(42, counter_group= com.example.superapp ) This example decrements an 8 bit unsigned integer belonging to the counter group com.example.superapp at index 42 by 1 (default decrement) decrement_counter_uint8(4, counter_group= com.example.superapp, counter_decrement=3) This example decrements an 8 bit unsigned integer belonging to the counter group com.example.superapp at index 4 by 3 pyfinity.counter_manager_versioned.decrement_counter_uint16(counter_index, counter_group, counter_decrement=1) Decrement a uint16 counter for specified counter index 8 Chapter 1. Fast Counters

13 counter_index (int) The index of the counter to decrement ( 0 <= counter_index < num_counters). counter_increment The integer amount to decrement the counter by. counter_group (string.) The counter group that this counter belongs to. Returns None decrement_counter_uint16(42, counter_group= com.example.superapp ) This example decrements a 16 bit unsigned integer belonging to the counter group com.example.superapp at index 42 by 1 (default decrement) decrement_counter_uint16(4, counter_group= com.example.superapp, counter_decrement=3) This example decrements a 16 bit unsigned integer belonging to the counter group com.example.superapp at index 4 by 3 pyfinity.counter_manager_versioned.decrement_counter_uint32(counter_index, counter_group, counter_decrement=1) Decrement a uint32 counter for specified counter index counter_index (int) The index of the counter to decrement ( 0 <= counter_index < num_counters). counter_increment The integer amount to increment the counter by. counter_group (string.) The counter group that this counter belongs to. Returns None decrement_counter_uint32(42, counter_group= com.example.superapp ) This example decrements a 32 bit unsigned integer belonging to the counter group com.example.superapp at index 42 by 1 (default decrement) decrement_counter_uint32(4, 3, counter_group= com.example.superapp, counter_decrement=3) This example decrements a 32 bit unsigned integer belonging to the counter group com.example.superapp at index 4 by 3 pyfinity.counter_manager_versioned.decrement_counter_uint64(counter_index, counter_group, counter_decrement=1) Decrement a uint64 counter for specified counter index 1.1. pyfinity.counter_manager_versioned 9

14 counter_index (int) The index of the counter to decrement ( 0 <= counter_index < num_counters). counter_increment The integer amount to decrement the counter by. counter_group (string.) The counter group that this counter belongs to. Returns None decrement_counter_uint64(42, counter_group= com.example.superapp ) This example decrements a 64 bit unsigned integer belonging to the counter group com.example.superapp at index 42 by 1 (default decrement) decrement_counter_uint64(4, counter_group= com.example.superapp, counter_decrement=3) This example decrements a 64 bit unsigned integer belonging to the counter group com.example.superapp at index 4 by 3 pyfinity.counter_manager_versioned.get_counter_file_mmap(counter_filename) Return a mmap.mmap object for a given counter filename Assumes that counter_filename has correct format and exists on the local file system. counter_filename The counter group that this counter belongs to. Returns mmap mmap.mmap() instance or None if not supported platform pyfinity.counter_manager_versioned.get_counter_uint8_from_mmap_ref(counter_index, mmap_ref ) Returns uint8 counter value at specified counter_index, or raises an exception if mismatched counter versions are detected This function is provided as a convenience for those callers that already have a valid mmap_ref and want to avoid the overhead of repeatedly creating a mmap_ref for each call to get_counter_file_mmap(counter_filename) counter_index (int) The counter index to read counter from. mmap_ref The memory map reference to use when fetching a counter CounterInvalidState if counter is unstable pyfinity.counter_manager_versioned.get_counter_uint16_from_mmap_ref(counter_index, mmap_ref ) Returns uint16 counter value at specified counter_index, or raises an exception if mismatched counter versions are detected This function is provided as a convenience for those callers that already have a valid mmap_ref and want to avoid the overhead of repeatedly creating a mmap_ref for each call to get_counter_file_mmap(counter_filename) counter_index (int) The counter index to read counter from. mmap_ref The memory map reference to use when fetching a counter CounterInvalidState if counter is unstable 10 Chapter 1. Fast Counters

15 pyfinity.counter_manager_versioned.get_counter_uint32_from_mmap_ref(counter_index, mmap_ref ) Returns uint32 counter value at specified counter_index, or raises an exception if mismatched counter versions are detected This function is provided as a convenience for those callers that already have a valid mmap_ref and want to avoid the overhead of repeatedly creating a mmap_ref for each call to get_counter_file_mmap(counter_filename) counter_index (int) The counter index to read counter from. mmap_ref The memory map reference to use when fetching a counter CounterInvalidState if counter is unstable pyfinity.counter_manager_versioned.get_counter_uint64_from_mmap_ref(counter_index, mmap_ref ) Returns uint64 counter value at specified counter_index, or raises an exception if mismatched counter versions are detected This function is provided as a convenience for those callers that already have a valid mmap_ref and want to avoid the overhead of repeatedly creating a mmap_ref for each call to get_counter_file_mmap(counter_filename) counter_index (int) The counter index to read counter from. mmap_ref The memory map reference to use when fetching a counter CounterInvalidState if counter is unstable pyfinity.counter_manager_versioned.get_counter(counter_index, counter_filename) Get a counter from a counter file at specified counter index Returns counter_index The index of the counter to fetch ( 0 <= counter_index < num_counters). counter_filename The name of the mmap counter file that user wishes to read. int (uint8/uint16/uint32/uint64) Unsigned integer representing counter value CounterInvalidState If counter is unstable get_counter(4, /tmp/counter_com.example.mondec.examples.example1_10_uint_64_mmap_3898_ This example fetches the 64 bit unsigned integer counter belonging to the counter group com.example.superapp at index 4 pyfinity.counter_manager_versioned.get_counters(counter_filename) Get all counters for specified counter filename counter_filename The name of the mmap counter file that user wishes to read. Returns int (uint8/uint16/uint32/uint64) List of Unsigned integers representing counter value. CounterInvalidState If ANY counter is unstable pyfinity.counter_manager_versioned 11

16 get_counter( /tmp/counter_com.example.mondec.examples.example1_10_uint_64_mmap_3898_ This example fetches ALL the 64 bit unsigned integer counter belonging to the specified counter filename pyfinity.counter_manager_versioned.get_counter_group_attributes(counter_filename) For a given counter group filename, return the following attributes as a named tuple of strings (group_name, group_size, group_type, group_pid, group_tid) 12 Chapter 1. Fast Counters

17 CHAPTER 2 Indices and tables genindex modindex search 13

18 14 Chapter 2. Indices and tables

19 Python Module Index p pyfinity.counter_manager_versioned, 3 15