SCRIPTING I/III EXTENDING PYTHON. References:

Size: px

Start display at page:

Download "SCRIPTING I/III EXTENDING PYTHON. References:"

Griffin McKenzie
5 years ago
Views:

1 SCRIPTING I/III EXTENDING PYTHON References:

2 GOALS Part I: Create a module (a C dll), callable from python [extension] Functions Class (with methods) Illustrate type conversions Introduce reference counting Part II: Create a python interpreter embedded in a C program. Coordinate python scripts and C functionality Part III: [?] Embed into our larger project. Make a (partial) mirror of our C GameObject in Python.

3 PYTHON C API OVERVIEW A massive collection of C (not C++) functions Naming conventions: Py_XXX: where XXX is a top-level function PyYYY_ZZZ: where YYY is a python type and ZZZ is the method. E.g. PyTuple_GetSize PyObject: nearly everything uses (and is one of) these primitives: int, floats complex primitives: strings, lists, tuples functions classes (and instances) modules exceptions You have to check the type (PyYYY_Check(PyObject*)). A lot of our work involves converting C <-> Python data

4 FIRST STEPS C code Python code DLL release 1 project (I'm calling mine testmod) Include directories = "%PythonHome\include" Library directories = "%PythonHome\libs" Input is "Python3.lib" (more limited, but usable on python 3.x) or "Python33.lib" (has any 3.3-specific functionality) Change name of extension to pyd Put any test python code next to the.pyd file. In this section, we'll run python code with the normal interpreter (PyCharm, Idle, cmd-line all OK) 1. Unless you build python from source, you're using a release build (so our dll has to be release as well)

5 PART I: MODULE CREATIONS + FUNCTION Warning: I know you could just zip through this lecture and copypaste. I suggest, however, that you do it with us I'll try to add running commentary as we go that will (hopefully) help you better understand it. Ask questions as we go through this!!

6 CREATING THE MODULE #include <Python.h> static struct PyModuleDef testmod_definition = PyModuleDef_HEAD_INIT, "testmod", "My super-awesome first python extension", -1, NULL }; PyMODINIT_FUNC PyInit_testMod(void) PyObject * mod = PyModule_Create(&testMod_definition); return mod; }

7 TESTING THE MODULE import testmod print("testmod. name = " + testmod. name ) print("testmod. doc = " + testmod. doc ) print(dir(testmod))

8 TANGENT: REFERENCE-COUNTING AND GARBAGE COLLECTION INTRO L = [["a", 4.7], ["b", 1.1]] # RC on all are 1 L[0] = ["c", 9.2] # RC on "a" dropped to # 0 GC G = L # RC on main list is 2 H = L[1] # RC on "b" is 2 L = "hi!" # RC on main list is 1 G = "bye!" # RC on main list is 0 GC # "b" has RC of 1 # so no GC H = "hasta la vista" # Now "b" has RC of 0 - GC

9 ADDING CONSTANTS (For example math.pi in the built-in math module) PyObject * mod = PyModule_Create(&testMod_definition); if (mod == NULL) return NULL; // Add a (totally pointless) integer constant to the module // You'll need PyModule_??? [Finish me] import testmod print("testmod.mysteryvalue = " + str(testmod.mysteryvalue))

10 ADDING A FUNCTION (GOAL) # We want to add this functionality to the module: # (as before) T = testmod.buildtuple(6, 0.3, 1) print(t) # A tuple of 6 floats in range # I was the last two params to be either # floats or int (any number type) # These should cause an error # T = testmod.buildtuple(3) # Too few parameters # T = testmod.buildtuple(6, 1.4, 1.2) # max before min

11 ADDING A FUNCTION (IMPLEMENTATION) static PyObject * testmod_buildtuple(pyobject * self, PyObject * args) if (!PyTuple_Check(args) PyTuple_Size(args)!= 3!PyLong_Check(PyTuple_GetItem(args, 0))!PyNumber_Check(PyTuple_GetItem(args, 1))!PyNumber_Check(PyTuple_GetItem(args, 2))) PyErr_SetString(PyExc_ValueError, "You must pass an int (num elements) and two floats (min & max)"); return NULL; } // Actually extract the parameters from the tuple. int num_elem = PyLong_AsLong(PyTuple_GetItem(args, 0)); PyObject * first_num = PyNumber_Float(PyTuple_GetItem(args, 1)); PyObject * second_num = PyNumber_Float(PyTuple_GetItem(args, 2)); double min_val = PyFloat_AsDouble(first_num); double max_val = PyFloat_AsDouble(second_num); Py_DECREF(first_num); Py_DECREF(second_num); if (min_val > max_val) PyErr_SetString(PyExc_ValueError, "The min-value must be smaller (or equal to) the max value"); return NULL; }

12 ADDING A FUNCTION (IMPLEMENTATION) continuation of testmod_buildtuple from last slide // Create and fill the tuple. // I want you to figure this part out. You'll likely need the // following functions: PyTuple_New, PyTuple_SetItem and PyFloat_??? [FINISH ME] }

13 ADDING A FUNCTION (IMPLEMENTATION) // Add this line to PyInit_testMod srand(time(null)); // Create this structure (top-level function "table of contents") static PyMethodDef testmod_functions[] = "buildtuple", testmod_buildtuple, METH_VARARGS, "Builds a tuple"}, NULL, NULL, 0, NULL} // Sentinel }; // Modify the testmod_defintion structure static struct PyModuleDef testmod_definition = PyModuleDef_HEAD_INIT, "testmod", "My super-awesome first python extension -1, testmod_functions };

14 PART II: PYTHON/C "CLASSES"

15 PYTHON OOP BASICS (REVIEW) # We're going to do this in C on later slides (and the labs). # Just for comparison here. class Foo(object): """ Class-level docstring """ def init (self, name): self.mname = name self.mval = 0 self.mlist = [] def str (self): return '["' + self.mname + " : " + str(self.mval) + "]" def bar(self): self.mval += 1 self.mlist.append(chr(random.randint(97,122))) return self.mval x = Foo("Bob") print(x.bar()) # 1 print(x) # "Bob" : 1 : ["q"]}

16 ADDING A CLASS (GOAL) # (Add this to our existing test code) x = testmod.testclass("bob") y = testmod.testclass("sue") print(x, y) # ["Bob" : 0], ["Sue" : 0] print(x.bar()) # 1 print(x, y) # ["Bob" : 1], ["Sue" : 0] for i in range(10): y.bar() print(x, y) # ["Bob" : 1], ["Sue" : 10]

17 OVERVIEW Create 4 additional structures: 1. [testclass] A C struct defining the C side of that "object" Can contain one or more basic types (floats, ints, etc.) Can contain PyObject's (e.g. for Python strings, lists, etc.) 2. [testclass_methods] An array of PyMethodDefs structures: one element for each non-special method in the class. 3. [testclasstype] A PyTypeObject: The definition of the new type (in Python). Includes "slots" for all the special functions (e.g. init, add, etc) 4. [testclass_members] An array of PyMemeberDef structures: one for each attribute of the class. The rest will be defining the C implementation of all methods. Most will have the signature PyObject* func(pyobject * self, PyObject * args)

18 1. THE C STRUCT AND 2. MEMBER-DEF typedef struct PyObject_HEAD PyObject * mname; int mvalue; } testclass; static PyMemberDef testclass_members[] = "mname", T_OBJECT_EX, offsetof(testclass, mname), 0, "the name of the thingy"}, "mvalue", T_INT, offsetof(testclass, mvalue), 0, "the value of the thingy"}, NULL} // Sentinel };

19 3. THE PYTYPEOBJECT A big one! Contains slots for all "special" methods (C function pointers) flags, pointers to structures (e.g. our member and method structs) NOTE: Make sure you include <structmember.h> static PyTypeObject testclasstype = PyVarObject_HEAD_INIT(NULL, 0) "testmod.testclass", /* tp_name */ sizeof(testclass), /* tp_basicsize */ 0, /* tp_itemsize */ 0, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */

20 3. THE PYTYPEOBJECT, CONT. 0, /* tp_reserved */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT Py_TPFLAGS_BASETYPE, /* tp_flags */ "This is the testclass class (within the testmod module)", /* tp_doc. */ 0, /* tp_traverse */ 0, /* tp_clear */

21 3. THE PYTYPEOBJECT, CONT. }; 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ testclass_methods, /* tp_methods */ testclass_members, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ 0, /* tp_new */

22 MODIFICATIONS TO PYINIT_TESTMOD if (PyType_Ready(&testClassType) < 0) return NULL; // Create our module PyObject * mod = PyModule_Create(&testMod_definition); if (mod == NULL) return NULL; Py_INCREF(&testClassType); PyModule_AddObject(mod, "testclass", (PyObject*)&testClassType);

23 3A. DEFINE TP_NEW AND TP_DEALLOC static PyObject * testclass_new(pytypeobject * type, PyObject * args, PyObject * kwargs) testclass * obj = (testclass*)type->tp_alloc(type, 0); if (obj!= NULL) // Initialize mname to an empty string obj->mname = PyUnicode_FromString(""); if (obj->mname == NULL) Py_DECREF(obj); return NULL; } obj->mvalue = 0; } } return (PyObject*)obj; // Insert a pointer to the above function in the tp_new "slot" in testclasstype // This effectively re-defines the " new " method in python static void testclass_dealloc(testclass * obj) Py_XDECREF(obj->mName); Py_TYPE(obj)->tp_free((PyObject*)obj); } // Insert a pointer to the above function in the tp_dealloc "slot" in testclasstype // You'll need to cast the function pointer as a destructor

24 3A. DEFINE TP_INIT static int testclass_init(testclass * self, PyObject * args, PyObject * kwargs) if (!PyTuple_Check(args) PyTuple_Size(args)!= 1!PyUnicode_Check(PyTuple_GetItem(args, 0))) PyErr_SetString(PyExc_TypeError, "You must pass a single string to this method"); return -1; // Error } PyObject * temp_str = PyTuple_GetItem(args, 0); PyObject * old_str = self->mname; Py_INCREF(temp_str); self->mname = temp_str; Py_XDECREF(old_str); self->mvalue = 0; } return 0; // Success // Insert a pointer to the above function in the tp_init "slot" in testclasstype // (You'll need to cast it as a initproc function pointer // This effectively re-defines the " init " method in python

25 4. DEFINE TESTCLASS_METHODS AND FOO (A NON-SPECIAL METHOD) static PyObject * testclass_bar(pyobject * self, PyObject * args) testclass * obj = (testclass*)self; obj->mvalue++; return PyLong_FromLong(obj->mValue); } static PyMethodDef testclass_methods[] = "foo", (PyCFunction)testClass_bar, METH_VARARGS, "Increments mvalue and returns it"}, NULL}// Sentinel };

26 QUESTIONS?

C - extensions. only a small part of application benefits from compiled code

C - extensions. only a small part of application benefits from compiled code C - EXTENSIONS C - extensions Some times there are time critical parts of code which would benefit from compiled language 90/10 rule: 90 % of time is spent in 10 % of code only a small part of application