django-import-export Documentation

Transcription

1 django-import-export Documentation Release Bojan Mihelac August 27, 2013

2

3 CONTENTS i

4 ii

5 django-import-export is a Django application and library for importing and exporting data with included admin integration. Features: support multiple formats (Excel, CSV, JSON,... and everything else that tablib support) admin integration for importing preview import changes admin integration for exporting export data respecting admin filters CONTENTS 1

6 2 CONTENTS

7 CHAPTER ONE USER GUIDE 1.1 Installation django-import-export is on the Python Package Index (PyPI), so it can be installed with standard Python tools like pip or easy_install: $pip install django-import-export 1.2 Configuration The only mandatory configuration is adding import_export to your INSTALLED_APPS. This isn t necessary, if admin integration is not used. 1.3 Getting started For example purposes, we ll be use simplified book app, here is our core.models.py: class Author(models.Model): name = models.charfield(max_length=100) def unicode (self): return self.name class Category(models.Model): name = models.charfield(max_length=100) def unicode (self): return self.name class Book(models.Model): name = models.charfield( Book name, max_length=100) author = models.foreignkey(author, blank=true, null=true) author_ = models. field( Author , max_length=75, blank=true) imported = models.booleanfield(default=false) published = models.datefield( Published, blank=true, null=true) price = models.decimalfield(max_digits=10, decimal_places=2, null=true, blank=true) 3

8 categories = models.manytomanyfield(category, blank=true) def unicode (self): return self.name Creating import-export resource To integrate django-import-export with Book model, we will create resource class that will describe how this resource can be imported or exported. from import_export import resources from core.models import Book class BookResource(resources.ModelResource): class Meta: model = Book Exporting data Now that we have defined resource class, we can export books: >>> dataset = BookResource().export() >>> print dataset.csv id,name,author,author_ ,imported,published,price,categories 2,Some book,1,,0, ,8.85, Customize resource options By default ModelResource introspects model fields and creates import_export.fields.field attribute with appopriate widget for each field. To affect which model fields will be included in import-export resource, use fields option to whitelist fields or exclude option for to blacklist fields: class BookResource(resources.ModelResource): class Meta: model = Book exclude = ( imported, ) When defining ModelResource fields it is possible to follow model relationships: class BookResource(resources.ModelResource): class Meta: model = Book fields = ( author name,) Note: Following relationship fields sets field as readonly, meaning this field will be skipped when importing data. See Also: 4 Chapter 1. User Guide

9 Resources Declaring fields It is possible to override resource fields to change some of it s options: from import_export import fields class BookResource(resources.ModelResource): published = fields.field(column_name= published_date ) class Meta: model = Book Other fields, that are not existing in target model may be added: from import_export import fields class BookResource(resources.ModelResource): myfield = fields.field(column_name= myfield ) See Also: class Meta: model = Book Fields Available field types and options Advanced data manipulation Not all data can be easily pull off an object/model attribute. In order to turn complicated data model into a (generally simpler) processed data structure, dehydrate_<fieldname> method should be defined: from import_export import fields class BookResource(resources.ModelResource): full_title = fields.field() class Meta: model = Book def dehydrate_full_title(self, book): return %s by %s % (book.name, book.name.author) Customize widgets ModelResource creates field with default widget for given field type. If widget should be initialized with different arguments, set widgets dict. In this example widget for published field is overriden to use different date format. This format will be used both for importing and exporting resource. class BookResource(resources.ModelResource): class Meta: model = Book 1.3. Getting started 5

10 See Also: widgets = { published : { format : %d.%m.%y }, } Widgets available widget types and options Importing data Let s import data: >>> import tablib >>> from import_export import resources >>> from core.models import Book >>> book_resource = resources.modelresource_factory(model=book)() >>> dataset = tablib.dataset([, New book ], headers=[ id, name ]) >>> result = book_resource.import_data(dataset, dry_run=true) >>> print result.has_errors() False >>> result = book_resource.import_data(dataset, dry_run=false) In 4th line we use modelresource_factory to create default ModelResource. ModelResource class created this way is equal as in Creating import-export resource. In 5th line Dataset with subset of Book fields is created. In rest of code we first pretend to import data with dry_run set, then check for any errors and import data. See Also: Import data workflow for detailed import workflow descripton and customization options. Deleting data To delete objects during import, implement for_delete method on resource class. Example resource with delete field: class BookResource(resources.ModelResource): delete = fields.field(widget=widgets.booleanwidget()) def for_delete(self, row, instance): return self.fields[ delete ].clean(row) class Meta: model = Book Import of this resource will delete model instances for rows that have column delete set to Admin integration Admin integration is achived by subclassing ImportExportModelAdmin or one of mixins: from import_export.admin import ImportExportModelAdmin 6 Chapter 1. User Guide

11 class BookAdmin(ImportExportModelAdmin): # resouce_class = BookResource pass See Also: Admin available mixins and options. 1.4 Import data workflow This document describes import data workflow, with hooks that enable customization of import process import_data method arguments import_data method of import_export.resources.resource class is responsible for import data from given dataset. import_data expect following arguments: dataset REQUIRED. should be Tablib Dataset object with header row. dry_run If True, import should not change database. Default is False. raise_errors If True, import should raise errors. Default is False, which means that eventual errors and traceback will be saved in Result instance import_data method workflow 1. import_data intialize new import_export.results.result instance. Result instance holds errors and other information gathered during import. 2. InstanceLoader responsible for loading existing instances is intitalized. Different InstanceLoader class can be specified with instance_loader_class option of import_export.resources.resourceoptions. import_export.instance_loaders.cachedinstanceloader can be used to reduce number of database queries. See import_export.instance_loaders for available implementations. 3. Process each row in dataset (a) get_or_init_instance method is called with current InstanceLoader and current row returning object instance and Boolean variable that indicates if object instance is new. get_or_init_instance tries to load instance for current row or calls init_instance to init object if object does not exists yet. Default ModelResource.init_instance initialize Django Model without arguments. You can override init_instance method to manipulate how new objects are initialized (ie: to set default values). (b) for_delete method is called to determine if current instance should be deleted: OR i. current instance is deleted 1.4. Import data workflow 7

12 i. import_obj method is called with current object instance and current row. import_obj loop through all Resource fields, skipping many to many fields and calls import_field for each. (Many to many fields require that instance have a primary key, this is why assigning them is postponed, after object is saved). import_field calls field.save method, if field has both attribute and field column_name exists in given row. ii. save_instance method is called. save_instance receives dry_run argument and actually saves instance only when dry_run is False. save_instance calls two hooks methods that by default does not do anything but can be overriden to customize import process: before_save_instance after_save_instance Both methods receive instance and dry_run arguments. iii. save_m2m method is called to save many to many fields. (c) RowResult is assigned with diff between original and imported object fields as well as import type(new, updated). 4. result is returned. If exception is raised inside row processing, and raise_errors is False (default), traceback is appended to RowResult Transaction support If transaction support is enabled, whole import process is wrapped inside transaction and rollbacked or committed respectively. All methods called from inside of import_data (create / delete / update) receive False for dry_run argument. 1.5 Example app cd tests &&./manage.py runserver Username and password for admin are admin, password. 1.6 Settings IMPORT_EXPORT_USE_TRANSACTIONS Global setting controls if resource importing should use database transactions. Default is False. 8 Chapter 1. User Guide

13 1.7 TODO 1.8 Contributing Code guidelines As most projects, we try to follow PEP8 as closely as possible Most pull requests will be rejected without proper unit testing 1.9 Change Log Use field_name instead of column_name for field dehydration, FIX #36 Handle OneToOneField, FIX #17 - Exception when attempting access something on the related_name. FIX #23 - export filter not working Fix packaging DB transactions support for importing data support for deleting objects during import bug fixes Allowing a field to be dehydrated with a custom method added documentation added ExportForm to admin integration for choosing export file format refactor admin integration to allow better handling of specific formats supported features and better handling of reading text files include all avialable formats in Admin integration bugfixes Refactor api 1.7. TODO 9

14 10 Chapter 1. User Guide

15 CHAPTER TWO API DOCUMENTATION 2.1 Resources Resource class import_export.resources.resource Resource defines how objects are mapped to their import and export representations and handle importing and exporting data. after_delete_instance(instance, dry_run) Override to add additional logic. after_save_instance(instance, dry_run) Override to add additional logic. before_delete_instance(instance, dry_run) Override to add additional logic. before_save_instance(instance, dry_run) Override to add additional logic. export(queryset=none) Exports a resource. for_delete(row, instance) Returns True if row importing should delete instance. Default implementation returns False. Override this method to handle deletion. get_diff(original, current, dry_run=false) Get diff between original and current object when import_data is run. dry_run allows handling special cases when object is not saved to database (ie. m2m relationships). get_diff_headers() Diff representation headers. classmethod get_field_name(field) Returns field name for given field. get_fields() Returns fields in export_order order. import_data(dataset, dry_run=false, raise_errors=false, use_transactions=none) Imports data from dataset. 11

16 use_transactions If True import process will be processed inside transaction. If dry_run is set, or error occurs, transaction will be rolled back. import_obj(obj, data) save_m2m(obj, data, dry_run) Saves m2m fields. Model instance need to have a primary key value before a many-to-many relationship can be used ModelResource class import_export.resources.modelresource ModelResource is Resource subclass for handling Django models. classmethod widget_from_django_field(f, default=<class import_export.widgets.widget >) Returns the widget that would likely be associated with each Django type. classmethod widget_kwargs_for_field(field_name) Returns widget kwargs for given field_name ResourceOptions (Meta) class import_export.resources.resourceoptions The inner Meta class allows for class-level configuration of how the Resource should behave. The following options are available: fields - Controls what introspected fields the Resource should include. A whitelist of fields. exclude - Controls what introspected fields the Resource should NOT include. A blacklist of fields. model - Django Model class. It is used to introspect available fields. instance_loader_class - Controls which class instance will take care of loading existing objects. import_id_fields - Controls which object fields will be used to identify existing instances. export_order - Controls export order for columns. widgets - dictionary defines widget kwargs for fields. use_transactions - Controls if import should use database transactions. Default value is None meaning settings.import_export_use_transactions will be evaluated. 2.2 Fields class import_export.fields.field(attribute=none, column_name=none, widget=none, readonly=false) Field represent mapping between object field and representation of this field. attribute string of either an instance attribute or callable off the object. column_name let you provide how this field is named in datasource. widget defines widget that will be used to represent field data in export. readonly boolean value defines that if this field will be assigned to object during import. clean(data) Takes value stored in the data for the field and returns it as appropriate python object. 12 Chapter 2. API documentation

17 export(obj) Returns value from the provided object converted to export representation. get_value(obj) Returns value for this field from object attribute. save(obj, data) Cleans this field value and assign it to provided object. 2.3 Widgets class import_export.widgets.widget Widget takes care of converting between import and export representations. Widget objects have two functions: converts object field value to export representation converts import value and converts it to appropriate python representation clean(value) Returns appropriate python objects for import value. render(value) Returns export representation of python value. class import_export.widgets.integerwidget Widget for converting integer fields. class import_export.widgets.decimalwidget Widget for converting decimal fields. class import_export.widgets.charwidget Widget for converting text fields. class import_export.widgets.booleanwidget Widget for converting boolean fields. class import_export.widgets.datewidget(format=none) Widget for converting date fields. Takes optional format parameter. class import_export.widgets.foreignkeywidget(model, *args, **kwargs) Widget for ForeignKey model field that represent ForeignKey as integer value. Requires a positional argument: the class to which the field is related. class import_export.widgets.manytomanywidget(model, *args, **kwargs) Widget for ManyToManyField model field that represent m2m field as comma separated pk values. Requires a positional argument: the class to which the field is related. 2.4 Instance loaders class import_export.instance_loaders.baseinstanceloader(resource, dataset=none) Base abstract implementation of instance loader Widgets 13

18 class import_export.instance_loaders.modelinstanceloader(resource, dataset=none) Instance loader for Django model. Lookup for model instance by import_id_fields. class import_export.instance_loaders.cachedinstanceloader(*args, **kwargs) Loads all possible model instances in dataset avoid hitting database for every get_instance call. This instance loader work only when there is one import_id_fields field. 2.5 Admin import_export.admin.default_formats = (<class import_export.formats.base_formats.csv >, <class import_export import / export formats class import_export.admin.exportmixin Export mixin. change_list_template = admin/import_export/change_list_export.html template for change_list view export_template_name = admin/import_export/export.html template for export view formats = (<class import_export.formats.base_formats.csv >, <class import_export.formats.base_formats.xls >, <cl available import formats get_export_formats() Returns available import formats. get_export_queryset(request) Returns export queryset. Default implementation respects applied search and filters. resource_class = None resource class to_encoding = utf-8 export data encoding class import_export.admin.importexportmixin Import and export mixin. change_list_template = admin/import_export/change_list_import_export.html template for change_list view class import_export.admin.importexportmodeladmin(model, admin_site) Subclass of ModelAdmin with import/export functionality. class import_export.admin.importmixin Import mixin. change_list_template = admin/import_export/change_list_import.html template for change_list view formats = (<class import_export.formats.base_formats.csv >, <class import_export.formats.base_formats.xls >, <cl available import formats from_encoding = utf-8 import data encoding 14 Chapter 2. API documentation

19 get_import_formats() Returns available import formats. import_action(request, *args, **kwargs) Perform a dry_run of the import to make sure the import will not result in errors. If there where no error, save the user uploaded file to a local temp file that will be used by process_import for the actual import. import_template_name = admin/import_export/import.html template for import view process_import(request, *args, **kwargs) Perform the actual import action (after the user has confirmed he wishes to import) resource_class = None resource class 2.6 Results Result class import_export.results.result(*args, **kwargs) 2.6. Results 15

20 16 Chapter 2. API documentation

21 PYTHON MODULE INDEX i import_export.admin,?? import_export.instance_loaders,?? 17