Pieces of the API Puzzle: Leveraging Typed Data in Drupal 8

Size: px

Start display at page:

Download "Pieces of the API Puzzle: Leveraging Typed Data in Drupal 8"

Audrey Sims
6 years ago
Views:

1 Pieces of the API Puzzle: Leveraging Typed Data in Drupal 8 Matthew Radcliffe pieces-api-puzzle-leveraging-typed-data-drupal-8

2 Composite Data Types Any data type made up of primitive and other composite data types i.e. structs, classes. Associative arrays Loosely-typed arrays to describe all of the things. Entity API Metadata Controllers Rules

3 Composite Data Types in Drupal 8 Typed Data is a meta for composite data types that we need in Drupal that are not otherwise defined. Strongly typed interfaces that can be passed to various subsystems in Drupal such as Serialization. Any class itself is composite data type. The Plugin API for instance is a way of describing composite data types. For now though Typed Data.

4 Use Cases for Typed Data Data Types Describe data defined elsewhere i.e. schemas from external systems. Education, Media, etc Define special snowflake data types that do not require entity storage.

5 Use Case: Xero Integration Xero accounting platform with a Restful API. Post invoices from Open Atrium cases. Invoice done. Click send. Back to coding. Post bank transactions from Commerce/Ubercart payment transactions. Auto-matching Bank Statements to Bank Transactions is huge cost savings for a small company. Do not need to store data twice. It s already stored offsite.

6 Escaping Procedural Reality Make OAuth signed requests via drupal_http_request or straight up Curl (xero_query). Validation via Drupal form validation that does not fit Web Services much at all (xero form helper). Model data as associative arrays or strongly-coupled entities (xero_make). And sometimes we have to maintain external PHP libraries we do not want to maintain

7 116 $filterid = ( count($arguments) > 0 )? strip_tags(strval($arguments[0])) : false; 117 if(isset($arguments[1])) $modified_after = ( count($arguments) > 1 )? str_replace( 'X','T', date( 'Y-m-dXH:i:s', strtotime($arguments[1])) ) : false; 118 if(isset($arguments[2])) $where = ( count($arguments) > 2 )? $arguments[2] : false; 119 if ( is_array($where) && (count($where) > 0) ) { 120 $temp_where = ''; 121 foreach ( $where as $wf => $wv ) { 122 if ( is_bool($wv) ) { 123 $wv = ( $wv )? "%3d%3dtrue" : "%3d%3dfalse"; 124 } else if ( is_array($wv) ) { 125 if ( is_bool($wv[1]) ) { 126 $wv = ($wv[1])? rawurlencode($wv[0]). "true" : rawurlencode($wv[0]). "false" ; 127 } else { 128 $wv = rawurlencode($wv[0]). "%22{$wv[1]}%22" ; 129 } 130 } else { 131 $wv = "%3d%3d%22$wv%22"; 132 } 133 $temp_where.= "%26%26$wf$wv"; 134 } 135 $where = strip_tags(substr($temp_where, 6)); 136 } else { 137 $where = strip_tags(strval($where)); 138 } 139 $order = ( count($arguments) > 3 )? strip_tags(strval($arguments[3])) : false; 140 $acceptheader = (!empty( $arguments[4] ) )? $arguments[4] : ''; 141 $method = $methods_map[$name]; 142 $xero_url = self::endpoint. $method; 143 if ( $filterid ) { 144 $xero_url.= "/$filterid"; 145 } 146 if ( isset($where) ) { 147 $xero_url.= "?where=$where"; 148 } 149 if ( $order ) { 150 $xero_url.= "&order=$order"; 151 } 152 $req = OAuthRequest::from_consumer_and_token( $this->consumer, $this->token, 'GET',$xero_url); 153 $req->sign_request($this->signature_method, $this->consumer, $this->token); 154 $ch = curl_init();

8 Drupal 8 and Packagist Provide Guzzle 3* OAuth 1.0 plugin XeroClient via XeroBundle The Composer problem. * At the time I stared through the Looking Glass, Guzzle 3 was the version in core. Guzzle 6 is included in Drupal 8 core at the time of writing this session. ** Core broke everything recently so I have to re-evaluate and figure out if I am permanently forking this library. Welp.

9 Getting off the Island The Dependency Nightmare Composer, Autoloaders, Composer Manager, Packaging. Libraries and duplicate code Some other better solution..? GitHub culture

10 Knowing the Drupal 8 API Dependency Injection Plugin API Field API, Entity API Serialization module Serializer component

11 So how do I get there? Goal: Make HTTP Requests to integrate with external system Need: Describe Xero types into something Drupal understands. Primitive and Composite data types.

12 Is Typed Data a Good Fit? Why Typed Data? Typed Data API gets all of the things, and is injectable into Forms, Route Controllers, Normalizers, etc So that can work well with Core and Contrib modules such as Serialization and Rules. Are these composite data types entities?

13 Xero Types Accounts Invoices Payments Credit Notes Bank Transactions Amount Line Items and more Contacts Users

14 Xero Types Accounts Invoices Payments Credit Notes Bank Transactions Amount Line Items and more Contacts Users

15 Xero Types Accounts Invoices Payments Credit Notes Bank Transactions Amount Line Items and more Contacts Users

16 Xero Types Accounts Invoices Payments Credit Notes Bank Transactions Amount Line Items and more Contacts Users

17 How to Read a Map Typed Data API provides most of these, yay! There s one complex data type that is not an entity. Map: the new associative array. Next challenge: how to create this elusive data type class.

18 Typed Data API Docs When I started, there was no documentation: Doc Page: API Topic: core.api.php/group/typed_data/8

19 Data Type Plugin Data Type is a plugin class. Extends TypedData. Plugin? What s that? All the things. Data Type Constructor usually inherited. Sets property values according to definition. Add useful methods to your class.

20 1 <?php 2 /** 3 4 * Provides \Drupal\xero\Plugin\DataType\Payment. 5 */ 6 7 namespace Drupal\xero\Plugin\DataType; 8 9 /** annotation type 10 * Xero Payment type. 11 * plugin id * id = "xero_payment", data definition 14 * label Payment"), 15 * definition_class = "\Drupal\xero\TypedData\Definition\PaymentDefinition" 16 * ) 17 */ 18 class Payment extends XeroTypeBase { static public $guid_name = 'PaymentID'; 21 static public $xero_name = 'Payment'; 22 static public $plural_name = 'Payments'; 23 static public $label = 'PaymentID'; }

21 Data Definitions Complex data type needs definition classes to describe its properties (or child data types). Symfony 2 Constraints Label Property data type When I began, definitions were associative arrays. This kind of still exists. The static create method takes an array as a parameter, not the class.

22 Data Definition 15 class PaymentDefinition extends ComplexDataDefinitionBase { public function getpropertydefinitions() { 23 if (!isset($this->propertydefinitions)) { 24 $info = &$this->propertydefinitions; $info['invoice'] = DataDefinition::create( xero_invoice') ->setrequired(true)->setlabel('invoice'); 27 $info['account'] = DataDefinition::create( xero_account') ->setrequired(true)->setlabel('account'); 29 $info['date'] = DataDefinition::create( string') ->setrequired(true)->setlabel('date'); 30 $info['amount'] = DataDefinition::create( float') ->setrequired(true)->setlabel('amount'); 31 $info['currencyrate'] = DataDefinition::create( string') ->setlabel('currency rate'); 32 $info['reference'] = DataDefinition::create( string') ->setlabel('reference'); 33 $info['isreconciled'] = DataDefinition::create( boolean') ->setlabel('is reconciled?'); 34 } 35 return $this->propertydefinitions; 36 } 37 }

23 List Data Definition Data definitions can be lists of data types i.e. multiple values or an indexed array equivalent. This is important for Xero because the API returns a list of items in addition to other odd types like Tracking Categories, Line Items, and Addresses.

24 Exercise: Spotify Album Track[] Artist[] Type Name

25 Album AlbumDefinition propertydefinitions name DataDefinition string type DataDefinition string artists ListDataDefinition Artist ArtistDefinition name DataDefinition string

26 Making Use of All of This Composite data types defined as Typed Data Data Types and Data Definitions. Normalization service that helps to transform Xero API into our data types and vice versa. So now we can use Serializer and Guzzle in a consistent and maintainable way. XeroQuery used by a Form to display a list of Accounts.

27 (De)normalization Normalizer classes transform data into Typed Data and vice versa. Normalization is obscured from use because it runs as part of serialization. Entity normalization is handled already. $context was a difficult parameter to understand. Typed Data Manager use is recursive and confusing.

28 Defining a Normalizer Normalization classes must be defined as services in the service container. These are also tagged so that they go through a container compiler pass. xero.services.yml services: xero.normalizer: class: Drupal\xero\Normalizer\XeroNormalizer arguments: ['@typed_data_manager'] tags: - { name: normalizer }

29 18 class XeroNormalizer extends ComplexDataNormalizer implements DenormalizerInterface { protected $supportedinterfaceorclass = 'Drupal\xero\TypedData\XeroTypeInterface'; public function construct(typeddatamanager $typed_data_manager) { 23 $this->typeddatamanager = $typed_data_manager; 24 } public function denormalize($data, $class, $format = NULL, array $context = array()) { 63 if (!isset($context['plugin_id']) empty($context['plugin_id'])) { 64 throw new UnexpectedValueException('Plugin id parameter must be included in context.'); 65 } $name = $class::$xero_name; 68 $plural_name = $class::$plural_name; if (count(array_filter(array_keys($data[$plural_name][$name]), 'is_string'))) { 72 $data[$plural_name][$name] = array($data[$plural_name][$name]); 73 } $list_definition = $this->typeddatamanager ->createlistdatadefinition($context['plugin_id']); 76 $items = $this->typeddatamanager->create($list_definition, $data[$plural_name] [$name]); return $items; 79 } 80

30 DefaultSettingsForm

31 24 class DefaultSettingsForm extends ConfigFormBase implements ContainerInjectionInterface { public function buildform(array $form, FormStateInterface $form_state) { // Get the configuration from ConfigFormBase::config(). 67 $config = self::config('xero.settings'); $account_options = array(); try { 72 $context = array('plugin_id' => 'xero_account'); 73 $accounts = $this->query 74 ->settype($context['plugin_id']) 75 ->setmethod('get') 76 ->setformat('xml') 77 ->execute(); foreach ($accounts as $account) { 80 // Bank accounts do not have a code, exclude them. 81 if ($account->get('code')->getvalue()) { 82 $account_options[$account->get('code')->getvalue()] = $account->get( Name') ->getvalue(); 83 } 84 } 85 } 86 catch (RequestException $e) { 87 $this->logger->error('%message: %response', array('%message' => $e->getmessage(), '%response' => $e->getresponse()->getbody(true)));

32 XeroQuery

33 102 public function construct( $client, Serializer $serializer, TypedDataManager $typed_data, LoggerChannelFactoryInterface $logger_factory) { 103 $this->client = $client; 104 $this->serializer = $serializer; 105 $this->typed_data = $typed_data; 106 $this->logger = $logger_factory->get('xero'); 107 }

34 459 public function execute() { 460 try { 461 $this->validate(); $this->explodeconditions(); $data_class = $this->type_definition['class']; 470 $endpoint = $data_class::$plural_name; 471 $context = array('plugin_id' => $this->type); if ($this->data!== NULL) { 474 $this->options['body'] = $this->serializer->serialize($this->data, $this->format, $context); 475 } $request = $this->client->{$this->method}($endpoint, $this->options['headers']); $response = $request->send(); 488 $data = $this->serializer->deserialize($response->getbody(true), $data_class, $this->format, $context); return $data; 491 }

35 * // Get an autocomplete for account. * $definition = \Drupal::service('typed_data_manager')- >createdefinition('xero_account'); * $form['account'] = $formbuilder->getelementfordefinition($definition, 'AccountID'); */ XeroFormBuilder

36 /** * The Xero form builder service provides methods to create form elements * from Xero data types defined in Drupal. * * Example usage: * // Get form elements required for a line item. * $formbuilder = XeroFormBuilder \Drupal::service('xero.form_builder'); * $form['lineitems'][] = $formbuilder->getelementfor('xero_line_item'); * * // Get an autocomplete for account. * $definition = \Drupal::service( typed_data_manager') * ->createdefinition('xero_account'); * $form['account'] = $formbuilder * ->getelementfordefinition($definition, 'AccountID'); */

37 public function getelementfor($plugin_id, $property_name = '') { $element = []; try { $definition = $this->typeddatamanager->createdatadefinition($plugin_id); '', if (is_subclass_of($definition, '\Drupal\Core\TypedData\ComplexDataDefinitionBase')) { if (empty($property_name)) { // Get all elements for the definition. $element = $this->getelementsfordefinition($definition); } else { // Get the element for the property of the definition. $element = $this->getelementfordefinition($definition, $property_name); } } elseif ($definition instanceof ListDataDefinitionInterface) { $element = $this->getelementsforlistdefinition($definition); } else { // Get the element for the definition. $element_type = $this->getelementtypefromdefinition($definition); $element = [ '#type' => $element_type, '#title' => $definition->getlabel(), '#description' => $definition->getdescription()? $definition->getdescription() : } ]; $element += $this->getadditionalproperties($element_type, $definition);

38 protected function getelementtypefromdefinition(datadefinitioninterface $definition) { $type = 'textfield'; if ($definition->getdatatype() === 'boolean') { $type = 'checkbox'; } elseif ($definition->getconstraint('choice')) { $type = 'select'; } } return $type;

39 protected function getadditionalproperties($type, DataDefinitionInterface $definition, $parent_type = '') { $properties = []; { if ($type === 'select') { // Add the Constraint options to the select element. $properties['#options'] = $definition->getconstraint('choice')['choices']; } elseif (array_key_exists('xeroguidconstraint', $definition->getconstraints())) } // Add an auto-complete path for data types with guids. $properties['#autocomplete_route_name'] = 'xero.autocomplete'; $properties['#autocomplete_route_parameters'] = ['type' => $parent_type]; if ($definition->isrequired()) { $element['#required'] = TRUE; } if ($definition->isreadonly()) { $element['#disabled'] = TRUE; } } } return $properties;

40 Typed Data Summary Data Type and Data Definition Normalizer service Inject Typed Data Manager into forms, controllers, and services.

41 Typed Data and PHPUnit Unit tests are awesome and fast, but Typed Data Manager is big and scary. :( Typed Data calls itself so mocking is not straight-forward. :( Need to mock the service container. :( What is actually necessary for decent code coverage and analysis? Composer, GitHub and Drupal are utterly broken as of rc1 on Travis CI for automated testing.

42 PHPUnit I like seeing pretty coverage and complexity graphs.

43 PHPUnit I like seeing pretty coverage and complexity graphs.

45 PHPUnit Usefulness DataDefinitionInterface::getPropertyDefinitions Easy to get coverage for this method as it s only returning an array of more data definitions. Coverage and testing here does not provide much value other than coving in my report in a pretty green color.

46 15 class PaymentDefinition extends ComplexDataDefinitionBase { public function getpropertydefinitions() { 23 if (!isset($this->propertydefinitions)) { 24 $info = &$this->propertydefinitions; $info['invoice'] = DataDefinition::create( xero_invoice') ->setrequired(true)->setlabel('invoice'); 27 $info['account'] = DataDefinition::create( xero_account') ->setrequired(true)->setlabel('account'); 29 $info['date'] = DataDefinition::create( string') ->setrequired(true)->setlabel('date'); 30 $info['amount'] = DataDefinition::create( float') ->setrequired(true)->setlabel('amount'); 31 $info['currencyrate'] = DataDefinition::create( string') ->setlabel('currency rate'); 32 $info['reference'] = DataDefinition::create( string') ->setlabel('reference'); 33 $info['isreconciled'] = DataDefinition::create( boolean') ->setlabel('is reconciled?'); 34 } 35 return $this->propertydefinitions; 36 } 37 }

47 Mocking Typed Data Manager The static create method on TypedData classes will call TypedDataManager: Mock objects cannot simply return a single known value for complex data types. Mock objects must carefully re-construct the order of how create will be called on for every definition in a complex data definition.

48 98 public function createinstance($data_type, array $configuration = array()) { 99 $data_definition = $configuration['data_definition']; 100 $type_definition = $this->getdefinition($data_type); if (!isset($type_definition)) { 103 throw new \InvalidArgumentException("Invalid data type '$data_type' has been given"); 104 } // Allow per-data definition overrides of the used classes, i.e. take over 107 // classes specified in the type definition. 108 $class = $data_definition->getclass(); if (!isset($class)) { 111 throw new PluginException(sprintf('The plugin (%s) did not specify an instance class.', $data_type)); 112 } 113 return $class::createinstance($data_definition, $configuration['name'], $configuration['parent']); 114 }

49 30 public function setup() { 31 // Typed Data Manager setup. 32 $this->typeddatamanager = $this->getmockbuilder('\drupal\core\typeddata \TypedDataManager') 33 ->disableoriginalconstructor() 34 ->getmock(); $this->typeddatamanager->expects($this->any()) 37 ->method('getdefinition') 38 ->with(static::xero_type, TRUE) 39 ->will($this->returnvalue(['id' => static::xero_type, 'definition class' => static::xero_definition_class])); 40 $this->typeddatamanager->expects($this->any()) 41 ->method('getdefaultconstraints') 42 ->willreturn([]); // Validation constraint manager setup. 45 $validation_constraint_manager = $this->getmockbuilder( \Drupal\Core \Validation\ConstraintManager') 46 ->disableoriginalconstructor() 47 ->getmock(); 48 $validation_constraint_manager->expects($this->any()) 49 ->method('create') 50 ->willreturn([]); 51 $this->typeddatamanager->expects($this->any()) 52 ->method('getvalidationconstraintmanager') 53 ->willreturn($validation_constraint_manager); 54

50 51 $this->typeddatamanager->expects($this->any()) 52 ->method('getvalidationconstraintmanager') 53 ->willreturn($validation_constraint_manager); // Mock the container. 56 $container = new ContainerBuilder(); 57 $container->set('typed_data_manager', $this->typeddatamanager); 58 \Drupal::setContainer($container); // Create data definition 61 $definition_class = static::xero_definition_class; 62 $this->datadefinition = $definition_class::create(static::xero_type); 63 } }

51 51 /** 52 * Test getphone method. 53 */ 54 public function testgetphonenumber() { 55 $string_def = DataDefinition::create('string'); 56 $country = new StringData($string_def); 57 $area = new StringData($string_def); 58 $number = new StringData($string_def); $this->typeddatamanager->expects($this->any()) 61 ->method('getpropertyinstance') 62 ->with($this->phone, $this->callback(function($subject) { 63 return in_array($subject, array( PhoneCountryCode, 'PhoneAreaCode', 'PhoneNumber')); 64 })) 65 ->will($this->onconsecutivecalls($country, $area, $number)); $this->phone->set('phonecountrycode', '01'); 68 $this->phone->set('phoneareacode', '805'); 69 $this->phone->set('phonenumber', ' '); $this->assertequals(' ', $this->phone->getphone()); 72 } 73 }

52 A poem about mocking the container What I learn not to do I resign myself to that fate. Besides, core does it too. So please do not hate this container so small is not actually in my app at all. I do not wish to load, or directly invoke the container, to get variables into my code. But to be a better maintainer, forgive that what I leverage. All I want is pretty coverage.

53 Re: Unit Tests, Mocking and Typed Data You re crazy! -webçick (June 2015)

54 Why Go Through All of That? Previous code was not testable Ever written an automated test for an external API? In a public, open source repository? Loosely-coupled and stricter typing is better. Better fit in Drupal 8: Serialization, Rules, Services Education (Score, Grade, etc )

What else..? The floor is open to ask, discuss or share: https://joind.

55 What else..? The floor is open to ask, discuss or share: Typed Data use cases Questions Other Drupal 8 APIs that relate Matthew Radcliffe

Mocking Drupal: Unit Testing in Drupal 8. Matthew Radcliffe

Mocking Drupal: Unit Testing in Drupal 8 Matthew Radcliffe mradcliffe @mattkineme Spoilers Quality Assurance PHPUnit Mocking Drupal things Quality Assurance Prevent defects from making it to the customer: