Magento Marketplace's New Extension Quality Program

Transcription

1

2 Magento Marketplace's New Extension Quality Program

3 Erika Talbott Product Manager - Marketplace J Ravi Menon Architect - Marketplace Tom Erskine Lead Engineer - MFTF

4 Overview Erika Talbott Marketplace Product

5 Extension Quality Program Quality Extensions. Rigorously Tested. "Quality is an asset not a cost or afterthought; it is insurance that the product will meet the customer's true needs with uninterrupted delivery and ample performance."

6 Manual QA Then and Now 700 Extensions 3400 Extensions 50% Failure Rate 100 Submissions /Week 48% Failure Rate 130 Submissions /Week Submissions Reviewed/ Week 2 QA Reviewers 115 Submissions Reviewed /Week 6 QA Reviewers 1+ Month Lead Time ~2 Week Lead Time

7 Other Notable Achievements Implemented two expedited test paths Minor updates to Magento core Patch version submissions Introduced Varnish Cache testing Infrastructure improvements on Repo Increased communications to the community Commerce dev licenses and Cloud sandboxes available for all

8 We heard you Top 5 Complaints Too Slow Can t find Test Reports Unclear requirements Must remove extensions from store to update Sales data is lacking New EQP (Early Q3) Parallel submission paths Consolidated test reports Improved documentation, contextual help and tooltips Extensions remain on store during updates Improved dashboard and reporting

9 Technical and Marketing in Parallel

10 Consolidated Test Reports

11 Contextual Help and Tooltips

12 Improved Dashboard

13 But wait, there s more! Version pulled from Code package Multiple Development Lines Withdraw a Submission Cancel a Submission Price changes on the fly Preview in Product Page Advanced Security Checks Test Automation Improvements Foundation for MFTF Responsive Design

14 Roadmap NOW NEXT FUTURE New EQP Support for M2.3 Quality based search rankings Authenticity certificates EQP automated functional tests Accept Developer MFTF tests with submissions Recurring billing Expanded Developer payout options Advanced Developer reports Public EQP APIs Merchant notification of new versions

15 Technical Enhancements J Ravi Menon Marketplace Architect

16 EQP Technical Enhancements Support for multiple development lines. A new extensible and scalable infrastructure for automated tests. Support for parallelization of technical (automated tests + manual QA) and marketing tracks of the EQP Pipeline. REST API support for the EQP Process.

17 EQP Technical Enhancements Support for multiple development lines for a given extension: A gap in the current Developer Portal. This will now allow for bug-fixes/patches and feature releases on any given line identified by the major version number (must follow semver best practices). This allows the developers to support merchants still running their older versions of extensions, while focusing on enhancing their current line.

18 EQP Technical Enhancements A new extensible and scalable infrastructure for automated tests: Ability to run automated tests in parallel. Ability to scale on-demand unlike in the current infrastructure, where the test jobs are queued and handled sequentially. The current set of test tools enhanced to work in this new infrastructure. A unified framework to add more new automated test tools.

19 EQP Technical Enhancements Setting up the foundation for the following features in some future release: Running extensions against upcoming Magento Releases and share reports to affected developers early on in the process. Ability to plug-in new automated test tools in a seamless manner; some examples upcoming ones in the roadmap: Support for MFTF tests Improved Security tools for static and dynamic tests. Tools around performance checks i.e. did the installation of the extension caused a significant performance degradation from baseline Magento 2 setup.

20 EQP Technical Enhancements Parallelization of the technical (automated tests + manual QA) and marketing tracks of the EQP pipeline: A consolidated report highlighting all technical and marketing errors in one-shot, and being programmatically accessible. No more back-n-forth in a sequential manner as errors are encountered in either or both of the tracks. Foundation for better tools for the marketing track to automate certain processes, and speed up the time spent here. Support for future fast-track options based on historical metrics and MFTF.

21 APIs

22 EQP APIs - Introduction The new Marketplace Developer Portal will support REST API access. The new Developer Portal website itself being built on top of these same REST APIs API First Design! Enables Programmatic access to the entire life-cycle of the extension quality process. Sets the foundation for more feature development and enhancements to the EQP process.

23 EQP APIs - Overview Send feedback on which APIs you d like to have available! marketplace-eqp-apis@magento.com REST API docs, Alpha version: Follows REST principles for all available resources: GET: To retrieve a resource POST: To submit a new resource PUT: Update an existing resource DELETE: Delete an existing resource Not all HTTP commands may be available for certain resources consult the documentation. Supports batch request and response models Error handling based on standard HTTP Response Codes with additional enhancements for batch responses.

24 EQP APIs Use Cases Integrating the submission process to CICD pipelines at the developers end. Accessing reports/metrics periodically and incorporating them to developer preferred analytics services. Foundation for a more streamlined and enhanced EQP process: With the upcoming support of MFTF, regressions can be run, and based on various metrics, the submissions could by-pass manual QA. Integrating with in-house developer tools for extension management: Example: Code in github, meta-info in Google spreadsheets, and assets in Dropbox/G- Drive managed by different stakeholders. It can be programmatically combined and integrated to the EPQ pipeline. And much more! It sets the foundation for the community to build more tooling around the EQP process: Example: Community contribution in building developer sdks in various languages to use the APIs.

25 EQP API - /rest/v1/apps Requires Client application registration for API ID and Secret, and obtaining session tokens. Enables Authentication and Authorization POST /rest/v1/apps/session/token ml

26 EQP API - /rest/v1/users Profile: GET /rest/v1/users/:mage_id PUT /rest/v1/users/:mage_id Keys Management (M1 and M2): GET /rest/v1/users/:mage_id/keys POST /rest/v1/users/:mage_id/keys PUT /rest/v1/users/:mage_id/keys/:url_encoded_label_of_m2_key DELETE /rest/v1/users/:mage_id/keys/:url_encoded_label_of_m2_key

27 EQP API - /rest/v1/users User-owned extension related reports: GET /rest/v1/users/:mage_id/reports/pageviews GET /rest/v1/users/:mage_id/reports/totals GET /rest/v1/users/:mage_id/reports/sales GET /rest/v1/users/:mage_id/reports/refunds More reports to follow in future releases. s.html

28 EQP API - /rest/v1/files Code (M1 Tarball, M2 ZIP files), PDF documentation and media assets artifact management: GET /rest/v1/files/uploads/:file_upload_id POST /rest/v1/files/uploads DELETE /rest/v1/files/uploads/:file_upload_id Supports batch uploads html

29 EQP API - /rest/v1/products/packages Package management and integration with the EQP process: GET /rest/v1/products/packages GET /rest/v1/products/packages/:submission_id GET /rest/v1/products/packages/skus GET /rest/v1/products/packages/skus/:url_encoded_sku GET /rest/v1/products/packages/items GET /rest/v1/products/packages/items/:item_id GET /rest/v1/products/packages/:submission_id/status GET /rest/v1/products/packages/sku/:url_encoded_sku/status GET /rest/v1/products/packages/item/:item_id/status POST /rest/v1/products/packages PUT /rest/v1/products/packages PUT /rest/v1/products/packages/:submission_id PUT /rest/v1/products/packages/:item_id DELETE /rest/v1/products/packages/:submission_id DELETE /rest/v1/products/packages/items/:item_id

30 EQP API - /rest/v1/products/packages Supports batch uploads and retrievals. Package submission is a 2-step process: 1. Upload Package code artifacts, PDF documents, and media assets via the /rest/v1/files resource endpoint and obtain associated file upload ids. 2. Submit package meta-info via /rest/v1/products/packages resource endpoints, linking file upload ids from step 1. Prior to submission in step 2, all malware checks on files must pass. The files can be periodically polled via the /rest/v1/files endpoint to check for malware status. Support for callback hooks instead of polling for all EQP state changes is being planned for some future release.

31 EQP API - /rest/v1/products/packages

32 EQP API - /rest/v1/reports General purpose aggregated Marketplace reports: GET /rest/v1/reports/metrics GET /rest/v1/reports/metrics/:metric_name Example metrics: sessions, overall pageviews, category-specific pageviews, EQP process-related metrics etc.. Marketplace Zeitgeist! Provides visibility to developers on where their extensions rank in various metrics compared to the overall trends in the Marketplace. html

33 MFTF Automation for Extension Development Tom Erskine MFTF Lead

34 What is MFTF MFTF is: For writing functional tests Using Selenium to drive a browser simulating user interaction A framework to structure testing A set of tools to aid rapid, reusable test creation Reporting tools and test metadata support

35 MFTF Technology PHP Composer XML Codeception (BDD-styled PHP testing framework) Allure (Reporting tool) Robo (PHP task runner)

36 Why use MFTF in extension development? Functional Testing is crucial for delivering quality MFTF is the future of functional testing in Magento As an extension developer: It allows simple reuse of existing Magento tests Will integrate with EQP Allows merging to existing tests Provides powerful reporting tools to measure and track quality metrics

37 Making new tests Tests and all test objects are written in XML Test Objects: Tests Pages Sections Data Guides

38 Modifying existing tests As well as running Magento supplied tests, you can: Edit existing tests to run locally Write additional tests for your extension Create and edit suites of tests Reuse existing ActionGroups Merge test changes reflecting your extension functionality

39 Shipping tests for extensions Tests written for your extension New tests, merge files, associated data and objects These can all be packaged or delivered with your extension This allows your customers to verify your functionality in their implementation

40 Custom Modules Pathing and Directory Structure

41 Where do I develop custom modules? Your tests should be in app/code/$vendor/$module/test/mftf Your custom module should follow the directory and naming structure of existing MFTF modules: Test Page Section Data Metadata ActionGroup

42 Merging

43 Explanation of merging mechanism

44 Types of Merging Action addition Actions can be added. They should specify ordering. Action removal Action modification This is possible however: To avoid potential problems with sequence ordering, best practice is to remove the line and replace via addition All references to Actions are done by that actions stepkey

45 MFTF Start now! Magento MSI and PageBuilder projects are actively using MFTF Ongoing development we re just about to drop v2.2.0 with expanded test suite support! Get ahead of the curve, start using MFTF now: You ll increase your own quality, fast You ll have a template for future EQP testing You ll be set up to accelerate your EQP submission from day one

46 Next Steps Get Involved: Join the #mftf slack channel for active discussion and questions Log issues in MFTF Github project ( functional-testing-framework/issues) Join the #marketplace slack channel for active discussions and questions around EQP APIs and other topics. for EQP API feedbacks: Further activities at Imagine 2018 Join us at our Marketplace booth MFTF Workshop Montrachet 01 Limited space! terskine@magento.com

47 Resources EQP Guide nsion_quality_program_overview.pdf User Guide APIs MFTF Guide Contact us at

48 Q&A