Document Sub Title Yotpo Technical Overview 07/18/2016 2015 Yotpo
Contents Introduction... 3 Yotpo Architecture... 4 Yotpo Back Office (or B2B)... 4 Yotpo On-Site Presence... 4 Technologies... 5 Real-Time Capabilities... 5 Infrastructure... 5 Network Architecture... 5 Akamai Architecture... 6 General... 6 Elastic Load Balancers... 6 RDS... 6 Caching... 7 Auto-Scaling... 7 External Services... 7 System Security... 8 Authentication... 8 API authentication... 8 Email & Password Authentication... 8 Authorization... 8 Other... 8 SSL... 8 CSRF Mitigation... 9 Backups... 10 General... 10 RDS... 10 MongoDB... 10 Source Code... 10 Backup Policy and Procedures... 10 RDS... 10 MongoDB... 10 1 1
Source Code... 10 Disaster Recovery Plan... 11 Introduction... 11 Infrastructure Disasters... 11 Types of Infrastructure Disaster... 11 Recovery Plans... 11 Regional Outage... 11 Zone Outage... 11 Application Outage... 11 Data Disasters... 11 Types of Data Disaster... 11 Recovery Plan for Data Disaster... 12 2 2
Introduction This document describes the Yotpo architecture, application security policy, and system backup policy and procedures. 3 3
Yotpo Architecture A very powerful API that does most of the heavy lifting for our major projects is at the core of Yotpo. Yotpo Back Office (or B2B) The Yotpo back office is the place for the website owner to manage the entire Yotpo solution. Here the owner can define emails, look and feel, coupons and integration to other complementary services they use. Yotpo On-Site Presence The Yotpo offering includes several types of widgets. Widgets are different ways to populate content that was created using Yotpo in the website. All of the above-mentioned projects make use of the Yotpo API. Another purpose of this API is to allow our customers to extend the core functionality of Yotpo. The following diagram illustrates this architecture: 4 4
Technologies Real-Time Capabilities Today Yotpo serves about five billion requests a month on its entire platform. To cope with such loads, Yotpo invests heavily in caching and static serving infrastructures. We have a strong cache layer implemented above Redis at the API level. Yotpo uses Akamai to serve all of its content (static and dynamic). Infrastructure The Yotpo architecture was designed and built from day one for scalability. The main parts for scale in the system is the On-Site Presence Components and the Yotpo API. Those parts of the system should have the ability to serve billions of monthly requests with an excellent response time and high SLA. Yotpo makes use of Dynamic CDN s solution that Akamai offers, which offers the following benefits: Removing stress and bottlenecks from the Yotpo API. Improve delivery performance in terms of response time in different geographical regions. Network Architecture 5 5
Akamai Architecture General Yotpo uses AWS for hosting our web servers and databases. All our servers and databases in AWS are located within Amazon Virtual Private Cloud (VPC). AWS VPC uses security groups as a firewall to control traffic at the instance level, while it also uses network ACL (access control lists) as a firewall to control traffic at the subnet level. Yotpo s servers are distributed among the different availability zones of AWS. This is done in order to ensure a high level of fault tolerance and stability. In each of the availability zones, Yotpo holds a NAT instance within public subnets to allow access to the Internet from the servers located inside the private subnets. Elastic Load Balancers Yotpo uses elastic load balancers to allow distribution of the data across all instances. Each of Yotpo s servers has an elastic load balancer. Yotpo s API has two load balancers: Internal load balancer to distribute requests from within all Yotpo s web servers. External load balancer to distribute direct requests to the API. RDS Yotpo uses AWS Relational Database Service as its main database. The database is deployed in a Multi- Availability Zone configuration. This means there is a primary database instance which synchronously replicates the data to a standby database instance, in a different availability zone. In case of an infrastructure failure, a failover to the standby instance is performed. 6 6
Caching Yotpo has two main levels of caching: Akamai - Yotpo is using Akamai as a content delivery network to cache data and deliver it to end user with minimum latency. The content that is cached is the static assets and widget responses. Redis - Yotpo uses Redis key-value database for caching of responses to reduce response time of both API and widget requests. Auto-Scaling All of Yotpo s servers have auto-scaling configurations. Using this configuration, certain system metrics such as CPU usage and traffic throughput will result in adding new instances automatically to prevent any negative affect on the performance of the system. External Services Yotpo uses MongoDB as part of its system. The database is hosted in Compose (formerly MongoHQ) hosting solutions. The deployment is of a replica set with automatic failover capabilities. Communication with Compose is done over a Virtual Private Network (VPN) 7 7
System Security Authentication Yotpo offers two methods of authentication for the system: API authentication Each account has two randomly generated keys: App Key - public key to uniquely identify the account. Secret Key - unique identifier per account - the secret key is visible only to store owner. Using the App Key and Secret Key, users can obtain an API Token which can be used to perform actions through the Yotpo API. The API Token provides access to a specific account. The API Token expires 30 days after its creation. Email & Password Authentication Each user has an email and password identifying his user within the Yotpo system. Both email and password are kept encrypted within Yotpo s database to ensure maximum security for sensitive data. Login to the Yotpo admin at my.yotpo.com is only possible using the email and password. A session will remain active until logging out or clearing session cookies. Upon sign-in, user has access to all accounts associated to them, with appropriate privileges. Authorization Different actions in the Yotpo system require different permissions. There are three user types in Yotpo: Reviewer - has access only to public information and public API calls. Reviewers cannot login to the Yotpo admin or authenticate using the Yotpo API. Account Owner - has full access to the account they own. The Account Owner can login to the Yotpo admin and authenticate using the Yotpo API. Account Moderator - has limited access to the account. Permissions are determined by the account owner. The Account Moderator cannot authenticate using the Yotpo API. Once authenticated by Yotpo using one of methods described in Authentication, a user is authorized to specific actions in relation to the privileges he has on the account. Every non-public action in the Yotpo System requires the user to have the correct set of permissions. For example, a store owner can publish/unpublish a review however only for the they own. Other SSL Yotpo supports communication over SSL protocol to ensure encrypted communication over the internet. The connection is using TLS 1.2 with 128 bit encryption. 8 8
List of requests over HTTPS: All traffic in Yotpo admin. API requests. All widget requests performed on secure pages. CSRF Mitigation Yotpo countermeasures any CSRF attacks using CSRF tokens included in all forms and AJAX requests. 9 9
Backups General Yotpo follows a strict policy to ensure backup and high security of customer data. This policy includes regular backup procedures of the different data and distribution of the data across various services. Yotpo s data resides within three major locations: RDS RDS is a Relational Database Service provided by AWS. Yotpo uses AWS RDS as a MySQL server. Yotpo uses MySQL database to save its business logic data: account data, user data, reviews, and so on. MongoDB MongoDB is a NoSQL database. Yotpo uses Compose as a hosting provider for MongoDB. Yotpo uses MongoDB mainly to store analytic data. Source Code Yotpo has multiple projects written in various languages/frameworks. Yotpo s code is a crucial part of its system, containing unique and complex algorithms. Backup Policy and Procedures RDS Daily snapshots - all MySQL Data is backed up on a daily basis. Backups are maintained in AWS s3. Snapshots are saved for a total of 15 days each. Distribution of backups between different services - increases the survivability of the data. Once a week, a snapshot of the database is backed up to Google Cloud. MongoDB Daily snapshots - all Mongo data is backed up on a daily basis. Backups are maintained in AWS s3. Snapshots are saved for a total of 15 days each. Source Code All code projects are saved in GIT source control. 10 10
Disaster Recovery Plan Introduction There are two main types of disasters that could happen in the Yotpo system: Infrastructure Disasters Data Disasters Yotpo has tools to solve each scenario. Infrastructure Disasters Types of Infrastructure Disaster Regional Outage in AWS (Amazon Web Services) Zone Outage in AWS Application Outage Recovery Plans Regional Outage Regional Outages are handled by maintaining a running auto scalable environment in multiple regions: Databases are replicated cross region with a replication lag monitoring and one-minute threshold. Hot standby application servers run in each region and are isolated to local DB replicas and cache layers. Automated DNS failover based on regional health checks. Zone Outage Zone Outages are handled by the AWS infrastructure For databases, Yotpo uses RDS in a multi availability zone mode, which utilizes multiple slaves and one master, each slave in a different zone. Once the master zone crashes, a slave is promoted and a DNS failover occurs. The application layer is spread across multiple zones and serves under a load balancer which has an instance in each zone. Application Outage Our application servers are measured by traffic and CPU utilization for the auto scaling mechanism. In an event of a crash in one application server, the auto-scaling policy spawns a new application server to compensate for the load that the crashed server created on the rest of the system and to prepare for a higher load of traffic. Data Disasters Types of Data Disaster Human error Data corruption 11 11
Recovery Plan for Data Disaster 1. A daily backup of data from the entire cluster. 2. A daily copy of the backup data to a remote and separate cluster hosted by Google. 3. In the event of corruption or a human error, restoring the DB layer that was affected can be performed if there is no other way of fixing it. 12 12