Microservices. GCPUG Tokyo Kubernetes Engine

Transcription

1 Microservices On GKE At Mercari GCPUG Tokyo Kubernetes Engine

2 @deeeet

3 Background

4 Start with Monolith

5 Small Overhead for cross domains Reusable code across domains Effective operation by SRE team

6 3 scalabilities

7 Growth of business Growth of features Growth of organization

8 Growth of business Growth of features Growth of organization

9 Growth of business Growth of features Growth of organization

10 Huge Monolith

11 Difficult to understand change effect Difficult to test Difficult to on-board Difficult to isolate failure Difficult to scale independently Difficult to try new technologies

12 Growth of business Growth of features Growth of organization

13 Unclear ownership Communication overhead

14 Velocity is stalled

15 Microservices

16 Microservices is a software development technique that structures an application as a collection of loosely coupled services with the smallest autonomous boundary.

17 Technical benefit Organization benefit

18 Technical benefit Organization benefit

19 Easy to test Easy to deploy Easy to on-board Easy to isolate failure Easy to scale independently

20 Technical benefit Organization benefit

21 Clear ownership Minimum communication overhead

22 Deliver new features faster

23 How Microservices?

24 Gateway pattern Strangler pattern

25 Gateway pattern Strangler pattern

26 Mercari API Service A Service B

27 API Gateway Mercari API Service A Service B

28 API Gateway Service X Mercari API Service A Service B

29 Multiple services on a single endpoint SSL Termination DDoS Protection Common AuthZ/AuthN API Gateway Service X Mercari API Service A Service B

30 Gateway pattern Strangler pattern

31 API Gateway Service X Mercari API Service A Service B

32 API Gateway Service X Service A Mercari API Service B

33 API Gateway Service X Service A Service B Mercari API

34 API Gateway Mercari API Service C Function X Function Y Function Z

35 API Gateway Facade C Mercari API Service C Function X Function Y Function Z

36 API Gateway Facade C Function X Mercari API Service C Function Y Function Z

37 API Gateway Facade C Function X Function Y Mercari API Service C Function Z

38 API Gateway Facade C Function X Function Y Function Z Mercari API Service C

39 API Gateway Service C Function X Function Y Mercari API Function Z

40 API Gateway Service C Function X Function Y Mercari API Service D Function Z

41 Current Status

42 API Gateway Service X Mercari API Service A Service B

43 Technical Stack

44 Authority API Gateway Service X Mercari API Service A Service B Sakura

45 Google Cloud Load balancing Authority API Gateway Service X Kubernetes Engine GCP Mercari API Service A Service B Sakura

46 Google Cloud Load balancing Authority API Gateway Service X Kubernetes Engine GCP Mercari API Service A Cloud Resources Managed Services Service B Sakura

47 Google Cloud Load balancing Authority API Gateway Service X Kubernetes Engine GCP Mercari API Service A Cloud Resources Managed Services Service B Container Sakura

48 Over HTTP Google Cloud Load balancing Authority API Gateway Service X Kubernetes Engine GCP Mercari API Service A Cloud Resources Managed Services Service B Container Sakura

49 Over HTTP SSL Termination DDoS Protection Cloud Amor? Google Cloud Load balancing Authority API Gateway Service X Kubernetes Engine GCP Mercari API Service A Cloud Resources Managed Services Service B Container Sakura

50 Over HTTP SSL Termination DDoS Protection Cloud Amor? Google Cloud Load balancing Authority API Gateway Service X Kubernetes Engine GCP Routing to microservices Protocol tranformation (HTTP to grpc) Common logging & Tracing Request buffering Mercari API Service A Service B Sakura Cloud Resources Managed Services Container

51 Common AuthZ/AuthN Over HTTP SSL Termination DDoS Protection Cloud Amor? Google Cloud Load balancing Authority API Gateway Service X Kubernetes Engine GCP Routing to microservices Protocol tranformation (HTTP to grpc) Common logging & Tracing Request buffering Mercari API Service A Service B Sakura Cloud Resources Managed Services Container

52 Common AuthZ/AuthN Over HTTP SSL Termination DDoS Protection Cloud Amor? Google Cloud Load balancing Authority API Gateway Service X Kubernetes Engine GCP Managed DB Routing to microservices Protocol tranformation (HTTP to grpc) Common logging & Tracing Request buffering Mercari API Service A Service B Sakura Cloud Resources Managed Services Container

53

54

55

56

57

58

59 Another important takeaway is that even though all of these listed items are important, ultimately the most critical thing is observability. As I like to say: observability, observability, observability - Matt Klein, Seeking SRE (Chapter6)

60 Network Service A Network Service B Logging? Tracing? (Observability) Logging? Tracing? (Observability)

61 Load balancing? Request timeout? Request retry with backoff? Circuit breaking? AuthN and AuthZ? API limit? Network Service A Network Service B Logging? Tracing? (Observability) Logging? Tracing? (Observability)

62 Different protocols.. Load balancing? Request timeout? Request retry with backoff? Circuit breaking? AuthN and AuthZ? API limit? Network Service A Network Service B Logging? Tracing? (Observability) Logging? Tracing? (Observability)

63 Service C Service A Service B Service D

64 Service C Se Service A Service B Se Service D Se

65

66

67

68 How we use GCP?

69 Google Cloud Load balancing Authority API Gateway Service X Kubernetes Engine GCP

70 Google Cloud Load balancing Authority API Gateway Service X Kubernetes Engine GCP How we use GKE?

71 Cluster strategy GCP project strategy Node pool strategy Namespace strategy

72 Cluster strategy GCP project strategy Node pool strategy Namespace strategy

73 europe-west1 us-west1 asia-northeast1 Each region has its own Cluster

74 All services in 1 cluster No special cluster for specific service Production Cluster Development Cluster Testing/QA will be done in development cluster

75 Production Cluster In future, 1 region 1 cluster like Google Borg

76 Cluster strategy GCP project strategy Node pool strategy Namespace strategy

77 Only SRE can access cluster nodes IAM: SRE GCP project: GKE Production IAM: SRE + α GCP project: GKE Development Production Cluster Development Cluster 1 cluster for 1 GCP project

78 Cluster strategy GCP project strategy Node pool strategy Namespace strategy

79 GCP project: GKE Production Normal applications n1-standard-16 node pool Auto scaling Enabled Automatic node repair Enabled Preemptible Enabled (only in US) Machine learning workloads n1-highmem-16 node pool Production Cluster

80 Cluster strategy GCP project strategy Node pool strategy Namespace strategy

81 GCP project: GKE Production Each team can only access its own kubernetes namespace Namespace: Service A RBAC: Team X Each services has its own kubernetes namespace Pod: A Pod: A Pod: A Namespace: Service B RBAC: Team X Pod: B Pod: B Production Cluster

82 Google Cloud Load balancing Authority API Gateway Service X Kubernetes Engine GCP How we use GCP services?

83 How access limit GCP services? Each service should be allowed to access only its own GCP resources

84

85 GCP project: GKE Production IAM: SRE Namespace: Service A RBAC: Team X Pod: A Pod: A Pod: A Namespace: Service B RBAC: Team Y Pod: B Pod: B Production Cluster

86 GCP project: GKE Production IAM: SRE GCP project: Service A Namespace: Service A RBAC: Team X IAM: Team X + SRE Pod: A Pod: A Pod: A Namespace: Service B RBAC: Team Y GCP project: Service B Pod: B Pod: B IAM: Team Y + SRE Production Cluster Each services has its own GCP project

87 GCP project: GKE Production IAM: SRE GCP project: Service A Namespace: Service A RBAC: Team X Cloud SQL IAM: Team X + SRE Pod: A Pod: A Pod: A Namespace: Service B RBAC: Team Y GCP project: Service B Service resources in its own GCP project Pod: B Pod: B Spanner IAM: Team Y + SRE Production Cluster Each services has its own GCP project

88 GCP project: GKE Production IAM: SRE GCP project: Service A Namespace: Service A RBAC: Team X Cloud SQL IAM: Team X + SRE Pod: A Pod: A Pod: A Namespace: Service B RBAC: Team Y GCP project: Service B Service resources in its own GCP project Pod: B Pod: B Spanner IAM: Team Y + SRE Production Cluster Each services has its own GCP project Each namespace has its own service account for its own GCP project

89 Each namespace has its own service account

90 GCP project: GKE Production IAM: SRE GCP project: Service A Namespace: Service A RBAC: Team X Cloud SQL IAM: Team X + SRE Pod: A Pod: A Pod: A Namespace: Service B RBAC: Team Y GCP project: Service B Service resources in its own GCP project Pod: B Pod: B Spanner IAM: Team Y + SRE Production Cluster Each services has its own GCP project Each namespace has its own service account for its own GCP project

91 GCP project: GKE Production IAM: SRE GCP project: Service A Namespace: Service A RBAC: Team X Cloud SQL IAM: Team X + SRE Pod: A Pod: A Pod: A Namespace: Service B RBAC: Team Y GCP project: Service B Pod: B Pod: B Spanner IAM: Team Y + SRE Production Cluster GCP project creation? Setup Spanner or Cloud SQL..?

92 Infrastructure as Code

93

94 CloudSQL instance creation

95 Spanner instance creation

96 mercari / microservices-terraform Private

97 Just create a PR to create new GCP project

98 Terraform plan on CI

99 Terraform apply on CI

100 Terraform apply on CI Tool for notifying terraform result is open sourced

101 Common part (GCP project creation, Pagerduty setup) can be bootstrapped

102 GCP project: GKE Production IAM: SRE GCP project: Service A Namespace: Service A RBAC: Team X Cloud SQL IAM: Team X + SRE Pod: A Pod: A Pod: A Namespace: Service B RBAC: Team Y GCP project: Service B Pod: B Pod: B Spanner IAM: Team Y + SRE Production Cluster Stackdriver

103 GCP project: GKE Production IAM: SRE GCP project: Service A Namespace: Service A RBAC: Team X Cloud SQL IAM: Team X + SRE Pod: A Pod: A Pod: A Namespace: Service B RBAC: Team Y GCP project: Service B Pod: B Pod: B Spanner IAM: Team Y + SRE Production Cluster Stackdriver Logging?

104 How access limit stackdriver logging? Each team should be allowed to access only its service log

105

106 GCP project: GKE Production IAM: SRE GCP project: Service A Namespace: Service A RBAC: Team X Cloud SQL IAM: Team X + SRE Pod: A Pod: A Pod: A Namespace: Service B RBAC: Team Y GCP project: Service B Pod: B Pod: B Spanner IAM: Team Y + SRE Production Cluster Stackdriver Logging?

107 GCP project: GKE Production IAM: SRE GCP project: Service A Namespace: Service A RBAC: Team X Cloud SQL IAM: Team X + SRE Pod: A Pod: A Pod: A Big Query Namespace: Service B RBAC: Team Y GCP project: Service B Pod: B Pod: B Spanner IAM: Team Y + SRE Production Cluster Big Query Stackdriver Create BQ for each services

108 GCP project: GKE Production IAM: SRE GCP project: Service A Namespace: Service A RBAC: Team X Cloud SQL IAM: Team X + SRE Pod: A Pod: A Pod: A Big Query Namespace: Service B RBAC: Team Y sink GCP project: Service B Pod: B Pod: B Spanner IAM: Team Y + SRE Production Cluster sink Big Query Stackdriver Create BQ for each services Create BQ sink for each services

109 BigQuery sink creation

110

111 GCP and k8s Ecosystem

112 with Cloud DNS Just create ingress it automatically creates DNS records

113 with Cloud Storage Disaster Recovering Take backups of your cluster and restore in case of loss.

114 Non GCP?

115 vs. Container Builder Notification or Integration with GitHub

116 vs. Stackdriver monitoring Integration with external services like CDN or AWS

117 vs. Stackdriver error report Notification and Integration with GitHub

118 vs.?? GCP does not have chaos as a service

119 Conclusion

120 Mercari

121 @deeeet