TIBCO StreamBase 10 Distributed Computing and High Availability. November 2017

Transcription

1 TIBCO StreamBase 10 Distributed Computing and High Availability November 2017

2 Distributed Computing

3 Distributed Computing location transparent objects and method invocation allowing transparent horizontal scaling

4 Distributed Applications Distributed Query Table Queries Local, sub-set of nodes, cluster-wide Cross-node dispatching Configurable routing polices Active & Replica node delta streams

5 High Availability Partitioning and Replication

6 High Availability topics Conceptual Model Partitions Replication Dynamic Partitioning Partition Migration Geographic Redundancy

7 High Availability Services Transactional replication across one or more nodes Complete application transparency Dynamic partition definition Dynamic cluster membership Dynamic object to partition mapping Geographic redundancy Multi-master detection with avoidance and reconciliation

8 High Availability Conceptual Model

9 High Availability Concepts - 1 An availability zone belongs to a single cluster, has one or more nodes, and one or more optional quorum policies. A cluster can have zero or more availability zones. A data distribution policy is associated with one or more availability zones and a data distribution policy binding. A data distribution policy binding associates a data distribution policy, an application fragment, and a data mapper. A data mapper is associated with one or more data distribution policy bindings.

10 High Availability Concepts - 2 An application fragment is associated with one or more data distribution policy bindings. A dynamic data distribution policy is associated with one or more dynamic partitions. A dynamic partition is associated with one dynamic data distribution policy. A node can belong to zero or more availability zones.

11 High Availability Concepts -3 A quorum policy is associated with one availability zone. A partition is associated with one data distribution policy. A static data distribution policy is associated with one or more static partitions. A static partition is associated with one static data distribution policy.

12 Data distribution policies Defines how application data is distributed, or partitioned, across multiple nodes Partitioning is the basis for horizontal scaling of data sets Contain How partitions are distributed across machines within an availability zone Replication requirements Partitioning algorithm May be static or dynamic

13 Dynamic distribution policies Automatically creates dynamic partitions based on the available nodes Data is redistributed across dynamic partitions when a node joins or leaves a cluster Triggered by reloading the configuration on all nodes in the cluster

14 Example: Round-robin dynamic data distribution

15 Single Data Center Availability Zone

16 Disaster Recovery Availability Zone West Coast - availability zone within the West Coast data center. East Coast - availability zone within the East Coast data center. Disaster Recovery - availability zone that spans the west and east coast data centers to provide data redundancy across data centers.

17 Partitions, Nodes, and Replication Unit of data distribution within a data distribution policy Identified by a name Must be globally unique Nodes are listed in priority order in a node list Highest priority node is active All other nodes are replicas Replicas may be synchronous or asynchronous When active node is unavailable, next highest priority node becomes active

18 Partition Definition Example

19 Partitioned Objects and Data Mappers Each partitioned objects has a data mapper Inherited by object type, may be overridden Data mapper assigns an object to a single partition During object creation During partition mapping updates Data mappers may consider Object contents System resource utilization (such as CPU and memory) Load balancing App-specific criteria Built-in data mappers Round-robin Consistent hashing

20 Distributed consistent hashing Distributes partitioned objects evenly across all nodes in a cluster Redistributes evenly as nodes enter or leave cluster Same data values always map to same partition # of partitions >> # of nodes in practice

21 Partitioning Lab

22 Partitioning Lab Run a 3 node cluster with a Query Table in transactional memory Partition the query table across 3 nodes Replicate query table data Follow LabAndExercises/PartitioningLab.docx Lab project to import: partitioning-10.2.zip

23 elastic.conf

24 app.conf