DDS TUTORIAL. For OMG RTE workshop July 2007 V3. Hans van t Hag, DDS product Manager

Transcription

1 DDS TUTORIAL For OMG RTE workshop July 2007 V3 Hans van t Hag, DDS product Manager hans.vanthag@prismtech.com

2 Agenda 1 Data-centric Foundation Into the mood Net-centric Future: the data is the network The information-centric approach Driving the Standard Naval Combat Systems (CMS) example The OMG DDS specification DDS By Example DDS profiles Corba Integration << 10 min. break >> Concept Demo DDS Chatroom concept demo

3 Into the mood (1): huh?... 2

4 Into the mood (2): The Problem 3

5 Into the mood (3): A scalable solution?... 4

6 5 C H J G K F B E I A L N U T V R M P Q S Z Y X W D A B C E F G H I J K L M N O P Q R S T U V W X Y Z D INFORMATION BACKBONE Into the mood (4): Client/server vs. Pub-sub: A mind-shift

7 Net-centric Future: The Data is the Network

8 ARMS (Doug Schmidt): R&D Challenge 7

9 ARMS (Doug Schmidt): R&D Challenge Develop DRE Systems Technologies to Protect Critical Infrastructure 8 The soft underbelly of commercial, military, & infrastructure DRE systems depend increasingly on information technology, making attacks both attractive & lethally effective Clusters Global Information Grids Information Appliances Clients Servers MEMS BioMonitoring R&D Challenges: Create efficient, scalable, reliable, secure, & predictable DRE system technologies from nano- to tera-scale

10 ARMS (Doug Schmidt): R&D Challenge: Current challenges and limitations = Scalability! 9 Emerging Trends Large-scale DRE system requirements are increasingly more Dynamic Diverse Demanding Total Ship Computing Environments: Global dynamic resource management (1,000 nodes) Distributed Network of embedded sensor motes for environment monitoring, tracking, & surveillance (10,000 nodes) ,000,000 node fusion of physical & information systems Distributed Active Control: Vibration Damping on Delta-4 Rocket Payload Fairing (1,000 nodes) Gossamer Space Antenna (1,000,000 nodes) Noiseless sonar on submarines to provide camouflage (3,000 nodes)

11 Net-centricity: The Data Is The Network - I 10 DoD Chief Information Officer, Network and Information Integration, Nov 05

12 Net-centricity: The Data Is The Network - II 11 DoD Chief Information Officer, Network and Information Integration, Nov 05

13 Net-centricity: The Data Is The Network - III 12 DoD Chief Information Officer, Network and Information Integration, Nov 05

14 The Information-centric Approach

15 TRENDS ( SUN) 14 Information - Backbone (DDS) Corba Web Java Information Grids

16 TRENDS ( SUN) 15 Dr. Dr. Richard Soley (OMG Chairman & CEO): The The DCPS DCPS publish/subscribe publish/subscribe model model stands stands as as a a natural natural complement complement to to the the object-centric object-centric centric client/server client/server model model provided provided by by CORBA CORBA (Consumer (Consumer Electronics, Electronics, September September 13, 13, 2004) 2004) Doc Doc Allen (OMG Co-Chair Chair & Mitre): Information - DDS DDS clearly clearly has has the the potential potential to to become become THE THE Backbone dominant dominant (real-time) (real-time) standard standard in in the the Net-centric Net-centric environment environment (OMG-RTESS plenary (OMG-RTESS meeting plenary meeting June 04) June 04) Dave Sharp (Boeing FCS FCS chief-architect): The The (Army) (Army) SOSCOE SOSCOE environment environment will will be be based based on on MDA/UML MDA/UML and and OMG-DDS OMG-DDS (OMG (OMG sponsor-presentation, presentation, sponsor-presentation, June 04) June 04) Corba Web Java Information Grids

17 BACKGROUND & TRENDS 16 The Idea: Reduced complexity pub/sub already patented in 1987 information backbone Right info, Right place, Right time UNCLASSIFIED UNCLASSIFIED THALES NEDERLAND B.V. AND/OR ITS SUPPLIERS Subject to restrictive legend on title page THALES NEDERLAND B.V. AND/OR ITS SUPPLIERS Subject to restrictive legend on title page L L H H I I K K Z Z G G C C J J F F Y Y SPLICE SPLICE Information Information Backbone Backbone S S R R A B A B Q D Q D P E P E V N V M M N W W T T U U X X UNCLASSIFIED UNCLASSIFIED TRENDS: SUN TRENDS: SUN A B C D E F G H I J K L M A B C D E F G H I J K L M S P L I C E S P L I C E N O P Q R S T U V W X Y Z N O P Q R S T U V W X Y Z Defence Defence 5 5 Defence Defence Corba Corba Information - Backbone Information - Backbone Web Java Web Java Information Information Grids Grids Towards an information-centric world Loosely coupled components Dynamic systems Traditional architectures don t suffice

18 ARCHITECTURE TRENDS: DDS, A Major Specification in Line with the Network Centric Warfare Paradigm 17 UNCLASSIFIED UNCLASSIFIED Key Trend Problems DoD Key system Trend Existing Problems architectures - brittle & configured statically DoD system Existing COTS architectures - too - brittle big, slow, & configured buggy, incapable, statically & inflexible requirements are Existing multiple COTS technology bases - proprietary - too big, slow, & limit buggy, effectiveness incapable, by & impeding inflexible increasingly requirements more are Existing multiple technology bases - proprietary Assurability & limit (of effectiveness QoS), by impeding dynamic, increasingly diverse, more Adaptability Assurability & (of QoS), & dynamic, demanding diverse, Affordability Adaptability & & demanding Affordability Today, each system Today, brings each its own: system brings networks its own: computers networks displays computers displays Applications Applications software Applications Applications software Sensor Command Engagement Weapon Weapon people Sensor Command Engagement Weapon Weapon people Systems & Control System Control Systems Systems & Control System Control Systems Consequences System Systems Consequences System Systems Hard to meet required Kill EO Sched Illum Eval Hard to meet required Kill performance EO Sched Illum levels Network Eval performance levels AAW TBM Network Hard to control AAW AAW TMB distributed EG EG AAW AAW MG MG AAW TBM AAW AAW TMB Hard to control distributed EG EG AAW AAW MG MG resources Technology base: Technology base: Technology base: Technology base: Technology base: resources Proprietary Technology MWbase: Technology DII-COE base: Proprietary Technology MWbase: Proprietary Technology MWbase: Proprietary Technology MWbase: High software lifecycle Proprietary Mercury MW POSIX DII-COE Proprietary POSIX MW Proprietary VxWorks MW Proprietary POSIX MW costs High software lifecycle Link16/11/4 Mercury ATM/Ethernet POSIX NTDS POSIX FDDI/LANS VxWorks VME/1553 POSIX Link16/11/4 ATM/Ethernet NTDS FDDI/LANS VME/1553 costs e.g., many accidental Operating Operating complexities e.g., many accidental & low-level System Operating Operating System platform complexities & low-level dependencies System System platform dependencies Defence Defence DDS mandated for US - OACE DDS is key for success in NAVY OA Evaluated since 2004 NDDS/SPLICE provide good performance and scalability as a publish-subscribe subscribe middleware for combat system applications (NSWC-DD, OA Technical assessment 07/2004) DARPA ARMS study DARPA ARMS study UNCLASSIFIED UNCLASSIFIED Open Architecture Computing Environment (OACE) Open Architecture Computing Environment (OACE) Defence Defence Application Computer Application Programs Computer (Domain Programs Unique) (Domain Unique) Middleware (Standards Middleware Based) (Standards Based) COTS Computing COTS Technology Computing (Standards Technology Based) (Standards Based) App App App App App App App App Distribution Distribution Middleware Middleware Common Services Common Services Adaptation Frameworks Adaptation Frameworks Middleware Middleware Real-Time Operating System Real-Time Operating System Computing Equipment Computing Equipment Cable Plant Cabinets Switches Drivers Processors Cable Plant Cabinets Switches Drivers Processors Middleware isolates applications from Middleware computing isolates technology applications change from computing technology change R es R es o uo rc u rc e e M an M an a ga e g m e em ne t n t The finalization and availability of the DDS specification really is a tremendous achievement that addresses a significant need in both government and civilian sectors (Dr. Richard Soley OMG Chairman, Consumer Electronics, September 13, 2004) Recognized potential DARPA recognizes DDS importance Dynamic Resource Management potential

19 Distribution Middleware Requirements for OACE 18 Of the major distributed technologies, CORBA and DDS offer the widest range of competitive vendor alternatives for a multi language, multi platform, real-time processing environment Support for Real-time QoS is essential

20 DDS APPLICABILITY (from US OACE documentation) 19

21 Traditional Architectures: fault propagation 20 L G S W R H C A B T I J D Q U F E P Z K Y M N V X

22 Traditional Architectures: fault propagation 21 L G S W R H C A B T I J Q U F E P Z K Y M N V X

23 Traditional Architectures: fault propagation 22 L G S W H T I J Q P Z K Y M N X

24 Traditional Architectures: fault propagation 23 L G H Z Y N X

25 DDS: AN INFORMATION-CENTRIC ARCHITECTURE 24 Information-model D A B C D E F G H I J K L L Information Backbone M N O P Q R S T Y Y Autonomous components Interacting only with the information-bus Spontaneous: Z, Self-healing: D Redundant & Replicated: L, Y QOS-driven Data Distribution Service (reliability, urgency, importance): Z DDS

26 INFORMATION BACKBONE: Under the hood 25 Application-1 Application-2 Application-3 Producer Consumer Producer Consumer Local Database Local Database Agent Agent Agent PUBLISH / SUBSCRIBE DATA NETWORK

27 Pub/Sub: Keep it simple stupid

28 Distributed Real-time Systems Engineering 27 Functionality Real-time Complexity Scalability Evolvability Fault-tolerance Many different types of requirements

29 OMG-DDS fundamentals 28 Design principles: Minimize dependencies between components Share stable properties Focus on: Autonomous component behavior Common information model Middleware delivers: The right information at the right place at the right time

30 Architectural Requirements: Lifecycle focus 29 System design provide a stable basis to operate upon by applications enhance component autonomy allow transparent and global QoS assurance System development reduce complexity and enhance re-usability provide shared/guaranteed properties small learning effort and flat learning curve System integration support effortless component integration provide easy monitor & control shift ratio between design and integration effort System deployment guaranty QoS for reliability, latency and persistency allow runtime migration of applications allow applications to join the system at any time System maintenance & evolution System maintenance & evolution allow runtime replacement and evolutionary upgrading support for logging & replay of information provide future-proof, re-usable, robust and scalable system

31 DDS, a distributed real-time software architecture 30 Decoupling in space and time Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

32 DDS, a distributed real-time software architecture 31 Decoupling in space and time Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

33 DDS, a distributed real-time software architecture 32 Decoupling in space and time Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

34 DDS, a distributed real-time software architecture 33 Data is dynamically forwarded to all subscribed components Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

35 Data Definition 34 Applications subscribe to TOPICS Each TOPIC has an associated name and data type: Data (type) definition in IDL key fields for unique identification more recent instances overwrite existing instances with same key value (keeping into account the history-depth QoS setting of a subscriber) Example (IDL types): Struct PointTrack { long source; // key long trackid; // key Struct TrackState { long source; // key long trackid; // key } Position pos; } long environment; long identity;

36 DDS, a distributed real-time software architecture 35 Data is dynamically forwarded to all subscribed processes Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

37 Realization 36 application application application cache cache cache real-time information broker real-time information broker real-time information broker Networking (UDP/IP)

38 DDS, a distributed real-time software architecture 37 Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

39 DDS, a distributed real-time software architecture 38 Content-based subscription through relational views Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

40 Relational Object Models 39 Track trackid source PointTrack position TrackState environment identity

41 Content-based Subscription 40 Data filtering in DDS, e.g. using SQL: select * from TrackPosition where position.range < Aggregation and projection in DDS, e.g. using SQL: select position, environment from PointTrack NATURAL JOIN TrackState where position.range < and identity = hostile

42 Typical Process Behavior 41 Register with DDS Declare publications Declare subscriptions Define queries (read_conditions) Set query parameters Select input data Wait for input data Process data Publish output data

43 DDS Publication 42 Topic Data Sample Domain Participant Data Writer Publisher S User Application: Creates related DDS entities Publisher Topic DataWriter Configures entities' QoS then Provides data to DataWriter

44 DDS Subscription with Listeners 43 Listener: read, take User Application: Creates related DDS entities Subscriber Topic DataReader Configures entities' QoS and attach listeners Receives Data from DataReader through attached listeners S Topic Data Reader Subscriber Domain Participant Listener DATA_AVAILABLE Listener DATA_ON_READERS

45 DDS Subscription with Wait-Set 44 User Application: Creates related DDS entities Subscriber Topic DataReader Configures entities' QoS Creates a Condition and attaches it to a WaitSet Waits on the WaitSet until data arrive, then picks it on the DataReader Topic Data Reader Subscriber Domain Participant Wait for Data S Read/take

46 DDS, a distributed real-time software architecture 45 Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

47 DDS, a distributed real-time software architecture 46 Persistent data remains available for later access Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

48 DDS, a distributed real-time software architecture 47 Persistent data remains available for later access Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

49 DDS, a distributed real-time software architecture 48 Persistent data remains available for later access Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

50 Data Persistency 49 Volatile data: no copies outside process space typically measurement related data Transient data: copies are kept on more than one host outlives process typically state related data Persistent data: same as transient data, but additionally stored on disk outlives system typically configuration data

51 Self-Healing: Fault-tolerant persistence 50 application application application cache cache cache store Agent store Agent store Agent Networking

52 DDS, a distributed real-time software architecture 51 Self forming architecture Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-)configuration quality of service fault-tolerance information partitioning

53 DDS, a distributed real-time software architecture 52 Self forming architecture Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

54 DDS, a distributed real-time software architecture 53 QoS is attributed to data (statically or dynamically) - reliability - priority - latency Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

55 Data Delivery 54 Reliability: Best effort No Guaranteed delivery i.e. no retransmissions when data gets lost Typical for periodic measurement related data Reliable Guaranteed delivery by automatic re-transmissions of lost data Typical for non-periodic and important published information Latency: Latency Budget Specifies how fast data should be delivered i.e. its urgency Allows middleware to balance between high-volume & low-latency Priority: Transport_Priority Specifies the priority of the published information i.e. its importance Allows middleware to pre-empt low-priority data with high-priority data

56 Overview: DDS QoS policies 55 QosPolicy name : string DurabilityQosPolicy kind : DurabilityQosKind LatencyBudgetQosPolicy duration : Duration_t PartitionQosPolicy name [*] : string OwnershipQosPolicy kind : OwnershipQosKind DeadlineQosPolicy period : Duration_t ReliabilityQosPolicy kind : ReliabilityQosKind OwnershipStrengthQosPolicy value : long TimeBasedFilterQosPolicy minimum_separation : Duration_t DestinationOrderQosPolicy kind : DestinationOrderQosKind LivelinessQosPolicy lease_duration : Duration_t kind : LivelinessQosKind UserDataQosPolicy data [*] : char HistoryQosPolicy kind : HistoryQosKind depth : long PresentationQosPolicy access_scope : PresentationQosAccessScopeKind coherent_access : boolean ordered_access : boolean ResourceLimitsQosPolicy max_samples : long max_instances : long max_samples_per_instance : long

57 QoS Policies Conceptual View 56 PresentationQoS QueryCondition ReadCondition Data Reader X Data Reader Y OwnerShipQoS DestinationOrderQoS DataReader Caches HistoryQoS ResourceLimitsQoS DurabilityQoS Resolve updates TimeBasedFilterQoS PartitionQoS ContentBasedFilter Storage Area Receive Computer

58 QoS Contract Request / Offered 57 Topic QoS:Durability QoS:Presentation QoS:Deadline QoS:Latency_Budget QoS:Ownership QoS:Liveliness QoS:Reliability QoS Request / Offered: Ensure that compatible QoS parameters are set. Topic QoS not compatible Data Writer Offered QoS Data Reader Requested QoS Publisher Subscriber X Communication not established not established if offered QoS < requested QoS

59 DDS, a distributed real-time software architecture 58 Self-healing software architecture Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

60 DDS, a distributed real-time software architecture 59 Passive process replication Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

61 DDS, a distributed real-time software architecture 60 Passive process replication Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

62 DDS, a distributed real-time software architecture 61 Semi-active process replication Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

63 DDS, a distributed real-time software architecture 62 Semi-active process replication Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

64 DDS, a distributed real-time software architecture 63 Semi-active process replication Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

65 DDS, a distributed real-time software architecture 64 Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

66 DDS, a distributed real-time software architecture 65 LAN WAN Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

67 DDS, a distributed real-time software architecture 66 simulation data operational data Built-in capabilities: P/S data distribution relational data access data persistence dynamic (re-) configuration quality of service fault-tolerance information partitioning

68 Summary: The Problem 67 Problem: engineering (-cost) of distributed systems too complex not reactive not future-proof not fault tolerant security distribution extensibility functionality timeliness availability Because multi-dimensional engineering is needed: What about the current state-of-the-art? architectures: client/server, message-passing, DBMS most efforts fall short in a number of dimensions: typically: limited RT performance (high-volume & low-latency balance) exploding complexity (dependencies in many dimensions) costly evolution (impact of changes & extensions)

69 Summary: An Information-Centric Approach 68 Towards a solution: make development effort more simple develop less develop solutions only once How: minimize component dependencies maximize component autonomy normalize component interactions ( simple ) ( re-use ) ( only once ) The clue: share the stable properties, localize the unstable ones information is what matters most, not how it is processed properly modeled data is stable, processing often is not so focus on data first and then on the processing of it

70 Summary: the DDS concept 69 Data-centric architecture: autonomous components with minimal dependencies separation of function and interaction architecture that focuses on data DDS realization: middleware that offers a normalized environment Z L I H K G C J F Y A M S B D P E N R Q V W T U X designed once, guarantees system properties delivers the right data at the right place at the right time What about the processing? standard operating platform (HW, OS) standard communication facilities standard programming languages standard development tools A B C D E F G H I J K L M D D S N O P Q R S T U V W X Y Z

71 SUMMARY: Requirements & Realization 70 System design Requirement: provide a stable basis to operate upon by applications enhance component autonomy allow transparent and global QoS assurance System development reduce complexity and enhance re-usability provide shared/guaranteed properties small learning effort and flat learning curve System integration support effortless component integration provide easy monitor & control shift ratio between design and integration effort System deployment guaranty QoS for reliability, latency and persistency allow runtime migration of applications allow applications to join the system at any time Realized by: - shared Information Model - state-based information-centric system - Information classification (QoS topic-defaults) - minimized component dependencies - standardized (DDS-) interaction-environment - intuitive concept, simple/powerful features - maximized component autonomy - globally accessible information (data+metadata) - focus on info-model & decoupled applications - realtime DDS information backbone - global & FT availability of transient state data - dynamic discovery and data persistence System maintenance & evolution allow runtime replacement and evolutionary upgrading - de-coupled & autonomous components support for logging & replay of information - global availability of all (time-stamped) data provide future-proof, re-usable, robust and scalable system - highly adaptive associative data-model

72 .. 15 minutes break..

73 Combat Systems Example A requirements-driving domain

74 DDS usage in Naval Combat Systems 73 CHARACTERISTICS CHARACTERISTICS Many Many different different customers: customers: Many Many different different ships/missions: ships/missions: Large-scale Large-scale & & mission-critical: mission-critical: Real-time Real-time and and Fault-tolerant: Fault-tolerant: >15 >15 Navies Navies world-wide world-wide use use DDS DDS pub/sub pub/sub >> Ships Ships classes classes (from (from FPB s FPB s up up to to Destroyers) Destroyers) >150 >150 CPU s, CPU s, >2200 >2200 applications, applications, >4.000 >4.000 tracks/sec tracks/sec Battle-damage Battle-damage resistant, resistant, deterministic, deterministic, reliable reliable

75 Example: Frigate-size environment 74 SMART-L APAR KH/SCOUT NAV. RADAR HELI APPR. RADAR VESTA APECS-3 ESM / ECM OFF LINE R/S SIRIUS TDS (2x) POS SYSTEM SHIP S REF. IRC IFF LINK 11 VESTA SRBOC CATV BUS TDLPP VEX/ RCP BTS OSD BTS OSD ATM NETWORK VIDEO INTERFACE CABINET LARGE SCREEN DISPLAY 1 LARGE SCREEN DISPLAY 2 VCR 1/2 SPARE FL. DECK TV CAMERA COMBAT INFORMATION CENTER STAFF ROOM BRIDGE COMMS ROOM TV CAMERA DEPARTMENT OFF. / OFF. QUARTERS ETC. WORKSTATION 01 WORKSTATION M COTS & SIGMA TYPE PROCESSOR PIOD / PTSU PRINTERS TORPEDO WEAPON SYSTEM MK32 MOD-9 (MK46) GIC- 127 MIC- MK41 MIC- MK41 INTEGRATED MACHINERY CONTROL SYSTEM CCC1 CCC2 COMMS NETWORK ATAS SPHERION + XBT / XSV + SURF. TEMP. REC. HARPOON GUN 127 MM 30 MM GUNS OFF LINE MK41 SM2 LAUNCHER GOALKEEPER Data-traffic: Programs: Data flows: >4.000 publications per second over the system-data bus programs allocated over 150 processors urgent & non-urgent data (latency), important & less-important data (priority)

76 DDS: Self-forming 3-tier data-planes 75 Near-Realtime HCI Realtime Command & Control Hard-realtime realtime sensor/weapon IO Dialogue & presentation Network tracks Data- sources Business- logic System tracks Fully- Distributed processing Missile uplinks actuators Display data consoles Primitive tracks Engagements sensors Reference data (time / platform-data)

77 the OMG DDS Specificiation

78 Object Management Group The DDS Specification Data Distribution Service for Real-Time Systems Adopted in June 2003, Finalized in April 2004 Joint submission Specifies the API required for a Data-Centric Publish- Subscribe communication environment for real-time distributed systems Dr. Richard Soley, OMG Chairman and CEO: The DCPS publish/subscribe model stands as a natural complement to the object-centric client-server model provided by CORBA. The finalization and availability of the DDS specification really is a tremendous achievement that addresses a significant need in both government and civilian sectors. -- in Consumer Electronics, September 13, /2/2007 Copyright Object Management Group 77

79 Object Management Group DDS Structure DCPS: Data Centric Publish/Subscribe Purpose: QoS-Driven Distributed Data Management DLRL: Data Local Reconstruction Layer (optional) Purpose: An OO Model to access data as local objects Related Specification: DDS-RTPS Purpose: provide (network-)interoperability between multiple DDS vendors Application DCPS DLRL DDS-RTPS Protocol 7/2/2007 Copyright Object Management Group 78

80 Object Management Group DLRL Object-Model Persistence Content- Subscription Ownership Minimum- Profile DLRL DCPS DCPS DCPS DCPS DDS Features Object Oriented information view Local object model extending the distributed DCPS data model Manages relationships and supports native language constructs Distributed QoS-driven information management Fault tolerance and global persistence of selected data Guaranteed data availability supports application fault-tolerance Content-aware filtering and dynamic queries: reduces application complexity improves system performance Real-time publish/subscribe messaging: Asynchronous one-to-many real-time data communication Dynamic data flow based on current interest (pub/sub) Platform independent data model (IDL) Strongly typed interfaces for multiple languages Information Ownership management for replicated publishers 7/2/2007 Copyright Object Management Group 79

81 Contributions to the OMG-DDS specification 80 DDS drivers DDS Compliance-profiles profiles OMG/CORBA IDL for data-definition Object orientation SPLICE Content awareness Information Management NDDS/SPLICE pub/sub messaging real-time networking Persistence Object- Model Content- Subscription Ownership Minimum- Profile DLRL DCPS DCPS DCPS DCPS DLRL Module Persistence Module Cont. Sub. Module Core Modules

82 Overall DCPS Model 81 QosPolicy qos * DCPSEntity * 0..1 listener <<interface>> Listener 1 WaitSet 1 * Condition 0..1 statuscondition StatusCondition DomainEntity * 1 DomainParticipant <<summary>> A DomainParticipant is the entry-point for the service and isolates a set on applications that share a physical network. Publisher DataWriter Topic 1 * 1 * * 1 <<interface>> DataType <<implicit>> * DataReader * Subscriber 1 Data

83 Overall DCPS Model 82 DCPS Entity Publisher DataWriter Description Responsible for data distribution taking into account the applicable QoS-policies Holds the data and enables modifications Subscriber DataReader Responsible for receiving data taking into account the applicable QoS policies Holds the data and provides access to the data Topic Content filtered topic MultiTopic Associates a name with a data type Expresses interest in content-filtered data Expresses interest in aggregated (& filtered) data

84 DCPS Architecture 83 QosPolicy qos * DCPSEntity <<interface>> * 0..1 listener Listener Infrastructure 1 Module WaitSet 1 * Condition 0..1 statuscondition StatusCondition DomainEntity Domain * 1 Module Domain Module DomainParticipant <<summary>> A DomainParticipant is the entry-point for the service and isolates a set on applications that share a physical network. Publisher * DataWriter Topic 1 * 1 Publication Module * 1 Topic Module <<interface>> DataType <<implicit>> * DataReader * Subscriber Subscription Module 1 Subscription Module Data

85 QoS Policies 84 QosPolicy name : string DurabilityQosPolicy kind : DurabilityQosKind LatencyBudgetQosPolicy duration : Duration_t PartitionQosPolicy name [*] : string OwnershipQosPolicy kind : OwnershipQosKind DeadlineQosPolicy period : Duration_t ReliabilityQosPolicy kind : ReliabilityQosKind OwnershipStrengthQosPolicy value : long TimeBasedFilterQosPolicy minimum_separation : Duration_t DestinationOrderQosPolicy kind : DestinationOrderQosKind LivelinessQosPolicy lease_duration : Duration_t kind : LivelinessQosKind UserDataQosPolicy data [*] : char HistoryQosPolicy kind : HistoryQosKind depth : long PresentationQosPolicy access_scope : PresentationQosAccessScopeKind coherent_access : boolean ordered_access : boolean ResourceLimitsQosPolicy max_samples : long max_instances : long max_samples_per_instance : long

86 DDS by Example

87 DDS by Example: building a mini CMS, 86 SENSOR PROCESS CLASSIFICATION PROCESS DISPLAY PROCESS Optical sensor Scans the environment Produces Tracks Position of objects Reports pointtrack Classifies tracks Determines their identity Analyses the trajectories Determines hostility Reports trackstate Displays track info Both position & identity Raises alerts Requires pointtrack Requires trackstate

88 DATA DEFINITION 87 Information modeled as TOPICS Each TOPIC has an associated name and data type Data-definition in IDL Key fields for unique identification pos Relational Data Model (keys) Our example: PointTrack Track trackid TrackState identity Topic PointTrack Struct PointTrackType { long trackid; // key Topic TrackState Struct TrackStateType { long trackid; // key } Position pos; } Id identity;

89 DDS by Example: the Minimum Profile 88 Sensor Topic PointTrack { long trackid; Position pos; } Key: trackid QoS: best-effort, volatile Display PointTrack Publisher PointTrack Topics PointTrack Subscriber OMG-DDS Real-time Information Backbone Characteristics Basic publish/subscribe data distribution Topics (types) specified in IDL QoS regarding: reliability, urgency, priority, etc. Features / Advantages Autonomous & loosely coupled applications Pub/Sub & QoS driven communication Strong-typed interfaces Smart networking based on priority & latency budget

90 DDS by Example: The Ownership Profile 89 Sensor-1 Sensor-2 Display Publisher-1 Strength=2 Publisher-2 Strength=1 PointTrack Subscriber OMG-DDS Real-time Information Backbone Characteristics Replicated publishers of data (with own strength) Only highest-strength will be received On failure, next highest-strength will take-over Features / Advantages Fault-tolerance by replication Notes: Requires a lot of resources Quality must be expressible as an integer

91 DDS by Example: The Persistence Profile 90 PERSISTENT (on Disk) TRANSIENT (in Memory) Built-in Durability-Service OMG-DDS Real-time Information Backbone Characteristics Built-in persistence for non-volatile data State preservation for transient publishers Settings persistence surviving system downtime Replicated durability service for maximal fault-tolerance

92 DDS by Example: The Persistence Profile 91 topic TrackState { long trackid; Id Identity;; } Key: trackid QoS: Reliable, Transient Display PERSISTENT (on Disk) TRANSIENT (in Memory) Built-in Durability-Service PointTrack Subscriber TrackState Subscriber Case-1 OMG-DDS Real-time Information Backbone Characteristics Built-in persistence for non-volatile data State preservation for transient publishers Settings persistence surviving system downtime Replicated durability service for maximal fault-tolerance Features / Advantages Case-1: late-joining of Display process Previously produced TrackStates readily available Case-2: restart of failed Classification process Internal state (already classified tracks) regained

93 DDS by Example: The Persistence Profile 92 Classification Case-2 topic TrackState { long trackid; Id Identity;; } Key: trackid QoS: Reliable, Transient PERSISTENT (on Disk) TRANSIENT (in Memory) PointTrack Subscriber TrackState Publisher & Subscriber Built-in Durability-Service OMG-DDS Real-time Information Backbone Characteristics Built-in persistence for non-volatile data State preservation for transient publishers Settings persistence surviving system downtime Replicated durability service for maximal fault-tolerance Features / Advantages Case-1: late-joining of Display process Previously produced TrackStates readily available Case-2: restart of failed Classification process Internal state (already classified tracks) regained

94 DDS by Example: The Content Subscription Profile 93 Track trackid Display PointTrack pos TrackState identity Content-filtered Multi-topic Subscriber OMG-DDS Real-time Information Backbone Characteristics Adds content awareness Content-filtered Topics & query-conditions Supports compound interest Multi-topics (combine/filter/re-arrange topics) Features / Advantages Reduced application complexity SQL-like querying and reconstitution of related data Improved system performance Only receive/process what is of interest

95 DDS by Example: The DLRL Profile 94 Track trackid Display PointTrack pos TrackState identity Topics - need to be shared Application DLRL DCPS DLRL classes are only local DLRL Subscriber OMG-DDS Real-time Information Backbone Characteristics Local Object Oriented Data-Access Layer Supports OO features: Inheritance, aggregation, composition Uses DCPS to distribute state by mapped topics Features / Advantages Ease of Management of (related) data Object oriented graphs of objects (value-types) Supports native language constructs (I.e. navigation) Automatic change-management of objects

96 CORBA & DDS integration

97 Object Management Group CORBA & DDS Server Behavior Data Availability App App App Client CORBA Server OMG - DDS CORBA DDS Distributed objects Model: Client/Server Remote Method Invocation Reliable Communication Purpose: Distributed Processing Synchronous Transactions Distributed data Model: Publish/Subscribe Distributed Information Access Fine-grained QoS Purpose: Real-time data distribution Fault tolerant Info Management 7/2/2007 Copyright Object Management Group 96

98 CORBA & DDS DDS: Combining Strength 97 Information-model IDL CORBA Object trackstate; Getpos(); ClassifyTrack(); + DDS Topic PointTrack { long trackid; Position pos; } Key: trackid QoS: best-effort, volatile App App Client CORBA Server App App OMG - DDS OMG - DDS Characteristics & Benefits Corba/DDS Common Definition language: IDL Type definition for CORBA-interfaces & DDS topics Code generation : Type-generation as well as generated (typed-)interfaces Potential seamless Runtime Cooperation Shared types allow direct passing-on of RPC-obtained information into DDS-topics Autonomous runtime-systems (ORB and DDS libraries)

99 CHATROOM EXAMPLE

100 Exercise Traditional Chat architecture 99 Chat-Client 1 Chat-Client 2 Chat-Server Chat-Client 3 Chat-Client 4

101 Exercise DDS based Chat architecture 100 Domain Participant Chatter 1 Chatter 3 DDS-Chat Domain Domain Participant Domain Participant Domain Participant Chatter 2 Chatter 4

102 Chatroom example: Sequence of events 101 Typical sequence of events on a traditional Chat-application: Typical sequence of events on a DDS-based Chat-application: Connect to the Chat-Server. Transmit your identity. Download the identities of the other chatters. Receive chat messages from other users. These messages are forwarded to you by the server. Write your own chat-messages. These messages are forwarded by the server to all the other users. Participate in the Chat-domain. Publish your identity. Subscribe yourself to the identities of all other chatters Subscribe yourself to all chatmessages in the Chat-domain. All messages are delivered to you directly by their respective writers. Publish your own chat-messages. Your messages are directly delivered to all the other interested users.

103 DEMO: DDS-based Chatroom 102 ChatMessage { Domain Participant Domain Participant DDS TUNER int userid; string Message; } Key: userid QoS: Reliable, Volatile NameService { int userid; string Name } Key: userid QoS: Reliable, Transient Chatter Hans userid = 1 Chatter Robert userid = 2 DDS-Chat Domain Information-model SHOWS ALL MESSAGES Domain Participant Message - Domain Participant User - TRACKS ALL CHATTERS Board Load

104 Demo DDS architecture: OpenSplice example 103 Choices: Scalability & Efficiency Single shared library for applications & services Ring-fenced shared memory segment Data urgency driven network-packing (code-footprint) (single copy regardless of nr. of applications) (Latency_budget QOS drives packing per channel) Choices: Determinism & Safety Pre-emptive network-scheduler (priority pre-emptive network-threads per priority-band with traffic-shaping) Data importance based network-channel selection (Transport_Priority QoS of actual data) Partition based multicast-group selection (dynamic mapping of logical DDS partitions) Managed critical network-resource (limited impact/damage of faulty-applications)

105 DEMO: DDS-based Chatroom: DCPS, DLRL.. and even SQL 104 Domain Participant Domain Participant OpenSplice TUNER ChatMessage { int userid; string Message; } Key: userid QoS: Reliable, Volatile NameService { int userid; string Name } Key: userid QoS: Reliable, Transient Chatter Hans userid = 1 Chatter Robert userid = 2 Information-model DDS-Chat Domain (ChatRoom (ChatRoomPartition) Domain Participant Domain Participant Chat userid DBMS Service Filter : userid!= 1 mysql DBMS DBMS Service Filter : userid!= 2 ANTs DBMS DCPS Message - Board DLRL-layer DLRL OO Message - Board ChatMessage Message NameService Name Object-model SQL mysql Message Board SQL ANTs Message- Board

106 MDE and DLRL: Messageboard example, Reducing Complexity!! % Reduction % 71 % DCPS DCPS reduction reduction MDE 67 DLRL MDE 123 % increase SQL 212 DCPS- Message- Board Managed info (DLRL) & MDE: a factor of 10! 393 DCPS Multitopic Emulation MDE- Generated DLRL Messageboard % reduction MDE- Generated SQL Messageboard DCPS DCPS-MDE DLRL DLRL-MDE SQL

107 CHATROOM LIVE DEMO

108 Q&A