New feature in SP08
www.sap.com TABLE OF CONTENTS KEYED STREAMS... 3 Introduction... 3 A Deeper Look... 3 Supported Operations... 3 Filter Behavior... 4 Join Behavior... 5 Flex Behavior... 6 Interaction with Windows, Streams and Delta Streams... 6 Keyed Stream Feeding a Window... 6 Stream Feeding a Keyed Stream... 6 Keyed Streams vs Delta Streams... 7 Keyed Streams in Action... 7 Use Case Description... 7 Implementation... 8 Known Limitations... 8 Potential Future Enhancements... 9
KEYED STREAMS Introduction There has been considerable interest in generating and processing insert/update and delete events without actually storing events in a Window. To date, Delta Streams have come the closest to supporting this, but they do not support the following use cases: A stateless input that supports all the op codes. Support for generating updates in Flex operator bound to a stateless element. For technical reasons the semantics of a Delta Stream, which behaves like a Window with no state in terms of op code handling, cannot support these two use cases. So the notion of a Keyed Stream is being introduced in ESP SP08 that will satisfy these two use cases. A Deeper Look Keyed Streams are designed in such a way so that you can build a completely stateless ESP application that propagates inserts/updates and deletes. When this is done all state necessary to process and validate the insert/update and deletes is done by the application either within or outside of ESP. You can use Keyed Streams to enhance, filter, manipulate data using supported operations or generate your own events in a stateless manner without incurring any memory expense to store the data. A Keyed Stream behaves like a Stream in the sense that it treats all rows as inserts even though the records it receives and produces may contain other opcodes. To accomplish this, the Keyed Stream applies the specified logic on the incoming event and produces an output event with the same opcode as the incoming event. This is unlike Windows and Delta Streams that use the previous state of the record to intelligently determine the opcode for the result record. This means that the user is responsible for ensuring that output it produces is truly meaningful as it may not be in all the cases. A Keyed Stream has the following important properties: It requires Primary Keys. A Stream becomes a Keyed Stream simply by defining a key on it. Just like in a regular window the insert/update or delete operations are assumed to be with respect to this primary key. It just passes through insert/update and delete opcodes and rejects upsert and safedeletes. This is unlike regular Streams, which convert updates and upserts to inserts and filters out deletes and safedeletes. They are stateless for the most part except in the case when it performs a join with a Window. In this scenario memory is used to store a reference to the records in the Window. It does not automatically detect duplicate inserts, bad updates or bad deletes. It is the applications responsibility to detect any such errors and trap them. Keyed Streams may send data to a Stream and also receive input from Streams. When it receives inputs from Streams it is the applications responsibility to ensure that the keys are unique. Keyed Streams may receive data from a Window and it may send data to a Window. However when data is sent to a Window you must specify a retention policy. This is described later. Does not interact with Delta Streams. This is done because Delta Streams is being deprecated in favor of Keyed Streams. Keyed Streams support most but not all operations. Also certain operations behave differently or are restricted. See details below. Supported Operations Keyed Streams support the following operations when it is the target for the result of the operation: Inputs Computes 3
Filters. See Filter Behavior for more information. Simple Joins. See Join Behavior for more information. Unions Flex operations. See Flex Behavior for more details. Pattern Matching The following operations cannot be performed when a Keyed Stream is the target for the operation: Aggregations Joins where the inputs to the joins are only Windows. This can be achieved by first performing the Window-Window join and then feeding the results into a Keyed Stream. Note: When performing these operations keep in mind that opcodes are meaningless to Keyed Streams. It does not validate them and it does not modify them; it just passes them through. This implies that the results it produces will differ from what a Window will produce under certain circumstances. These changes are described below. Filter Behavior As mentioned earlier Keyed Streams ignores opcodes and just passes them through. The consequence of this behavior is that if filters must be performed on immutable expressions i.e whose value does not changes across insert/update and delete operations on the record. This means that you should not for example write a filter based upon a column whose value can be updated or use a function such as sysdate(), which produces different results each time it runs. The example below illustrates what may happen if this is done CREATE INPUT STREAM In1 SCHEMA (Key1 integer, Val1 string, Val2 integer) PRIMARY KEY(Key1); CREATE OUTPUT STREAM Out1 PRIMARY KEY DEDUCED AS SELECT * from In1 where Val2 > 10; Now consider the following input: <In1 ESP_OPS="i" Key1="1" Val1="SAP" Val2="5"/> <In1 ESP_OPS="u" Key1="1" Val1="SAP" Val2="15"/> <In1 ESP_OPS="d" Key1="1"/> If one where to feed these records into a Window you will get the following output: //Notice how the update is converted into an insert and the delete contains //the values from the last event for the key <In1 ESP_OPS="i" Key1="1" Val1="SAP" Val2="15"/> <In1 ESP_OPS="d" Key1="1" Val1="SAP" Val2="15"/> However a Keyed Stream produces the following output because it treats all rows as inserts regardless of the actual opcodes in the event: //Notice how the opcode has not changed and there is no delete event produced //because the delete event did not pass the filter. <In1 ESP_OPS="u" Key1="1" Val1="SAP" Val2="15"/> ESP itself is not affected by the fact an update event is produced by Out1 without a corresponding insert and does not complain. However an external client may be affected by this if it is not designed to handle such a situation. The preferred way to use a Filter with Keyed Streams is to use it with one of the immutable columns i.e. Key1 or Val1 in the above example. 4
Join Behavior Keyed Stream behaves very much like a Stream when it comes to Joins in the sense that the joins are more like a look-up than a true join which happens in the case of a Window-Window join. The following rules apply to Keyed Streams in relation to Joins Only one Keyed Stream can participate in a join while any number of Windows, Unnamed Windows and References can participate. Keyed Streams cannot be joined with Streams and Delta Streams. Keyed Streams can participate only in simple joins where all windows participating in a join are directly joined to the keyed streams. Complicated joins that require intermediate join steps are not supported. Like a Stream only events arriving in the Keyed Streams trigger a join. Changes to Windows do not propagate a joins. Keyed Streams support the following types of joins o Inner Joins o Outer Joins (Left and Right Joins) as long as the Keyed Stream is the outer member of an outer join. Full joins are not supported. Keyed Streams support all the cardinalities namely o One-One o One-Many o Many-Many The result of the join can be fed into a Keyed Stream or a Stream. Although the later will produce a compiler warning. Just like in the case of a Window, Key rules are strictly enforced if the target of the join is a Keyed Stream. As the joins that a Keyed Stream participate in are more like a look up than a true join it is important that in the case of an inner join the columns on both sides of the join on which the joins are performed do not change. Otherwise it is possible for the join to produce an output with an update or delete opcode without a preceding insert or for a delete to never get propagated because the join failed. Consider the following example that illustrates this issue: CREATE INPUT STREAM In1 SCHEMA (Key1 integer, Val1 string, FK1 integer) PRIMARY KEY(Key1); CREATE INPUT WINDOW InWin1 SCHEMA (Key2 integer, Val2 string, Val3 integer) PRIMARY KEY(Key2); CREATE OUTPUT WINDOW Out1 PRIMARY KEY DEDUCED AS SELECT In1.Key1, In1.FK1, InWin1.Val3 FROM In1 INNER JOIN InWin1 ON In1.FK1 = InWin1.Key2; Now consider the following inputs for In1 and InWin1 that are processed in the specified order: <InWin1 ESP_OPS="i" Key2="10" Val3="SAP10" Val4="100"/> <In1 ESP_OPS="i" Key1="1" Val1="SAP1" FK1="5"/> <In1 ESP_OPS="u" Key1="1" Val1="SAP" FK1="10"/> <InWin1 ESP_OPS="d" Key2="10" Val3="SAP10" Val4="100"/> <In1 ESP_OPS="d" Key1="1" Val1="SAP" FK1="20"/> The output produced by this sequence of events is as follows: <Out1 ESP_OPS="u" Key1="1" FK1="10" Val3="SAP10"/> The issue here is that Out1 produces an update event for key "1" without a corresponding insert event preceding it and also the delete event does not get propagated because the corresponding event in WinIn1 5
has been deleted previously. if this is acceptable then the client application must be able to handle such scenarios. If In1 is a Window instead of a Keyed Stream the output will be consistent and is as follows: <Out1 ESP_OPS="i" Key1="1" FK1="10" Val3="SAP10"/> <Out1 ESP_OPS="d" Key1="1" FK1="10" Val3="SAP10"/> The preferred way to get correct results from inner joins involving Keyed Streams is that the value of FK1, in this example, remains unchanged across insert/update and deletes and the contents of Win1 are not deleted. Flex Behavior Keyed Stream can work with Flex operators just like a Window or a Delta Stream with a couple of differences When a Keyed Stream is an input to a Flex operator you don't have access to the old record when you receive an update; the old record will always be null. When a Keyed Stream is an target to ta Flex operator you can only output insert/update and delete opcodes. Upsert and SafeDelete opcodes will be rejected by the server. Interaction with Windows, Streams and Delta Streams The table below summarizes the supported interactions when a Keyed Stream is a source or a target for a Window, Streams, Delta Streams or another Keyed Streams Source Target Stream Window Delta Stream Keyed Stream Yes. Compiler Generates a Yes. Must create an unnamed No Yes warning. window on a Keyed Stream Yes. Application must ensure Yes No Yes keys are valid Keyed Stream Feeding a Window When a Keyed Stream feeds a Window you need to create an unnamed window over the Keyed Stream before you can use it as input to a Window. So for example: //You cannot say CREATE OUTPUT WINDOW Win1 PRIMARY KEY DEDUCED AS SELECT * FROM KeyedStream; //Instead You will have to say CREATE OUTPUT WINDOW Win1 PRIMARY KEY DEDUCED AS SELECT * FROM KeyedStream KEEP ALL; There are two reasons for doing this The major reason is that, for performance reasons, not all operators can handle inconsistent data i.e duplicate inserts, bad updates and deletes. Unnamed Windows is one of the operators that can handle such inconsistent data. It makes it obvious that a stateless element is being converted to have state and thus will incur a memory cost. Stream Feeding a Keyed Stream When a Stream feeds a Keyed Stream the application is responsible for ensuring that there are no duplicates insert or invalid updates or deletes. The ESP server cannot not detect this because of the stateless nature of Keyed Streams. 6
Keyed Streams vs Delta Streams Keyed Streams and Delta Streams on the surface are very similar in that they are both stateless and both can propagate insert/update and delete opcodes. However they are different in a few subtle but significant ways. The differences arise from the core semantics that the two follow when dealing with data. A Keyed Stream follows the Stream semantics while allowing all opcodes to pass through it unchanged whereas the Delta Stream follows Window Semantics while being stateless. The table below describes the differences. Treatment of Opcodes Behavior with Filters Behavior with Joins Input to a Flex Operator Output from a Flex Operator Source to a Window Keyed Streams Ignores opcodes. Just passes them through. Works like a Stream except the opcodes on the incoming record is preserved and not converted to inserts. Works like a Stream except the opcodes on the incoming record is preserved. No access to old records on an update. Can output insert/update and delete opcodes Can feed a Window but always need to create a unnamed window (specify a KEEP clause) Delta Streams Manipulates opcodes just like Windows. Works like a Window. Modifies opcodes as needed. Cannot participate in Joins. Access to the previous record on an update. Can only generate insert and delete opcodes Can feed a Window. Specifying a retention is optional mostly but required when it is an input for a join. Use for Inputs Can be used as a input element. Cannot be used as an input element. Needs to always be fed by a Window. Ease of Use Easier to use because of consistent semantics. Semantics is easier on the surface but inconsistent and prone to create subtle bugs that are difficult to catch. With the introduction of Keyed Streams the direct use of Delta Streams is being deprecated. Existing projects will work as is and you will be able to create and modify Delta Streams only using CCL. You can still view the Delta Streams you create via CCL visually. The compiler will create Delta Streams when you use a Window as a source to a Splitter and may optimize memory consumption by replacing locally visible windows with a Delta Streams when feasible. Keyed Streams in Action This is an example how you can use Keyed Streams to reduce memory consumption by not storing results when it is not needed. Use Case Description The applications receives a Price Feed containing all Symbols traded in an exchange. The application needs to identify only trades for Symbols in a Portfolio and store the results in a Hana database. The price feed may receive corrections(updates) to the trade price or trade size. When either the trade price or the trade size is 0 then the trade needs to be deleted from Hana. 7
Implementation //Create a Keyed Stream that can accept data in a stateless fashion. CREATE INPUT STREAM PriceFeed SCHEMA (Id integer, Symbol string, Price double, Shares integer, TradeTime date) PRIMARY KEY(Id); //Create a Input Window to hold the current positions. CREATE INPUT WINDOW Positions SCHEMA (Symbol string, SharesHeld integer, AveragePrice double) PRIMARY KEY (Symbol) KEEP ALL; //A Keyed Stream that essentially does a filter by producing // an output only when there is a position for a Symbol. CREATE LOCAL STREAM TradesForPositions PRIMARY KEY (Id) AS SELECT PriceFeed.* FROM PriceFeed INNER JOIN Positions ON PriceFeed.Symbol = Positions.Symbol; //A Flex Keyed Stream that produces inserts, updates or deletes. //The Hana table has the same schema as the PriceFeed Keyed Stream. CREATE FLEX HanaOutput IN TradesForPositions OUT OUTPUT STREAM HanaOutput SCHEMA (Id integer, Symbol string, Price double, Shares integer, TradeTime date) PRIMARY KEY(Id) BEGIN ON TradesForPositions { integer opcode := getopcode(tradesforpositions); //sets the opcode to delete if the Price or the Shares is 0 if((tradesforpositions.price = 0.0 or TradesForPositions.Shares = 0)){ opcode := delete; } //Outputs the incoming record with the incoming opcode //unless if the price or shares is 0 in which case it //issues a delete. output setopcode(tradesforpositions, opcode); }; END; //Attach an adapter to send the results to Hana ATTACH OUTPUT ADAPTER tosaphana TYPE hana_out TO HanaOutput PROPERTIES service = 'hanaservice', table = 'TradePositions'; As you will notice, the entire application was made stateless by using Keyed Streams except for the Positions Window. Without Keyed Streams the PriceFeed, TradesForPositions and HanaOutput elements would have had to be Windows and thus would consume significant amount of memory. Known Limitations When a Stream feeds an aggregation it is possible do an aggregation without actually storing the individual events when the aggregation only uses additive operations. This is possible because Streams have no keys and produce only inserts. With a Keyed Streams this is not possible because we need some way to validate that the insert/update and delete events are valid. This can only be done by storing the events in an unnamed window first. 8
Potential Future Enhancements Add support for upserts and safedeletes. This has been intentionally left out because most of our output adapters cannot support this feature. When the adapters can support these opcodes we can reconsider adding support for them in the Keyed Streams. Add a Property to Keyed Streams to make it optionally behave like Delta Streams with respect to Filters. This is only possible when the Keyed Stream is directly fed by a window. This can however be simulated by using a Flex Operator with the target of the operator being a Keyed Stream. 9
www.sap.com 2014 SAP AG. All rights reserved. SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, SAP HANA, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an SAP company. Sybase and Adaptive Server, ianywhere, Sybase 365, SQL Anywhere, and other Sybase products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Sybase Inc. Sybase is an SAP company. Crossgate, m@gic EDDY, B2B 360, and B2B 360 Services are registered trademarks of Crossgate AG in Germany and other countries. Crossgate is an SAP company. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. These materials are subject to change without notice. These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.