Building Python Messaging Apps with Oracle Database

Transcription

1 Building Python Messaging Apps with Oracle Database CON 7344 Anthony Tuininga ConsulGng Member of Technical Staff Data Access Development, Oracle Database October 3 rd, 2017

2 Safe Harbor Statement The following is intended to outline our general product direcgon. It is intended for informagon purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or funcgonality, and should not be relied upon in making purchasing decisions. The development, release, and Gming of any features or funcgonality described for Oracle s products remains at the sole discregon of Oracle. 2

3 About Me Anthony Tuininga SoQware Developer for Oracle Database, ScripGng Language Drivers (and related technology) Creator/maintainer of cx_oracle Contact 3

4 Program Agenda IntroducGon Code Future cx_oracle Enhancements AQ Features 4

5 Program Agenda IntroducGon Code Future cx_oracle Enhancements AQ Features 5

6 IntroducGon ConfidenGal Oracle Internal/Restricted/Highly Restricted 6

7 cx_oracle Background Open Source package giving CPython access to Oracle Database Covers all of Python Database API specificagon with many addigons to support Oracle advanced features Begun by Anthony Tuininga in 1998 using Oracle 8i and Python releases covering Oracle 8i through 12c Current cx_oracle version is 6.0, released August 2017 Support for advanced queueing (AQ) added in 5.3 7

8 What is Advanced Queuing (AQ)? Stores messages in abstract storage units called queues Messages are enqueued (added to the queue) by the producer Messages are dequeued (removed from the queue) by the consumer Producers and consumers do not need to be aware of each other Messages stay in the queue ungl they are consumed or they expire Consumers can be other queues in the same or a remote database Messages can be persistent or buffered (in-memory) 8

9 AQ Features In ApplicaGon CondiGons can be specified by consumers to limit the messages they see MulGple consumer queues allow mulgple consumers to read the same message independently of each other Messages can have a shelf life aqer which they expire Messages can specify an idengfier ( correlagon ) which is not managed by AQ in any way 9

10 AddiGonal AQ Features Messages can be delayed a period of Gme before they can be dequeued The order in which messages are dequeued can be specified Messages can be transformed from one type to another before being enqueued or dequeued Can peek at messages without consuming them Can define excepgon queues for managing excepgons Messages can be in-memory only for increased performance for transient messages which don t require persistence 10

11 ApplicaGon Scenario Customers Manufacturers Stores 11

12 Program Agenda IntroducGon Code Future cx_oracle Enhancements AQ Features 12

13 Code ConfidenGal Oracle Internal/Restricted/Highly Restricted 13

14 ApplicaGon Architecture Head Office Store S Store T Manufacturer A Manufacturer B Send Orders Process Orders Process Orders Process RFQs Process RFQs Process Quotes Process Quotes Orders Quotes Quotes Quote Responses Quote Responses RFQs 14

15 Send Orders Architecture Head Office Send Orders Orders 15

16 Send Orders Overview Microservice sends orders from customers to the stores using a single consumer queue called OrderQueue Object type UDT_ORDER used as message payload AQ property correlagon used to idengfy the order number 16

17 Send Orders SQL Code Part 1 1 create type udt_orderline as object ( 2 WidgetTypeCode varchar2(30), 3 NumBoxes number(9) 4 ); 5 / 6 7 create type udt_orderlines as varray(20) of udt_orderline; 8 / 9 10 create type udt_order as object ( 11 CustomerName varchar2(60), 12 Customer Address varchar2(250), 13 StoreCode varchar2(30), 14 Lines udt_orderlines 15 ); 16 / 17

18 Send Orders SQL Code Part 2 1 begin 2 dbms_aqadm.create_queue_table('ordermessages', 'UDT_ORDER'); 3 dbms_aqadm.create_queue('orderqueue', 'OrderMessages'); 4 dbms_aqadm.start_queue('orderqueue'); 5 end; 6 / 18

19 Send Orders Python Code Part 1 1 orderobjtype = connection.gettype("udt_order") 2 orderobj = orderobjtype.newobject() 3 orderobj.customername = order.customername 4 orderobj.customer address = order.customer address 5 orderobj.storecode = order.storecode 6 7 orderlineobjtype = connection.gettype("udt_orderline") 8 orderlinesobjtype = connection.gettype("udt_orderlines") 9 orderlinesobj = orderlinesobjtype.newobject() 10 for line in order.lines: 11 lineobj = orderlineobjtype.newobject() 12 lineobj.widgettypecode = line.widgettypecode 13 lineobj.numboxes = line.numboxes 14 orderlinesobj.append(lineobj) 15 orderobj.lines = orderlinesobj 19

20 Send Orders Python Code Part 2 16 options = connection.enqoptions() 17 messageproperties = connection.msgproperties() 18 messageproperties.correlation = order.ordernumber 19 connection.enq("orderqueue", options, messageproperties, orderobj) 20 connection.commit() 20

21 Process Orders Architecture Head Office Store S Store T Process Orders Process Orders Orders RFQs 21

22 Process Orders Overview A separate microservice is used by each store to process its orders from the OrderQueue queue The AQ property condigon is used to ensure that only messages for the pargcular store are dequeued As each order is dequeued, a quote is requested for each order line The store locagon is included in the message so that the manufacturers can use that informagon to make a decision on whether to respond to the request for a quote A mulg-consumer queue RFQQueue is used so that each manufacturer can manage its messages independently of all other manufacturers The object type UDT_RFQ is used as the message payload 22

23 Process Orders SQL Code Part 1 1 create type udt_rfq as object ( 2 Location sdo_geometry, 3 WidgetTypeCode varchar2(30), 4 NumBoxes number(9), 5 QueueName varchar2(30) 6 ); 7 / 23

24 Process Orders SQL Code Part 2 1 begin 2 dbms_aqadm.create_queue_table('rfqmessages', 'UDT_RFQ', 3 multiple_consumers => true); 4 dbms_aqadm.create_queue('rfqqueue', 'RFQMessages'); 5 dbms_aqadm.start_queue('rfqqueue'); 6 end; 7 / 24

25 Process Orders Python Code Part 1 1 orderobjtype = connection.gettype("udt_order") 2 orderobj = orderobjtype() # or orderobjtype.newobject() 3 deqoptions = connection.deqoptions() 4 deqoptions.navigation = cx_oracle.deq_first_msg 5 deqoptions.wait = cx_oracle.deq_no_wait 6 deqoptions.condition = "tab.user_data.storecode = '%s'" % STORE_CODE 7 messageproperties = connection.msgproperties() 8 9 rfqobjtype = connection.gettype("udt_rfq") 10 rfqobj = rfqobjtype.newobject() 11 rfqobj.location = STORE_LOCATION 12 enqoptions = connection.enqoptions() 25

26 Process Orders Python Code Part 2 13 while connection.deq("orderqueue", deqoptions, messageproperties, orderobj): 14 ordernumber = messageproperties.correlation 15 for linenum, lineobj in enumerate(orderobj.lines.aslist()): 16 rfqobj.widgettypecode = lineobj.widgettypecode 17 rfqobj.numboxes = lineobj.numboxes 18 rfqobj.queuename = "QuoteQueue" + STORE_CODE 19 messageproperties.correlation = "%s_%s" % (ordernumber, linenum + 1) 20 connection.enq("rfqqueue", enqoptions, messageproperties, rfqobj) 21 connection.commit() 26

27 Process RFQs Architecture Head Office Store S Store T Manufacturer A Process RFQs Manufacturer B Process RFQs RFQs Quotes Quotes 27

28 Manufacturer Response to RFQ? 28

29 Process RFQs Overview A separate microservice is used by each manufacturer to process requests for quotes from stores and send quotes back to the stores Each manufacturer subscribes to the mulgple consumer queue RFQQueue using its own rule SpaGal Data Objects (SDO) is used to limit the messages received to only those from stores within a certain distance AQ filters the messages for subscribers as messages are added to the queue The object type UDT_QUOTE is used as the message payload The AQ property expiragon is used to specify how long the quote is valid Once the Gme has expired without being dequeued, it is put into an excepgon queue 29

30 Process RFQs SQL Code Part 1 1 create type udt_quote as object ( 2 WidgetTypeCode varchar2(30), 3 NumBoxes number(9), 4 AvailableDate date, 5 TotalCost number(11, 2), 6 ManufacturerCode varchar2(30), 7 QueueName varchar2(30) 8 ); 9 / 30

31 Process RFQs SQL Code Part 2 1 begin 2 for r in (select * from Manufacturers) loop 3 dbms_aqadm.add_subscriber('rfqqueue', 4 sys.aq$_agent('manufacturer' r.code, null, null), 5 rule => 'sdo_within_distance(tab.user_data.location, ' 6 '( select location ' 7 ' from Manufacturers ' 8 ' where Code = ' r.code 9 '), ''distance=' r.maxdistancetostore 10 ' unit=km'') = ''TRUE'''); 11 end loop; 12 dbms_aqadm.create_queue_table('quotemessages', 'UDT_QUOTE'); 13 for r in (select * from Stores) loop 14 dbms_aqadm.create_queue('quotequeue' r.code, 'QuoteMessages'); 15 dbms_aqadm.start_queue('quotequeue' r.code); 16 end loop; 17 end; 18 / 31

32 Process RFQs Python Code Part 1 1 rfqobjtype = connection.gettype("udt_rfq") 2 rfqobj = rfqobjtype.newobject() 3 deqoptions = connection.deqoptions() 4 deqoptions.navigation = cx_oracle.deq_first_msg 5 deqoptions.wait = cx_oracle.deq_no_wait 6 deqoptions.consumername = "Manufacturer" + MANUFACTURER_CODE 7 messageproperties = connection.msgproperties() 8 9 quoteobjtype = connection.gettype("udt_quote") 10 quoteobj = quoteobjtype.newobject() 11 enqoptions = connection.enqoptions() 32

33 Process RFQs Python Code Part 2 13 while connection.deq("rfqqueue", deqoptions, messageproperties, rfqobj): 14 quoteobj.widgettypecode = rfqobj.widgettypecode 15 quoteobj.numboxes = rfqobj.numboxes 16 quoteobj.availabledate = datetime.datetime.today() + \ 17 datetime.timedelta(days[rfqobj.widgettypecode]) 18 quoteobj.totalcost = rfqobj.numboxes * PRICES[rfqObj.WIDGETTYPECODE] 19 quoteobj.queuename = "QuoteResponseQueue" + MANUFACTURER_CODE 20 quoteobj.manufacturercode = MANUFACTURER_CODE 21 messageproperties.expiration = 24 * 60 * connection.enq(rfqobj.queuename, enqoptions, messageproperties, quoteobj) 23 connection.commit() 33

34 Process Quotes Architecture Head Office Store S Store T Manufacturer A Manufacturer B Process Quotes Process Quotes Quotes Quotes Quote Responses Quote Responses 34

35 Process Quotes Overview A separate microservice is used by each store to process quotes from manufacturers and send a response back to them The object type UDT_QUOTERESPONSE is used as the message payload 35

36 Process Quotes SQL Code Part 1 1 create type udt_quoteresponse as object ( 2 Accepted char(1), 3 Comments varchar2(4000) 4 ); 5 / 36

37 Process Quotes SQL Code Part 2 1 begin 2 dbms_aqadm.create_queue_table('quoteresponsemessages', 3 'UDT_QUOTERESPONSE'); 4 for r in (select * from Manufacturers) loop 5 dbms_aqadm.create_queue('quoteresponsequeue' r.code, 6 'QuoteResponseMessages'); 7 dbms_aqadm.start_queue('quoteresponsequeue' r.code); 8 end loop; 9 end; 10 / 37

38 Process Quotes Python Code Part 1 1 quoteobjtype = connection.gettype("udt_quote") 2 quoteobj = quoteobjtype.newobject() 3 deqoptions = connection.deqoptions() 4 deqoptions.navigation = cx_oracle.deq_first_msg 5 deqoptions.wait = cx_oracle.deq_no_wait 6 messageproperties = connection.msgproperties() 7 8 quoteresponseobjtype = connection.gettype("udt_quoteresponse") 9 quoteresponseobj = quoteresponseobjtype.newobject() 10 enqoptions = connection.enqoptions() 38

39 Process Quotes Python Code Part 2 11 queuename = "QuoteQueue" + STORE_CODE 12 while connection.deq(queuename, deqoptions, messageproperties, quoteobj): 13 responsecomments = "Some comments if not accepted and None if accepted" 14 quoteresponseobj.accepted = "Y" if responsecomments is None else "N" 15 quoteresponseobj.comments = responsecomments 16 connection.enq(quoteobj.queuename, enqoptions, messageproperties, 17 quoteresponseobj) 18 connection.commit() 39

40 Further steps Manufacturers need to process quote responses and ensure that they are able to ship by the date originally promised Messages are dequeued from manufacturer specific QuoteResponseQueue Stores need to send an to customers when all order lines have been sagsfied by successful quotes from manufacturers Failure to send an should not abort the transacgon Failure to commit the transacgon cannot unsend A queue is needed to ensure that s are sent independently, but are sgll guaranteed to be sent 40

41 Program Agenda IntroducGon Code Future cx_oracle Enhancements AQ Features 41

42 Future cx_oracle Enhancements ConfidenGal Oracle Internal/Restricted/Highly Restricted 42

43 Future cx_oracle Enhancements? Support for raw queues Support for nogficagon of availability of messages to dequeue 43

44 Program Agenda IntroducGon Code Future cx_oracle Enhancements AQ Features 44

45 AQ Features ConfidenGal Oracle Internal/Restricted/Highly Restricted 45

46 About Me Anil Madan SoQware Development Director, Oracle Advanced Queuing Contact informagon 46

47 Advanced Queuing Overview IBM MQ TIBCO ORACLE Advanced Queues Application 3 PYTHON PL/SQL JMS JDBC OCI SOAP Web Services/ XML Publish ORACLE Transformation & Rules Engine Advanced Queues Queue Tables Subscribe Priority > 2 Subscribe corrid= ORDER_ID Application 1 Application 2

48 AQ Usage AQ is a crigcal part of the Fusion ApplicaGons SaaS and Middleware PaaS Several reasons for this Queues and asynchronous messaging is an established programming model TradiGonally these were done via Java Messaging Service (JMS) With the drive to stateless mid-gers (pupng state in the database instead), they have migrated from mid-ger JMS provider to using AQ This simplifies HA, DR, backup/recovery Major Middleware PaaS services (e.g., IdenGty Cloud Service) are being re-architected to use microservices The goal is to scale subsystems independently, greater resiliency, and agile development

49 AQ Usage Sample of Oracle SaaS, Middleware and Database using AQ Fusion ApplicaGons (Supply Chain Management, Distributed Order OrchestraGon, Enterprise Scheduler Service, ) Oracle Retail Oracle DaaS (Data as a Service) SOA Suite (Event Delivery Network) Eloqua E-Business Suite has over 100 queues in the schema Enterprise Manager Database internal users Database Change NoGficaGons, Database Alerts, Database Scheduler, Messaging Gateway, Datapump, EM alert events, FAN nogficagon and many more

50 Sharded Queues Architecture for Scalability and Performance Single logical queue model Physical Sharded Queue: automatic management of partitions on RAC

51 Sharded Queues Key benefits Higher throughput Less system resource consumpgon Large number of subscribers Event-based listener with fewer database connecgons Many concurrent enqueuers and dequeuers across mulgple RAC instances Backward compagble for Standard JMS and PL/SQL based applicagons just recreate the AQ in the database

52 InteresGng Sessions for your OpenWorld Experience Hands-on Lab Session: Python and Oracle Database 12c: ScripGng for the Future [HOL7605] Wednesday, Oct 04, 9:45 a.m. - 10:45 a.m. Hilton San Francisco Union Square (Ballroom Level) - ConGnental Ballroom 6 Python and Oracle Database: Tips, Tricks, and the Best New Features [CON6714] Wednesday, Oct 04, 4:30 p.m. - 5:15 p.m. Moscone West - Room 3012 Demo Booth: ApplicaGon Development with Node.js, Python, PHP, R, C, and C++ SOA-042, Moscone West 52

53 InteresGng Sessions for your OpenWorld Experience Node.js: JavaScript ApplicaGon Development for Oracle Database [CON6712] Tuesday, Oct 03, 5:45 p.m. - 6:30 p.m. Moscone West - Room 3008 Best PracGces for ApplicaGon High Availability [CON6711] Wednesday, Oct 04, 3:30 p.m. - 4:15 p.m. Moscone West - Room 3012 Performance and Scalability Techniques for Oracle Database ApplicaGons [CON6710] Wednesday, Oct 04, 5:30 p.m. - 6:15 p.m. Moscone West - Room

54 Resources Mail: Twiwer: Group Blog: cx_oracle Home: Install from: Code haps://blogs.oracle.com/opal haps://oracle.github.io/python-cx_oracle haps://pypi.python.org/pypi/cx_oracle haps://github.com/oracle/python-cx_oracle 54

55 Safe Harbor Statement The preceding is intended to outline our general product direcgon. It is intended for informagon purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or funcgonality, and should not be relied upon in making purchasing decisions. The development, release, and Gming of any features or funcgonality described for Oracle s products remains at the sole discregon of Oracle. 55

56 56

57