Developing node-mdb. SimpleDB emulation using Node.js and GT.M

Transcription

1 Developing node-mdb SimpleDB emulation using Node.js and GT.M Rob Tweed M/Gateway Developments Ltd

2 Could you translate that title? SimpleDB: Amazon s NoSQL cloud database Node.js: evented server-side Javascript (using V8) GT.M: Open source global-storage based NoSQL database node-mdb Open source emulation of SimpleDB

3 SimpleDB Amazon s cloud database Pay as you go Secure interface Schema-free NoSQL database Spreadsheet-like database model Domains (= tables) Items (= rows) Attributes (=cells)» Values (1+ per attribute allowed) SQL-like query API

4 Why emulate SimpleDB? Because I could! Kind of cool project

5 Why emulate SimpleDB? To provide a free, locally-available database that behaved identically to SimpleDB Lots of off-the-shelf available clients Standalone Bolso Mindscape s SimpleDB Management Tools Language-specific clients boto (Python) Official AWS clients for Java,.Net Node.js etc

6 Why emulate SimpleDB? To perform local tests prior to committing to production on SimpleDB To provide a live, local backup database A SimpleDB database for private clouds To provide an immediately-consistent SimpleDB database SimpleDB is eventually consistent

7 Why the GT.M database? I m familiar with it Free Open Source NoSQL database Schema-free Globals : Sparse persistent multi-dimensional arrays Hierarchical database Completely dynamic storage No pre-declaration or specification needed Result: trivial to model SimpleDB in globals node-mdb: Good way to demonstrate the capabilities of the otherwise littleknown GT.M More info Google: GT.M database universalnosql

8 Why write it using Node.js? M/DB originally written in late 2008 Implemented using GT.M s native scripting language (M) Apache + m_apache gateway to GT.M for interface I ve been working with Node.js for about a year now Rewriting M/DB in Javascript would make it more widely interesting and comprehensible Some performance issues reported with M/DB when being pushed hard

9 Why Node.js? Conclusion: Re-implementing M/DB using Node.js should provide better performance and scalability Fewer moving parts: Apache + m_apache + GT.M / multi-threaded Node.js + GT.M as child processes / single-thread Cool Node.js project to attempt Great example of non-trivial use of Node.js + database

10 How does SimpleDB work? Incoming SDB Request Security Key Id Secret Key SimpleDB Database Copy 1 Outgoing SDB Response Server Authenticate Request (HMacSHA) Error Generate Response Execute API Action Success and/or data/results SimpleDB SimpleDB Database SimpleDB Database Copy Database 2 Copy 2 Copy 2 SimpleDB Database Copy n

11 Node.js can emulate all this Incoming SDB Request Security Key Id Secret Key SimpleDB Database Copy 1 Outgoing SDB Response Server Authenticate Request (HMacSHA) Error Generate Response Execute API Action Success and/or data/results SimpleDB SimpleDB Database SimpleDB Database Copy Database 2 Copy 2 Copy 2 SimpleDB Database Copy n

12 GT.M can emulate this Incoming SDB Request Server Security Key Id Secret Key Authenticate Request Execute API Action SimpleDB Database Copy 1 Outgoing SDB Response Error Generate Response Success and/or data/results

13 Node.js characteristics Single threaded process Event loop Non-blocking I/O Asynchronous calls to functions that handle I/O Event-driven call-back functions when function completes Data fetched Data saved

14 Result: deeply nested call-backs Security Key Id Secret Key Server Authenticate Request Execute API Action Error Generate Response Success and/or data/results

15 Flattening the call-back nesting http.createserver(function(req,res) {..} var executeapi = function() { }; http server processsdbrequest() executeapi() sendresponse() var processsdbrequest = function() { };

16 Node.js Server http.createserver(function(request, response) { request.content = ''; request.on("data", function(chunk) { request.content += chunk; }); request.on("end", function(){ var SDB = {starttime: new Date().getTime(), request: request, response: response }; var urlobj = url.parse(request.url, true); if (request.method === 'POST') { SDB.nvps = parsecontent(request.content); } else { SDB.nvps = urlobj.query; } var uri = urlobj.pathname; if ((uri.indexof(sdburlpattern)!== -1) (uri.indexof(mdburlpattern)!== -1)) { processsdbrequest(sdb); } else { var uristring = ' + request.headers.host + request.url; var error = {code:'invaliduri', message: 'The URI ' + uristring + ' is not valid',status:400}; returnerror(sdb,error); } }); }).listen(httpport);

17 processsdbrequest() var processsdbrequest = function(sdb) { var accesskeyid = SDB.nvps.AWSAccessKeyId; if (!accesskeyid) { var error = {code:'authmissingfailure', message: 'AWS was not able to authenticate the request: access credentials are missing',status:403}; returnerror(sdb, error); } else { MDB.getGlobal('MDBUAF', ['keys', accesskeyid], function (error, results) { if (!error) { if (results.value!== '') { accesskey[accesskeyid] = results.value; validatesdbrequest(sdb, results.value); } else { var error = {code:'authmissingfailure', message: 'AWS was not able to authenticate the request: access credentials are missing',status:403}; returnerror(sdb, error); } } }); } };

18 validatesdbrequest() var validatesdbrequest = function(sdb, secretkey) { var type = HmacSHA256 ; var stringtosign = createstringtosign(sdb, true); var hash = digest(stringtosign, secretkey, type); if (hash === SDB.nvps.Signature) { processsdbaction(sdb); } else { errorresponse('signaturedoesnotmatch', SDB) } };

19 stringtosign() POST{lf} :8081{lf} /{lf} AWSAccessKeyId=rob&Action=ListDomains& MaxNumberOfDomains=100&SignatureMethod=HmacSHA1& SignatureVersion=2& Timestamp= T22%3A39%3A30%2 B00%3A00& Version= ie: reconstruct the same string that the SDB client used to sign the request then use rob s secret key to sign it:

20 digest() var crypto = require("crypto"); var digest = function(string, secretkey, type) { var hmac = crypto.createhmac(type, secretkey); hmac.update(string); return hmac.digest('base64'); };

21 Ready to execute an API! Incoming SDB Request Security Key Id Secret Key SimpleDB Database Copy 1 Outgoing SDB Response Server Authenticate Request Error Generate Response Execute API Action Success and/or data/results SimpleDB SimpleDB Database SimpleDB Database Copy Database 2 Copy 2 Copy 2 SimpleDB Database Copy n

22 SimpleDB APIs (Actions) CreateDomain ListDomains DeleteDomain PutAttributes GetAttributes DeleteAttributes Select DomainMetaData (BatchPutAttributes) (BatchDeleteAttributes)

23 Accessing the GT.M Database Accessed via node-mwire TCP-based wire protocol Extension of Redis protocol Adapted redis-node module APIs allow you to set/get/delete/edit Globals

24 GT.M Globals Globals = unit of persistent storage Schema-free Hierarchically structured Sparse Dynamic persistent associative array

25 GT.M Globals A Global has: A name 0, 1 or more subscripts String value globalname[subscript1,subscript2,..subscriptn]=value

26 SDB Domain in Globals CreateDomain AWSAccessKeyId = rob DomainName = books

27 Multiple Domains in Globals name books 1 created domains modified name accounts 2 created MDB rob modified domainindex books 1 accounts 2

28 Creating a new domain (1) increment() 1 name books created domains modified MDB rob domainindex books 1

29 Creating a new domain (2) setglobal() 1 name books created domains modified name accounts 2 created MDB rob modified domainindex books 1 accounts 2

30 Key Node.js async patterns for db I/O Dependent pattern: Can t set the global nodes until the value of the increment() is returned Parallel pattern: Global nodes can be created in parallel No interdependence BUT: Need to know when they re all completed

31 Dependent pattern name books 1 created domains modified MDB rob IncrBy MDB.increment([accessKeyId, 'domains'], 1, function (error, results) { var id = results.value; //.now create the other global nodes inside callback });

32 Dependent pattern name books 1 created domains modified MDB rob MDB.increment([accessKeyId, 'domains'], 1, function (error, results) { var id = results.value; //.now create the other global nodes inside callback });

33 Parallel Pattern (semaphore) var count = 0; MDB.setGlobal([accessKeyId, 'domains', id, 'name'], domainname, function (error, results) { count++; if (count === 4) sendcreatedomainresponse(count, SDB); }); MDB.setGlobal([accessKeyId, 'domains', id, 'created'], now, function (error, results) { count++; if (count === 4) sendcreatedomainresponse(count, SDB); }); MDB.setGlobal([accessKeyId, 'domains', id, 'modified'], now, function (error, results) { count++; if (count === 4) sendcreatedomainresponse(count, SDB); }); MDB.setGlobal([accessKeyId, 'domainindex', nameindex, id], '', function (error, results) { count++; if (count === 4) sendcreatedomainresponse(count, SDB); });

34 New domain nodes created name books 1 created domains modified name accounts 2 created MDB rob modified domainindex books 1 accounts 2

35 Send CreateDomain Response Incoming SDB Request Security Key Id Secret Key SimpleDB Database Copy 1 Outgoing SDB Response Server Authenticate Request Error Generate Response Execute API Action Success and/or data/results SimpleDB SimpleDB Database SimpleDB Database Copy Database 2 Copy 2 Copy 2 SimpleDB Database Copy n

36 CreateDomain Response <?xml version="1.0"?> <CreateDomainResponse xmlns=" <ResponseMetadata> <RequestID>e4e9fa45-f9dc-4e5b-8f0a-777acce6505e</RequestID> <BoxUsage> </BoxUsage> </ResponseMetadata> </CreateDomainResponse> var okresponse = function(sdb) { var nvps = SDB.nvps; var xml = responsestart({action: nvps.action, version: nvps.version}); xml = xml + responseend(nvps.action, SDB.startTime, false); responseheader(200, SDB.response); SDB.response.write(xml); SDB.response.end(); };

37 Node.js Server Response http.createserver(function(request, response) { // numerous call-backs deep: }); response.writehead(status, { "Server": "Amazon SimpleDB", "Content-Type": "text/xml", "Date": datenow.toutcstring()}); response.write('<?xml version="1.0"?>\n'); response.write(xml); response.end(); Entire request/response SDB round-trip completed

38 Demo using Bolso List Domains Create Domain Add an item (row) and some attributes (columns + cells)

39 Node.js Gotchas Async programming is not immediately intuitive! Loops Calling functions that use call-backs inside a for..in loop will go horribly wrong! Understanding closures How externally-defined variables can be used inside call-back functions

40 Example BatchPutAttributes Intuitively a for.. in loop around PutAttributes Had to be serialised Completion of one PutAttributes calls the next Copy state of SDB object and use for..in? var SDBx = SDB; SDBx is a pointer to SDB, not a clone of it!

41 Conclusions node-mdb is now nearly complete Only BatchDeleteAttributes not implemented Other APIs emulate SimpleDB 100% Free Open Source Give it a try! Use mdb.js for examples to build your own Node.js database applications Check out GT.M! Follow me on Twitter Slides: