Database extensions for fun and profit. Andrew Dalke Andrew Dalke Scientific, AB Gothenburg, Sweden

Size: px

Start display at page:

Download "Database extensions for fun and profit. Andrew Dalke Andrew Dalke Scientific, AB Gothenburg, Sweden"

Lilian Pearson
6 years ago
Views:

1 Database extensions for fun and profit Andrew Dalke Andrew Dalke Scientific, AB Gothenburg, Sweden

2 Set the Wayback Machine to 1997!

3 Relational databases - strings and numbers - SQL - clients written in any language Object databases - complex data types - no standard query language - single client language (Smalltalk) Object-relational databases

4 Object-Relational database Domain-specific database extensions Select cid From structures Where smarts_match(smiles, "c1ccccc1o") Create Function tanimoto(bit, bit) Returns Real As 'Select nbits_set($1 & $2)::real / (nbits_set($1) + nbits_set($2) - nbits_set($1 & $2))::real;' Language SQL; Second example from TJ O Donnell

5 Uses PostgreSQL Teaches SQL for queries, constraints and triggers. Examples schemas for LIMS and PubChem substructure matches, fingerprints, 3D and reactions PostgreSQL extensions using C, pgsql, Perl, Python Client examples CHORD database

6 Oracle Data Cartridge Informix Data Blade

7 And the winner is...? MySQL Good enough Fast, free, well-understood, widely used, scalable Used by YouTube, Flickr, Digg, Wikipedia and other popular sites Supported by every web application framework. (On 20 April, Oracle agreed to buy Sun for US $7.4 billion. On 16 Jan 2008, Sun acquired MySQL for US $1 billion.)

8 Client Scenario - Internal web application - Several data sets, each O(10,000) compounds - Search based on simple properties (MW, source) and chemistry (similarity, substructure) - Other in-house tools based on OEChem (aromaticity) - In-house similarity fingerprints, available as a library - Corporate database using Oracle, controlled by IS - Okay to use MySQL; existing in-house experience I asked: no other RDBMS was acceptable - Did not want to pay annual support fees

9 MySQL Good enough Supports user-defined functions and aggregates. Does not support user-defined data types. Select cid From structures Where smarts_match(smiles, "c1ccccc1o") Order By tanimoto(fp, similarity_fp("c1ccccc1o")) Limit 25

10 valid_smiles(smiles) Return 1 if it was valid, otherwise 0

11 mysql_oechem.cc 1/3 #include <mysql.h> #include <oechem.h> using namespace OEChem; // OEChem requires C++ but MySQL can only call C functions // Tell the C++ compiler to generate a C function interface extern "C" { // MySQL's security model requires that the function and // at least one of "*_init" or "*_deinit" must be defined. my_bool valid_smiles_init(udf_init *initid, UDF_ARGS *args, char *message); long long valid_smiles(udf_init *initid, UDF_ARGS *args, char *is_null, char *error); void valid_smiles_deinit(udf_init *initid); Different prototypes if the function returns a string or a double.

12 mysql_oechem.cc 2/3 // This is called once, at the start of a table search my_bool valid_smiles_init(udf_init *initid, UDF_ARGS *args, char *message) { if (args->arg_count!= 1 args->arg_type[0]!= STRING_RESULT) { strcpy(message, "valid_smiles takes a single SMILES string"); return 1; return 0; // This is called once, at the end of a table search void valid_smiles_deinit(udf_init *initid) { // I don't need to do anything.

13 mysql_oechem.cc 3/3 long long valid_smiles(udf_init *initid, UDF_ARGS *args, char *is_null, char *error) { OEGraphMol mol; // Get the SMILES string and its length. const char *smiles = args->args[0]; int smiles_len = args->lengths[0]; if (smiles == NULL) { // This row contains a SQL NULL. It's not valid. return 0; // The MySQL string might not be NUL-terminated. // OEParseSmiles takes either a 'char *' or a 'string&' // but not a (char *, int length). Convert to a string. std::string smiles_s(smiles, smiles_len); // Parse the string and return its result return OEParseSmiles(mol, smiles_s);

14 How to compile g++ -Wall -O3 -I/usr/local/openeye/toolkits/include \ `/usr/local/mysql/bin/mysql_config --cflags` \ -I. -c -o mysql_oechem.o mysql_oechem.cc Linux shared library: g++ -shared -o mysql_oechem.so mysql_oechem.o \ -L/usr/local/openeye/toolkits/lib \ -loeiupac -loechem -loesystem -loeplatform -lz Mac shared library: g++ -bundle -bundle_loader /usr/local/mysql/bin/mysqld \ -o mysql_oechem.so mysql_oechem.o \ -L/usr/local/openeye/toolkits/lib \ -loeiupac -loechem -loesystem -loeplatform -lz The result is the shared library mysql_oechem.so

15 How to install In MySQL 4: in the server s LD_LIBRARY_PATH In MySQL 5: in the server s plug-in directory % mysql Welcome to the MySQL monitor. Commands end with ; or \g.... mysql> SHOW VARIABLES LIKE 'plugin_dir'; Variable_name Value plugin_dir /usr/local/mysql osx10.4-i686/lib/plugin row in set (0.90 sec) mysql> cp mysql_oechem.so /usr/local/mysql osx10.4-i686/lib/plugin

16 mysqld needs OEChem settings Make sure that its LD_LIBRARY_PATH includes the OEChem shared libraries and that OE_DIR points to the OEChem directory with the license file.

17 Tell MySQL to load the new functions mysql> Select valid_smiles("c"); ERROR 1305 (42000): FUNCTION valid_smiles does not exist mysql> Create Function valid_smiles Returns INT SONAME "mysql_oechem.so"; Query OK, 0 rows affected (0.03 sec) mysql> Select valid_smiles("c"); valid_smiles("c") row in set (0.01 sec) Can be INT, STRING or FLOAT mysql> Select valid_smiles("q"); valid_smiles("q") row in set (0.00 sec)

18 Which functions are loaded? Available from the special mysql.func table. mysql> Select * From mysql.func; name ret dl type valid_smiles 2 mysql_oechem.so function row in set (0.00 sec)

19 Create a new database using the MySQL shell mysql> Create Database pubchem; mysql> Use Database pubchem; mysql> Create Table Compounds ( -> cid INTEGER PRIMARY KEY, -> mw FLOAT, -> smiles TEXT); Query OK, 0 rows affected (0.76 sec)

20 import MySQLdb from openeye.oechem import * Populate the database using a Python client db = MySQLdb.connect(user="skroob", passwd="12345", db="pubchem") cur = db.cursor() ifs = oemolistream() if not ifs.open("/users/dalke/databases/compound_ _ sdf.gz"): raise IOError("Cannot open file") for mol in ifs.getoegraphmols(): cid = OEGetSDData(mol, "PUBCHEM_COMPOUND_CID") mw = OEGetSDData(mol, "PUBCHEM_EXACT_MASS") if not cid or not mw: # any missing values? continue OEAssignAromaticFlags(mol) smiles = OECreateCanSmiString(mol) cur.execute(""" INSERT INTO Compounds (cid, mw, smiles) VALUES (%s, %s, %s)""", (cid, mw, smiles)) cur.execute("select count(*) FROM Compounds") n = cur.fetchone()[0] print "Database table contains", n, "compounds" 24,983 compounds 19s to load (14s sending 1000 records/insert)

21 Searching for invalid SMILES mysql> Select cid, smiles From Compounds Where valid_smiles(smiles) = 0; Empty set (1.08 sec) mysql> Insert Into Compounds (cid, mw, smiles) Values (12345, 0.0, "Q"); Query OK, 1 row affected (0.00 sec) mysql> Select cid, smiles From Compounds Where valid_smiles(smiles) = 0; cid smiles Q row in set (0.97 sec) mysql> Best search time after 5 tests was 0.94 seconds

22 Need for Speed! Parsing 27,000/sec is great. I want it faster. Each row test creates a new OEGraphMol. What about reusing the same OEGraphMol, calling Clear() each time?

23 Store arbitrary data in initid->ptr my_bool valid_smiles_init(udf_init *initid, UDF_ARGS *args, char *message) { if (args->arg_count!= 1 args->arg_type[0]!= STRING_RESULT) { strcpy(message, "valid_smiles takes a single SMILES string"); return 1; // Create a molecule once for the entire query initid->ptr = reinterpret_cast<char *>(new OEGraphMol); return 0; void valid_smiles_deinit(udf_init *initid) { delete reinterpret_cast<oegraphmol *>(initid->ptr);

24 Reuse the OEGraphMol from initid->ptr long long valid_smiles(udf_init *initid, UDF_ARGS *args, char *is_null, char *error) { OEGraphMol *mol = reinterpret_cast<oegraphmol *>(initid->ptr); const char *smiles = args->args[0]; int smiles_len = args->lengths[0]; if (smiles == NULL) { return 0; std::string smiles_s(smiles, smiles_len); // Remember, each SMILES adds to the existing molecule mol->clear(); return OEParseSmiles(*mol, smiles_s); Search time goes to 0.87s, or 29,000/second; 7% faster. (Before going further: Delete From Structures Where cid = 12345; )

25 SMARTS searches Select count(*) from Compounds Where smarts_match(smiles, "c1ccccc1o"); #include <mysql.h> #include <oechem.h> using namespace OEChem; extern "C" { my_bool smarts_match_init(udf_init *initid, UDF_ARGS *args, char *message); long long smarts_match(udf_init *initid, UDF_ARGS *args, char *is_null, char *error); void smarts_match_deinit(udf_init *initid);

26 my_bool smarts_match_init(udf_init *initid, UDF_ARGS *args, char *message) { if (args->arg_count!= 2) { strcpy(message, "smarts_match take two arguments (smiles, smarts)"); return 1; if (args->arg_type[0]!= STRING_RESULT) { strcpy(message, "smarts_match: SMILES parameter must be a string"); return 1; if (args->arg_type[1]!= STRING_RESULT) { strcpy(message, "smarts_match: SMARTS parameter must be a string"); return 1; return 0; void smarts_match_deinit(udf_init *initid) {

27 long long smarts_match(udf_init *initid, UDF_ARGS *args, char *is_null, char *error) { // Get the SMILES and SMARTS strings const char *smiles = args->args[0]; int smiles_len = args->lengths[0]; const char *smarts = args->args[1]; int smarts_len = args->lengths[1]; // Check for NULLs and convert to strings if (smiles == NULL smarts == NULL) { return 0; std::string smiles_s(smiles, smiles_len); std::string smarts_s(smarts, smarts_len); OEGraphMol mol; OESubSearch pat; // If the string isn't a valid SMILES or SMARTS // then regard it as an error. if (!OEParseSmiles(mol, smiles_s)!pat.init(smarts_s.c_str(), true)) { *error = 1; return 0; return pat.singlematch(mol)!= 0;

28 Does it work? mysql> select smarts_match("c#n", "*#*"); smarts_match("c#n", "*#*") row in set (0.00 sec) mysql> select smarts_match("c#n", "O#*"); smarts_match("c#n", "O#*") row in set (0.00 sec) mysql> select count(*) from Compounds where smarts_match(smiles, "c1ccccc1o"); count(*) row in set (2.90 sec) 8,600 SMARTS tests/second

29 I feel the need the need for speed! - Reuse the OEGraphMol - The SMARTS pattern is constant; compile it once? The *_init function knows about all parameters which are constant values my_bool smarts_match_init(udf_init *initid, UDF_ARGS *args, char *message) {... verify that there are 2 inputs and both are strings... const char *smarts = args->args[1]; if (smarts) {... this is constant for all rows... else {... the SMARTS may be different for each row return 0;

30 struct SmartsMatchState { OEGraphMol mol; int pattern_is_constant; OESubSearch pattern; ; my_bool smarts_match_init(udf_init *initid, UDF_ARGS *args, char *message) {... verify that the two input fields are strings... SmartsMatchState *state = new SmartsMatchState; const char *smarts = args->args[1]; if (smarts) { // use the same SMARTS for every match int smarts_len = args->lengths[1]; std::string smarts_s(smarts, smarts_len); if (!state->pattern.init(smarts_s.c_str(), true)) { delete state; strcpy(message, "Could not parse SMARTS"); return 1; state->pattern_is_constant = 1; else { state->pattern_is_constant = 0; initid->ptr = reinterpret_cast<char *>(state); return 0; void smarts_match_deinit(udf_init *initid) { delete reinterpret_cast<smartsmatchstate *>(initid->ptr);

31 long long smarts_match(udf_init *initid, UDF_ARGS *args, char *is_null, char *error) { SmartsMatchState *state = reinterpret_cast<smartsmatchstate *>(initid->ptr); // Get the SMILES strings const char *smiles = args->args[0]; int smiles_len = args->lengths[0]; std::string smiles_s(smiles, smiles_len); state->mol.clear(); if (!OEParseSmiles(state->mol, smiles_s)) { *error = 1; return 0; if (!state->pattern_is_constant) { // Compile the new SMARTS string const char *smarts = args->args[1]; int smarts_len = args->lengths[1]; std::string smarts_s(smarts, smarts_len); if (!state->pattern.init(smarts_s.c_str(), true)) { *error = 1; return 0; return state->pattern.singlematch(state->mol)!= 0;

32 Is it faster? mysql> Select count(*) from Compounds -> Where smarts_match(smiles, "c1ccccc1o"); count(*) row in set (1.11 sec) (was 2.90 sec) 22,500 SMARTS tests/sec (was 8,600) About 2.6x faster

33 Going to Plaid

34 Next trick up our collective sleeves... Fingerprint Screens My client wanted substructure tests, not SMARTS matching. (SMILES canonical SMILES treated as SMARTS is good enough) - Compute and store substructure fingerprints for each compound in the database. - Compound the query substructure fingerprint - Screen out targets where target_fp does not contain query_fp - Do the SMARTS test on the remaining targets Screening seems to remove about 85% of the targets. 6-7x faster

35 Which fingerprints? Substructure keys MACCS SMARTS from TJ O Donnell and from RDKit CACTVS definitions from PubChem Hash keys Implementation in RDKit and OpenBabel I implemented a subset of the 881 CACTVS keys.

36 Assume a substructure_fp(smiles) function Alter Table Compounds Add fp Text; Update Compounds Set fp = substructure_fp(smiles); Depends on how you implement fingerprints Select count(*) from Compounds Where fp_contains(fp, substructure_fp("c1ccccc1o")) And smarts_match(smiles, "c1ccccc1o"); substructure_fp(fixed_smarts) is common. The *_init function must check for it and cache the result.

37 Fingerprint implementations PostgreSQL has a built-in bitset data type (See TJ s book) Create Function tanimoto(bit, bit) Returns Real As 'Select nbits_set($1 & $2)::real / (nbits_set($1) + nbits_set($2) - nbits_set($1 & $2))::real;' Language SQL;

38 Fingerprint implementations PostgreSQL has a built-in bitset data type Use multiple integer fields Create Table Compounds ( cid INTEGER PRIMARY KEY, mw FLOAT, smiles TEXT, fp1 INTEGER, fp2 INTEGER, fp3 INTEGER, fp4 INTEGER,... fp28 INTEGER); Select cid from Compounds Where (qfp1 & fp1 = qfp1 and qfp2 & fp2 = qfp2 and qfp3 & fp3 = qfp3 and... qfp28 & fp28 = qfp28); Need bit integers to store 881 bits

39 Fingerprint implementations PostgreSQL has a built-in bitset data type Use multiple integer fields Store the fingerprints as raw bytes Text fields store characters, not bytes! Unicode encoding problems. Could use a blob, but Python libraries access them through file-like interface.

40 Fingerprint implementations PostgreSQL has a built-in bitset data type Use multiple integer fields Store the fingerprints as raw bytes Encode the fingerprint as characters I used a hex encoding: 881 bits 222 hex chars CDE00 Requires twice as much memory

41 substructure_fp, fp_contains, and tanimoto work in hex space mysql> Select count(*) from Compounds -> Where fp_contains(fp, substructure_fp("c1ccccc1o")); count(*) row in set (0.20 sec) mysql> Select count(*) from Compounds -> Where fp_contains(fp, substructure_fp("c1ccc(c)cc1o")); count(*) row in set (0.19 sec) 130,000 tests per second

42 Estimated average performance (Are there any good benchmark data sets?) compounds (1.11s full search)*20% + (0.19s screening) 60,000 compounds / sec 6 minutes to search PubChem (assuming you had ~7GB of memory)

43 Availability My client owns the code I wrote for them. I reimplemented the plugin, to be released soon. It will be announced on my writings blog at I ll include a section on using relational databases at my training course, next week in Leipzig:

44 Conclusions Databases are commodities; what you do with data is not. Get TJ s book to learn how to use databases in chemistry Modern machines + OpenEye = fast If only OEParseSmiles and OESubSearch.Init took: (char *s, int s_len)... A hash fingerprint for OEMols would be so useful. Why pay a lot of money when DIY (or DIWithAConsultant) is so much fun?

45 SQLite In-memory relational database. Comes with recent Pythons. Used in Apple s Mail.app and other applications. Supports user-defined extensions.

Migrating by using User Defined Functions. Jan Kneschke

Migrating by using User Defined Functions Jan Kneschke Consulting, MySQL AB jan@mysql.com MySQL UC 2005 Santa Clara 1 Overview User Defined Functions (UDF) Use of UDFs Writing UDFs Using UDFs Limitations