Protecting MySQL network traffic Daniël van Eeden 25 April 2017
Booking.com at a glance Started in 1996; still based in Amsterdam Member of the Priceline Group since 2005 (stock: PCLN) Amazing growth; continuous scaling challenges Online Hotel/Accommodation/Travel Agent (OTA): Over 1.2 million active properties in 227 countries Over 1.2 million room nights reserved daily 40+ languages (website and customer service) Over 13,000 people working in 187 offices in 70 countries We use a lot of MySQL and MariaDB: Thousands (1000s) of servers, ~90% replicating >150 masters: ~30 >50 slaves & ~10 >100 slaves 2
Why protect MySQL network traffic? Protect leaking of authentication data (passwords, etc) Protect leaking of sensitive data (PII, credit card numbers, medical records) Ensure data is not tampered with. Because of regulations Because why not? Are you still using telnet to manage servers?
How? Use SSL! Done!
SSL Support in MySQL MySQL doesn't have SSL support MySQL never had any SSL support MySQL has TLS support.. this is what is called SSL but isn't Supported since 4.0.0 (~2003) For now just assume SSL and TLS are the same
What is NOT protected by TLS Data-at-rest InnoDB and MyISAM data files Binlogs, redo logs, slow query logs Backups Does not protect against a DoS e.g. corrupting traffic Might not protect the query text performance_schema etc. Does not hide the traffic pattern
First steps with TLS 1. Get a certificate 2. Restart MySQL 3. Enable TLS on the client 4. Check if the connection actually uses TLS
Generating the certificate With 5.7 and up: Might already be done by your installation If not use mysql_ssl_rsa_setup For older versions: https://github.com/dveeden/mysslgen Or use the openssl commandline utilities as described in the reference manual on https://dev.mysql.com/doc/mysql/en/creating-ssl-files-using-openssl.html Did you know MySQL Workbench has a SSL Wizard?
Configuration On 5.7+: Place the ca.pem, server-cert.pem and server-key.pem in your datadir. (already the case if you use mysql_ssl_rsa_setup) Or set ssl-ca, ssl-key, ssl-cert in your my.cnf Restart MySQL Enable SSL in your application. You probably want to copy your ca.pem file to your client
Checking your connection 'status' or \s Look for 'Cipher in use' Or check the 'Ssl_cipher' session status.
What if it doesn't work? Check your mysqld.log Check the permissions on the pem files Should be readable for the mysql user Try to connect with --ssl-mode=required Use the OpenSSL commandline tools to see what's in the certificate. Use Wireshark. ERROR 2026 (HY000): SSL connection error: error:00000001:lib(0):func(0):reason(1) ERROR 2026 (HY000): SSL connection error: unknown error number ERROR 2026 (HY000): SSL connection error: SSL certificate validation failure ERROR 2026 (HY000): SSL connection error: SSL_CTX_set_default_verify_paths failed ERROR 2026 (HY000): SSL connection error: protocol version mismatch ERROR 2026 (HY000): --ssl-mode=required option forbids non SSL connections ERROR 2026 (HY000): SSL connection error: Failed to set ciphers to use ERROR 2026 (HY000): SSL connection error: Unable to get certificate ERROR 2026 (HY000): SSL connection error: Unable to get private key
Now let's make it more secure Require the use of TLS on the server Require the use of TLS on the client Enable more security checks Security updates
Make TLS a requirement On a per user basis: ALTER USER foo REQUIRE SSL Undo: ALTER USER foo REQUIRE NONE But what happens if you accidentally create a user? e.g. GRANT on a nonexistent user? Set: sql_mode=no_auto_create_user, On a server level: SET GLOBAL require_secure_transport=on This still allows UNIX socket connections w/o TLS
Issues with full-on TLS Is your monitoring capable of using TLS connections? What about load balancer health checks?
On the client Use --ssl-mode=required The default in 5.7 is PREFERRED Older releases default to DISABLED This only makes a TLS connection a requirement Does not check if issued by a trusted CA Does not check if the hostname matches the cert To do this use VERIFY_CA or VERIFY_IDENTITY On older versions: Use --ssl-ca to allow TLS and enable CA checks Use --ssl-verify-server-cert to do hostname checks. Often not possible to force the use of TLS: this is the BACKRONYM vulnerability Use --ssl-ca=/path/to/ca.pem to specify which CA(s) are trusted.
Client checks The client could do these checks: Is the certificate signed by a trusted CA? Does the CommonName (CN) in the certificate match the hostname we are connecting to? Is the certificate expired?
Certificate Authority validation Validates that the server certificate is signed by one of the CA's present in the specified CA file. Note that a CA file can have multiple CA's There is also a CA path option. The auto generated certificates from mysql_ssl_rsa_setup all have their own CA.
Hostname validation mysql_ssl_rsa_setup generates certificates with CN=MySQL_Server_5.7.18_Auto_Generated_Server_Certificate So generate the certificates manually if want this to match your hostname A certificate can have a list of hostnames in SubjectAltName MySQL doesn't check those... Bug #68052 So if you use a virtual-ip, cname, etc. it might be difficult to match this. What if your clients connect on a CNAME and your replicas connect on the hostname? You can't have both!
Security updates I reported a few issues to Oracle. CVE-2017-3590 for Connector/Python CVE-2017-3469 for MySQL Workbench CVE-2017-3467 for libmysqlclient Those are fixed. See the Oracle Critical Patch Update for details. But if you care about security you should follow the release notes and Critical Patch Update anyways...
What library does MySQL use? Community Edition: YaSSL Because GPL and the OpenSSL license are not really compatible This library is maintained by WolfSSL This not CyaSSL/WolfSSL WolfSSL made a patch to include WolfSSL in MySQL 5.6.30 (https://github.com/wolfssl/mysql-patch) Enterprise Edition: OpenSSL If you build MySQL yourself: you can compile against either of them.
Why not TLS? Because it is SLOW! Because we trust our network! Because we encrypt with: The application (store encrypted data) SSH (Also works great with Workbench) VPN Because we want to inspect our network traffic! Wireshark can decrypt it if you hand over your private key. Some ciphers require you to somehow extract session keys.
How slow is slow? Overhead in milliseconds for setting up a TLS connection on localhost with TCP. Client: go 5.7 is faster than 5.6 OpenSSL is faster than YaSSL Using TLS tickets (OpenSSL only) helps Best case: 0.99ms (5.7 OpenSSL w/ tickets) vs. 0.60ms (no TLS) TLS does need more roundtrips, but this will change with TLS 1.3 OpenSSL performs better because it uses AVX2 and AES-NI
Bulk transfer performance Easy to test: mysqldump with and without TLS Different ciphers do make a difference. mysqldump performance with MySQL 5.6.35 (YaSSL) No TLS 4.5s TLS Default 10.4s RC4-MD5 7.1s DES-CBC3-SHA 23.2s
Monitoring Monitor the Expiry of certificates Not just the certificate on disk, also the one in memory. Use TLS for your monitoring on 5.6 and earlier, otherwise you might not see the status vars Performance schema can show you the ciphers and TLS versions in use by all connections Using SYS is even easier: SELECT * FROM session_ssl_status
Client certificates This allows mutual authentication Often used together with a password You might want to use REQUIRE SUBJECT or REQUIRE ISSUER on accounts. At least use REQUIRE X509 instead of REQUIRE SSL
Replication Use CHANGE MASTER TO MASTER_SSL=1, etc Think about what happens if your certificate(s) expire Does the hostname match the certificate?
Changing certificates Needs restart Moving slaves around might not work until you restart.. Same for a switchover.
CRL and OCSP Only possible with OpenSSL Does not auto download the CRL from the distribution point Does not use OCSP Basically restart MySQL every time your CRL changes.. which is not practical
Where to get your certificate? Official CA? Internal CA? Self signed?
TLS handshake with MySQL server helo with ssl flag set 'empty' login packet with ssl flag set Start SSL handshake Basically STARTTLS-ish SSL and non-ssl on the same port
Protection of authentication data native password with nonce sha256 password with RSA keys or TLS cleartext plugin
TLS ciphers Possible to set restriction on Server and Client How are you going manage and maintain that? 'REQUIRE cipher' also requires client certificates One practical use case would be to use a faster cipher for mysqldump Might help with compliance 5.7.10 already places more strict requirements on the list of ciphers
TLS versions Can be limited on the server and client Note that YaSSL only has TLS 1.0 and TLS 1.1 support Minimum is TLS 1.0
What about MariaDB? Doesn't use --ssl-mode Does have good TLS support MariaDB Connector/C has support for fingerprint verification password protected private keys 19 Open MDEV's tagged with SSL
Connector support Works for C, C++, Python (multiple), Perl, Java, ODBC, Go, etc The Go MySQL driver lets you specify a TLS Config, which is really flexible. Do update your Connector.. Many connectors did have security updates related to TLS.
Don't forget these MySQL Cluster (NDB) communication within the cluster Galera communication Sending backups to a central location (xbstream etc) Network traffic for iscsi, FCP, NFS
Future TLSv1.3 with 0-RTT WolfSSL?
Oh, and Booking.com is hiring! Almost any role: MySQL Engineer / DBA System Administrator System Engineer Site Reliability Engineer Developer Designer Technical Team Lead Product Owner Data Scientist And many more https://workingatbooking.com/ 39
Thank you! All references to Booking.com", including any mention of us, we and our refer to Booking.com BV, the company behind Booking.com
Title Item Item