HBase Security. Works in Progress Andrew Purtell, an Intel HBase guy

Transcription

1 HBase Security Works in Progress Andrew Purtell, an Intel HBase guy

2 Outline Project Rhino Extending the HBase ACL model to per-cell granularity Introducing transparent encryption I don t always need encryption, but when I do

3 Project Rhino Project Rhino is An open source effort Available as patches on Apache JIRAs and on Github under ASL v2.0 Enhance the existing data protection capabilities of the Apache Hadoop ecosystem Address the security and compliance challenges of a broad number of use cases: Financial services Health care Government Defense Corporate environments containing sensitive and legally protected data

4 HBase Cell ACLs Goals Support optional authorization on a per-cell granularity Leverage the HBase authorization model and machinery Extend the AccessController coprocessor Use existing API facilities to transmit per-cell ACLs (operation attributes) Compatible with existing installations and code Easy transition for users

5 HBase Cell ACLs Current HBase security / AccessController(0.92+)

6 HBase Cell ACLs AccessController with per-cell ACL support

7 HBase Cell ACLs How They Work Users can supply per cell ACLs in operation attributes We treat ACLs on a KV as timestamped like the KV itself This allows simple and straightforward evolution of security policy over time without requiring expensive updates To actually change the ACL on an existing cell, the cell must be replaced by a new Put to its exact location We require mutations to have covering permission The union of the user s table perms, CF perms, and perms in the most recent visible [1] version, if the value already exists, all must allow the pending mutation in order for it to be applied For Deletes, in addition, all visible prior versions covered by the Delete must allow the Delete 1. Visibleis defined here as not covered already by a committed delete marker

8 HBase Cell ACLs How They Work Gets and Scans We inject a filter that determines for each encountered value if it is visible to the user, as KVs are streamed through Mutations (Except Delete) We inject an internal scanner bounded on the parameters of the mutation (row, column, qualifier, timestamp (if provided)) to seek for the most recent visible existing value within bounds; if one is found, any ACLs are checked If there are table or CF permissions granting access already, we can early out and do not require additional IO Deletes As with mutations, but we must look for all visible values that might be covered by the tombstone

9 Implementation Alternatives and Trade Offs Option 1 Tags Extend KeyValue with tags Use tags as ACL storage Clients may see ACLs of KVs they are authorized for, unless stripped (depends on tag implementation) Piggybacks on I/O in progress Option 2 ACLCF Store ACLs in a hidden shadow column family in each region No changes to core required at all ACLs are hidden from the clients Introduces additional I/O

10 Implementation Option Tags Reads Minicluster load test Hadoop SNAPSHOT + crypto HBase 0.95-SNAPSHOT (just before branch)

11 Implementation Option ACLCF Reads Minicluster load test Hadoop SNAPSHOT + crypto HBase 0.95-SNAPSHOT (just before branch)

12 Implementation Option Tags Writes Minicluster load test Hadoop SNAPSHOT + crypto HBase 0.95-SNAPSHOT (just before branch)

13 Implementation Option ACLCF Writes Minicluster load test Hadoop SNAPSHOT + crypto HBase 0.95-SNAPSHOT (just before branch)

14 HBase Cell ACL - Future Work Added IO pressure due to the additional column family is obvious We can avoid the extra CF overhead with a mechanism for inline storage of permissions metadata with the KV itself = Contribute to the cell tags work in trunk

15 Hadoop Common Crypto Framework Extends the CompressionCodec Establishes common API abstractions that can be shared by all crypto codec implementations Provides a foundation for other components in Hadoop such as MapReduce or HBase to support encryption features

16 Hadoop AES-NI Crypto Codec Advanced Encryption Standard (AES) The specification of a cryptographic algorithm adopted by the U.S. government and used extensively worldwide AES-NI An extension to the x86 instruction set architecture for microprocessors from Intel and AMD, proposed by Intel Dramatically accelerates AES up to 10x compared to an optimized software-only implementation AES-NI Hadoop crypto codec Uses OpenSSL 1.0.1c+ as engine Powers Hadoop and HBase encryption

17 HBase Transparent Encryption Goals Protect against any leakage of data at rest (keys and values) Be consistent with best practices Tiered key architecture Transparent encryption of sensitive application columns Built-in key management Flexible and non-intrusive key rotation Hardware security module integration Scope HBase on disk structures HFiles, the on disk storage of HBase data One or more HFiles per column family The write-ahead-log (WAL) One per RegionServer A stream, not a block based format

18 HBase Transparent Encryption

19 HBase Transparent Encryption Symmetric block encryption introduces an overhead roughly on par with GZIP compression for reads, and half that as for writes Read throughput can be improved by compressing data with SNAPPY first Bottom line: We can mitigate the costs of employing encryption if it is required, and further mitigation will be continuing work

20 HFile Encryption Read Throughput Hadoop SNAPSHOT + crypto HBase 0.95-SNAPSHOT GZ AESNI+SNAPPY

21 HFile Encryption Write Throughput Hadoop SNAPSHOT + crypto HBase 0.95-SNAPSHOT GZ AESNI+SNAPPY

22 WAL Encryption Microbenchmark Hadoop SNAPSHOT + crypto HBase 0.95-SNAPSHOT Ideal Target

23 Encryption Future Work Further performance tests and code optimizations are planned The code is correct, now look at being clever with buffers We employ AES in CTR mode to enable future work on parallel decryption of HFile blocks using multiple (hardware) threads The WAL is both latency sensitive and has a limited lifespan, we will consider a reduced round AES variant (14 -> 8) after an analysis of the risk/reward tradeoff

24 End Questions?