Hardening cassandra q2_2016
- 1. Securing Cassandra for Compliance (or Paranoia)
- 2. Hi, I'm Nate. @zznate https://www.linkedin.com/in/zznate http://www.slideshare.net/zznate/ Co-Founder, CTO The Last Pickle Cassandra user since 2009 (v0.4) Austin, Texas
- 3. Security presentations can be scary. Here's a cat.
- 4. First, how did we get here and why is securing Cassandra important?
- 5. "Target CEO Gregg Steinhafel Resigns In Data Breach Fallout" http://www.forbes.com/sites/clareoconnor/2014/05/05/target-ceo-gregg-steinhafel-resigns-in-wake-of-data-breach-fallout/ First, how did we get here and why is securing Cassandra important?
- 6. I have your personal information Customers place a lot of trust in technology companies
- 7. LOL! Me too! Sometimes too much.
- 8. Ease of scalability comes with a price
- 9. HA! A bin-packed message format with no source verification!* Ease of scalability comes with a price * <currently reading o.a.c.net.MessageIn#read>
- 10. nmap -Pn -p7000 -oG logs/cass.gnmap 54.88.0.0/14
- 11. I'm publicly discussing your technical shortcomings Then you end up in this situation.
- 12. Meanwhile, at the FCC... We have to require two factor, secure socket transport encryption, something something... ZZZzzzzzzzZZZzz
- 13. We did a regulation! My staffers still print out my email :)
- 14. Why are we doing this again? Sssshhhh. I'm AES'ing... ...even though the traffic never leaves a backplane. Some industries will require node to node SSL
- 15. 1. Encrypting data at rest 2. Encrypting data on the wire 3. Authentication and authorization 4. Management and tooling Focusing our Discussion: Architecture
- 16. 1. Encryption at rest
- 17. No matter what: understand the failure modes
- 18. bit rot, entropy, etc. Horrible things can happen with on disk encryption.
- 19. Don't mind me, I'm just your key server.
- 20. Haha! Later! x What's on this disk again? Shrug.
- 21. ...but you may not have a choice. Because we said "at rest"
- 22. dmcrypt, eCryptFS Open source options:
- 24. DSE Encryption CREATETABLE users ... WITH compression_parameters:sstable_compression = 'Encryptor' and compression_parameters:cipher_algorithm = 'AES/ECB/ PKCS5Padding' and compression_parameters:secret_key_strength = 128;
- 25. DSE Encryption CREATETABLE users ... WITH compression_parameters:sstable_compression = 'Encryptor' and compression_parameters:cipher_algorithm = 'AES/ECB/ PKCS5Padding' and compression_parameters:secret_key_strength = 128; WARNING: commitlog not included* *eCryptFS would work fine for this
- 26. EBS Encryption (a.k.a "not my problem")
- 27. (Looks like this) EBS Encryption (a.k.a "not my problem") http://www.slideshare.net/AmazonWebServices/bdt323-amazon-ebs-cassandra-1-million-writes-per-second See Crowdstrike's presentation on Cassandra GP2 performance (with encryption):
- 28. Maybe Client Side? The Java Driver now has custom codecs which would make this easy to implement https://github.com/datastax/java-driver/tree/3.0/manual/custom_codecs
- 29. Maybe Client Side? The Java Driver now has custom codecs which would make this easy to implement https://github.com/datastax/java-driver/tree/3.0/manual/custom_codecs Column-level encryption!
- 30. New in Cassandra 3.4 (DSE 5.1?): Commitlog Encryption: CASSANDRA-6018 Hint File Encryption: CASSANDRA-11040 https://issues.apache.org/jira/browse/CASSANDRA-6018 https://issues.apache.org/jira/browse/CASSANDRA-11040
- 31. 2. Encryption on the wire
- 32. Because: It is really easy to attack an un-protected cluster
- 33. It takes a single Message to insert an admin account into the system table
- 34. -Dcassandra.write_survey=true How to steal writes in real time:
- 35. The fix is straight forward: node to node encryption and SSL client certificate authentication to cluster traffic
- 36. Awwwwww. The fix is straight forward: node to node encryption and SSL client certificate authentication to cluster traffic
- 37. Awwwwww. The fix is straight forward: node to node encryption and SSL client certificate authentication to cluster traffic Bonus: can be done with NO downtime!!!
- 38. Awwwwww. The fix is straight forward: node to node encryption and SSL client certificate authentication to cluster traffic Bonus: can be done with NO downtime!!! How-to guide: http://thelastpickle.com/blog/2015/09/30/hardening-cassandra-step-by-step-part-1-server-to- server.html
- 39. When you are done it should look like:
- 40. Things to note: Use "dc" or "rack" to limit encryption to connections between racks and data centers
- 41. Thanks for that!! Huzzah! (But AES on modern hardware will not be a bottleneck)
- 42. Things to note: Keystore and key password must match (artifact of JDK X.509 Impl complexity)
- 43. Things to note: 256 bit means export restrictions (requires JCE provider JAR) http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html http://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#importlimits
- 44. Don't forget this part or else... Things to note:
- 45. Hahaha! Now I'm hacking you over SSL. *Still* vulnerable AND you can't see what the attacker is doing.
- 46. Client to Server SSL
- 47. Client to Server SSL (see slides 30 to 35)
- 48. Client to Server SSL (see slides 30 to 35) Now with NO downtime!!! https://issues.apache.org/jira/browse/CASSANDRA-10559 Available in: 2.1.12, 2.2.4, 3.0.0
- 49. Need to Debug SSL? -Djavax.net.debug=ssl http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html
- 50. Certs are hard :( Netflix Lemur: x.509 Certificate Orchestration Framework http://techblog.netflix.com/2015/09/introducing-lemur.html https://github.com/Netflix/lemur
- 51. Certs are hard :( Hashicorp Vault "secures, stores, and tightly controls access to tokens, passwords, certificates, API keys, and other secrets in modern computing. " https://www.vaultproject.io/
- 52. 2. Encryption on the wire But wait! There's more!
- 53. The internode authentication API: BYO identity verification
- 54. Looks like this:
- 55. 3. Authentication and Authorization
- 56. Best practices should not be new to you. user segmentation schema access limitation etc.
- 57. (Everything we did with an RDBMS) Best practices should not be new to you. user segmentation schema access limitation etc.
- 58. Best practices should not be new to you. user segmentation schema access limitation etc. (Everything we did with an RDBMS) New in 2.2: Role-based access control!
- 59. An Example
- 60. An Example
- 61. An Example
- 62. An Example
- 64. An Example
- 65. An Example
- 66. Turning it all on authenticator: PasswordAuthenticator Tip: keep your read-only cqlsh credentials in $HOME/.cassandra/cqlshrc of the system's admin account
- 67. Turning it all on authorizer: CassandraAuthorizer
- 68. Turning it all on role_manager: CassandraRoleManager
- 69. Turning it all on authorizer: CassandraAuthorizer authenticator: PasswordAuthenticator role_manager: CassandraRoleManager WARNING: potential downtime!
- 70. authorizer: CassandraAuthorizer authenticator: PasswordAuthenticator role_manager: CassandraRoleManager Turning it all on WARNING: potential downtime! WARNING: stupid defaults
- 71. authorizer: CassandraAuthorizer authenticator: PasswordAuthenticator role_manager: CassandraRoleManager Turning it all on WARNING: potential downtime! WARNING: stupid defaults TIP: turn these WAY UP: permissions_validity_in_ms roles_validity_in_ms Also: use permissions_update_interval_in_ms for async refresh if needed
- 72. authorizer: CassandraAuthorizer authenticator: PasswordAuthenticator role_manager: CassandraRoleManager Turning it all on WARNING: potential downtime! WARNING: stupid defaults NEW in 3.4:credentials_validity_in_ms* * https://issues.apache.org/jira/browse/CASSANDRA-7715
- 73. Turning it all on authorizer: TransitionalAuthorizer authenticator: TransitionalAuthenticator DSE plugins to avoid downtime
- 74. Turning it all on system.schema_keyspace system.schema_columns system.schema_columnfamilies system.local system.peers These tables have default read permissions for every authenticated user:
- 75. Turning it all on IMPORTANT cassandra.yaml line note: "Please increase system_auth keyspace replication factor if you use this..." Tip: replication factor for the system_auth keyspace should be the same as the number of nodes in the data center
- 76. Turning it all on IMPORTANT cassandra.yaml line note: "Please increase system_auth keyspace replication factor if you use this..." Tip: replication factor for the system_auth keyspace should be the same as the number of nodes in the data center WARNING: stupid defaults* *https://issues.apache.org/jira/browse/CASSANDRA-11340
- 77. 4. Management and tooling
- 78. 4. Management and tooling
- 79. Securing JMX
- 80. nmap -Pn -p7199 -oG logs/cass.gnmap 54.88.0.0/14 Always a few suckers that TL,DR'ed
- 81. Why do I need to secure JMX?
- 85. Securing JMX SSL setup is like node to node and client to server http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html
- 86. Securing JMX JMX Authentication is straightforward and well documented $JAVA_HOME/jre/lib/management/jmxremote.access $JAVA_HOME/jre/lib/management/ jmxremote.password.template http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html
- 87. Securing JMX $JAVA_HOME/jre/lib/management/jmxremote.access $JAVA_HOME/jre/lib/management/ jmxremote.password.template Now you can: nodetool -u admin -pw secret compactionstats http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html JMX Authentication is straightforward and well documented
- 88. Securing JMX $JAVA_HOME/jre/lib/management/jmxremote.access $JAVA_HOME/jre/lib/management/ jmxremote.password.template Now you can: nodetool -u admin -pw secret compactionstats Tip: -pwf option will read the password from a file http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html JMX Authentication is straightforward and well documented
- 89. Securing JMX $JAVA_HOME/jre/lib/management/jmxremote.access $JAVA_HOME/jre/lib/management/ jmxremote.password.template Now you can: nodetool -u admin -pw secret compactionstats JMX Authentication is straightforward and well documented THIS JUST IN!!! RBAC for JMX Authentication and Authorization https://issues.apache.org/jira/browse/CASSANDRA-10091
- 90. Thanks!@zznate