LA Cassandra Day 2015 - Testing Cassandra
1 like•1,195 views
The document discusses testing Cassandra applications using Stubbed Cassandra, a tool that allows priming Cassandra to return specific responses and errors to integration tests. It provides an overview of Stubbed Cassandra's features like starting an embedded server, priming queries and prepared statements, verifying query executions, and testing error scenarios. Examples are given of how to test different behaviors like retries, timeouts, and coordinator issues. The speaker encourages helping with the open source project and answers questions.
LA Cassandra Day 2015 - Testing Cassandra
- 1. ©2013 DataStax Confidential. Do not distribute without consent. @chbatey Christopher Batey Technical Evangelist for Apache Cassandra LA 2015: Testing Cassandra applications
- 2. @chbatey Who am I? • Based in London • Technical Evangelist for Apache Cassandra •Work on Stubbed Cassandra •Help out Apache Cassandra users • Previous: Cassandra backed apps at BSkyB
- 3. @chbatey Agenda • Cassandra failure scenarios • Developer tools • Stubbed Cassandra
- 7. @chbatey Read timeout Application C R1 R2 R3 C=QUROUM Replication factor: 3 timeout timeout Read timeout
- 8. @chbatey Read timeout • Received acknowledgements • Required acknowledgements • Consistency level • wasDataReceived
- 9. @chbatey Write timeout Application C R1 R2 R3 C=QUROUM Replication factor: 3 timeout timeout Write timeout
- 10. @chbatey Retrying writes • Cassandra does not roll back!
- 11. @chbatey Write timeout • Received acknowledgements • Required acknowledgements • Consistency level • CAS and Batches are more complicated: - WriteType: SIMPLE, BATCH, UNLOGGED_BATCH, BATCH_LOG, CAS
- 12. @chbatey Batches • BATCH_LOG - Timed out waiting for batch log replicas - Retry if you like • BATCH - Written to batch log but timed out waiting for actual replica - Will eventually be committed (serial read) • UNLOGGED_BATCH - Unknown - retry if you like
- 13. @chbatey Idempotent writes • All writes are idempotent with the following exceptions: - Counters - lists
- 14. @chbatey Unavailable Application C R1 R2 R3 C=QUROUM Replication factor: 3 Unavailable
- 15. @chbatey Unavailable • Alive Replicas • Required Replicas • Consistency
- 16. @chbatey Coordinator issue Application C R1 R2 R3 C=QUROUM Replication factor: 3 ??? Set a socket read timeout!
- 17. @chbatey Unit & Integration testing • Requirements: Quick to run, deterministic, not brittle!! • Integration tests against an embedded Cassandra? • cassandra-unit • Integration tests against an external Cassandra running on developer / CI machines • CCM • Regular install • Docker • Vagrant • Mocking the driver
- 19. @chbatey How do this if it was a HTTP service? • Release it - great book • Wiremock - mocking HTTP services • Saboteur - adding network latency If you want to build a fault tolerant application you better test faults!
- 20. @chbatey Stubbed Cassandra Application Acceptance test prime on admin port (REST) verification on admin port Admin endpoints Native protocol
- 21. @chbatey Two sides of Scassandra • Java -Acts as a unit/integration testing library -90% of effort on this side • Standalone -Runs as a separate process
- 22. @chbatey Java Client • Embeds Stubbed Cassandra in a Java library - Start / stop the server - Prime the server to return rows and/or errors - Verify exactly the queries your application has executed
- 23. @chbatey Starting / Stopping @ClassRule public static final ScassandraServerRule SCASSANDRA = new ScassandraServerRule(); Scassandra scassandra = ScassandraFactory.createServer(); scassandra.start(); PrimingClient primingClient = scassandra.primingClient(); ActivityClient activityClient = scassandra.activityClient();
- 24. @chbatey Activity Client • Query ‣ Query text ‣ Consistency • PrepreparedStatementExecution ‣ Prepared statement text ‣ Bound variables public List<Query> retrieveQueries(); public List<PreparedStatementExecution> retrievePreparedStatementExecutions() public List<Connection> retrieveConnections(); public void clearAllRecordedActivity() public void clearConnections(); public void clearQueries(); public void clearPreparedStatementExecutions();
- 25. @chbatey Priming Client • PrimingRequest ‣ Either a Query or PreparedStatement ‣ Query text or QueryPattern (regex) ‣ Consistency (default all) ‣ Result (success, read timeout, unavailable etc) ‣ Rows for successful response ‣ Column types for rows ‣ Variable types for prepared statements public void prime(PrimingRequest prime) public List<PrimingRequest> retrievePreparedPrimes() public List<PrimingRequest> retrieveQueryPrimes() public void clearAllPrimes() public void clearQueryPrimes() public void clearPreparedPrimes()
- 26. @chbatey Example time public interface PersonDao { void connect(); void disconnect(); List<Person> retrievePeople(); List<Person> retrievePeopleByName(String firstName); void storePerson(Person person); }
- 27. @chbatey Testing the connect method @Test public void shouldConnectToCassandraWhenConnectCalled() { //given activityClient.clearConnections(); //when underTest.connect(); //then assertTrue("Expected at least one connection", activityClient.retrieveConnections().size() > 0); }
- 28. @chbatey Testing retrieve public List<Person> retrieveNames() { ResultSet result; try { Statement statement = new SimpleStatement("select * from person"); statement.setConsistencyLevel(ConsistencyLevel.QUORUM); result = session.execute(statement); } catch (ReadTimeoutException e) { throw new UnableToRetrievePeopleException(); } List<Person> people = new ArrayList<>(); for (Row row : result) { people.add(new Person(row.getString("first_name"), row.getInt("age"))); } return people; }
- 29. @chbatey Test the query + consistency @Test public void testQueryIssuedWithCorrectConsistencyUsingMatcher() { //given Query expectedQuery = Query.builder() .withQuery("select * from person") .withConsistency("QUORUM").build(); //when underTest.retrievePeople(); //then assertThat(activityClient.retrieveQueries(), containsQuery(expectedQuery)); } Hamcrest matcher
- 30. @chbatey Testing behaviour @Test public void testRetrievingOfNames() throws Exception { // given Map<String, ?> row = ImmutableMap.of( "first_name", "Chris", "last_name", "Batey", "age", 29); primingClient.prime(PrimingRequest.queryBuilder() .withQuery("select * from person") .withColumnTypes(column("age", PrimitiveType.INT)) .withRows(row) .build()); //when List<Person> names = underTest.retrievePeople(); //then assertEquals(1, names.size()); assertEquals(“Chris Batey", names.get(0).getName()); } Each Cassandra Row == Java map Tell Scassandra about your schem
- 31. @chbatey Testing errors @Test(expected = UnableToRetrievePeopleException.class) public void testHandlingOfReadRequestTimeout() throws Exception { // given PrimingRequest primeReadRequestTimeout = PrimingRequest.queryBuilder() .withQuery("select * from person") .withResult(Result.read_request_timeout) .build(); primingClient.prime(primeReadRequestTimeout); //when underTest.retrievePeople(); //then } Expecting custom exception
- 32. @chbatey Testing retries @Override public RetryDecision onReadTimeout(Statement statement, ConsistencyLevel cl, int requiredResponses, int receivedResponses, boolean dataRetrieved, int nbRetry) { if (nbRetry < configuredRetries) { return RetryDecision.retry(cl); } else { return RetryDecision.rethrow(); } } @Override public RetryDecision onWriteTimeout(Statement s, ConsistencyLevel cl, WriteType wt, int requiredAcks, int receivedAcks, int nbRetry) { return DefaultRetryPolicy.INSTANCE.onWriteTimeout(s, cl, wt, receivedAcks, receivedAcks, nbRetry); } @Override public RetryDecision onUnavailable(Statement statement, ConsistencyLevel cl, int requiredReplica, int aliveReplica, int nbRetry) { return DefaultRetryPolicy.INSTANCE.onUnavailable(statement, cl, requiredReplica, aliveReplica, nbRetry); }
- 33. @chbatey Testing retries @Test public void testRetriesConfiguredNumberOfTimes() throws Exception { PrimingRequest readTimeoutPrime = PrimingRequest.queryBuilder() .withQuery("select * from person") .withResult(Result.read_request_timeout) .build(); primingClient.prime(readTimeoutPrime); try { underTest.retrievePeople(); } catch (UnableToRetrievePeopleException e) { } assertEquals(CONFIGURED_RETRIES + 1, activityClient.retrieveQueries().size()); }
- 34. @chbatey Coordinator issue Application C R1 R2 R3 C=QUROUM Replication factor: 3 ???
- 35. @chbatey Testing slow connection @Test(expected = UnableToSavePersonException.class) public void testThatSlowQueriesTimeout() throws Exception { // given PrimingRequest preparedStatementPrime = PrimingRequest.preparedStatementBuilder() .withQueryPattern("insert into person.*") .withVariableTypes(VARCHAR, INT, list(TIMESTAMP)) .withFixedDelay(1000) .build(); primingClient.prime(preparedStatementPrime); underTest.connect(); underTest.storePerson(new Person("Christopher", 29, Collections.emptyList())); } Delays the response by 1000ms Expect a custom exception
- 36. @chbatey Testing slow connection SocketOptions socketOptions = new SocketOptions(); socketOptions.setReadTimeoutMillis(500); cluster = Cluster.builder() .addContactPoint("localhost") .withPort(port) .withRetryPolicy(new RetryReads()) .withSocketOptions(socketOptions) .build(); Set a read timeout @Override public void storePerson(Person person) { try { BoundStatement bind = storeStatement.bind(person.getName(), person.getAge()); session.execute(bind); } catch (NoHostAvailableException e) { throw new UnableToSavePersonException(); } }
- 37. @chbatey Summary of features • Start as many instances as you like in the same JVM • Prime all primitive types + collections • Prime prepared statements • Verify number of connections your application makes • Test slow queries
- 38. @chbatey Future features • Loading of schema for priming prepared statements • Pretending to be multiple nodes
- 39. @chbatey How do I get it? •It is on maven central •Or go to www.scassandra.org •Executable jar + REST API <dependency> <groupId>org.scassandra</groupId> <artifactId>java-client</artifactId> <version>0.6.0</version> <scope>test</scope> </dependency>
- 40. @chbatey I want to help • Server: https://github.com/scassandra/scassandra- server • Client: https://github.com/scassandra/scassandra-java- client • Tests: https://github.com/scassandra/scassandra-it-java- driver-2
- 41. @chbatey Summary • Testing is good - I want more tools to help me • Existing tools for Cassandra are great at happy path scenarios • Stubbed Cassandra allows testing of edge cases
- 42. @chbatey The end - Questions? www.scassandra.org http://christopher-batey.blogspot.co.uk/ Ping me on twitter @chbatey
- 43. @chbatey How does this work? • Server implemented in Scala • Binary port for your Cassandra application to connect to • Admin port for the REST API • Thin Java wrapper to make it easy to be used in JUnit tests