Skip to content

Sketch Environment/Producer/Consumer API #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 46 commits into from
Aug 6, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
93cd259
Sketch producer API
acogoluegnes Jul 9, 2020
8c9d2b7
Merge branch 'master' into producer-spike
acogoluegnes Jul 10, 2020
ba18b3b
Introduce Environment API
acogoluegnes Jul 15, 2020
1087abf
Add URI(s) parameters to build Environment
acogoluegnes Jul 17, 2020
e8f874d
Merge branch 'master' into producer-spike
acogoluegnes Jul 17, 2020
18390f6
Remove some publish methods in Client
acogoluegnes Jul 17, 2020
96dfc24
Organize classes between API/implementation packages
acogoluegnes Jul 17, 2020
be54e5c
Introduce Consumer API
acogoluegnes Jul 20, 2020
80f64e9
Handle publishing error in producer
acogoluegnes Jul 20, 2020
7baa62c
Support stream creation/deletion in environment
acogoluegnes Jul 20, 2020
bc5e567
Document Environment API
acogoluegnes Jul 21, 2020
c6b2d68
Document Producer API
acogoluegnes Jul 21, 2020
d8cc437
Document Consumer API
acogoluegnes Jul 21, 2020
91f52ab
Publish temp API documentation
acogoluegnes Jul 21, 2020
c5c45e2
Support sub-entry batching in producer
acogoluegnes Jul 27, 2020
d8cf412
Synchronize message accumulator access
acogoluegnes Jul 27, 2020
afb7203
Limit number of outstanding publish confirms
acogoluegnes Jul 27, 2020
ea960d0
Document maxUnconfirmedMessages and subEntrySize
acogoluegnes Jul 27, 2020
07f8681
Recover locator connection in environment
acogoluegnes Jul 28, 2020
504a68b
Close producers and consumers in environment
acogoluegnes Jul 28, 2020
7d3bea5
Document environment settings
acogoluegnes Jul 29, 2020
9e2300e
Use new API for sample application
acogoluegnes Jul 29, 2020
a0771a2
Use Consumer in performance tool
acogoluegnes Jul 29, 2020
147b8d5
Downsample latency calculation in performance tool
acogoluegnes Jul 29, 2020
b41aa94
Deal with stream unavailibility in consumer
acogoluegnes Jul 30, 2020
4bc6744
Add delay before consumer re-assignment after metadata update
acogoluegnes Jul 31, 2020
87b866d
Add unit tests for DefaultClientSubscriptions
acogoluegnes Jul 31, 2020
c5e3736
Unit test DefaultClientSubscriptions sub/unsub
acogoluegnes Jul 31, 2020
a4e5abe
Update data structure before subscription
acogoluegnes Jul 31, 2020
eae2887
More DefaultClientSubscriptions unit tests
acogoluegnes Jul 31, 2020
6782fa8
More DefaultClientSubscriptions unit tests
acogoluegnes Jul 31, 2020
cd5cf2f
Schedule candidates lookup on metadata update
acogoluegnes Aug 3, 2020
cc0806e
Create async retry utility for metadata update
acogoluegnes Aug 3, 2020
f0ded28
More DefaultClientSubscriptions unit tests
acogoluegnes Aug 3, 2020
c4fea33
Use async retry utility for locator recovery
acogoluegnes Aug 3, 2020
53ea1d9
Add some Environment unit tests
acogoluegnes Aug 4, 2020
5efe862
Handle connection loss in consumer
acogoluegnes Aug 5, 2020
83c3c12
Rename RecoveryBackOffDelayPolicy to BackOffDelayPolicy
acogoluegnes Aug 5, 2020
fe91c97
Add node failure test for consumer
acogoluegnes Aug 6, 2020
6afab81
Add unit for consumer connection recovery
acogoluegnes Aug 6, 2020
ea52a49
Disable a couple of recovery tests
acogoluegnes Aug 6, 2020
a6bea16
Remove Client documentation
acogoluegnes Aug 6, 2020
b3d4bc0
Improve wording in documentation
acogoluegnes Aug 6, 2020
0190f1d
Add some Javadoc to Client
acogoluegnes Aug 6, 2020
a2f2394
Kill connection instead of stopping node in test
acogoluegnes Aug 6, 2020
298826e
Publish documentation to snapshot directory
acogoluegnes Aug 6, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add unit tests for DefaultClientSubscriptions
  • Loading branch information
acogoluegnes committed Jul 31, 2020
commit 87b866d75491ae8994ffeb69e6e5744963119ab9
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,8 @@ private Client locator() {
return environment.locator();
}

private List<Client.Broker> findBrokersForStream(String stream) {
// package protected for testing
List<Client.Broker> findBrokersForStream(String stream) {
// FIXME make sure locator is not null (retry)
Map<String, Client.StreamMetadata> metadata = locator().metadata(stream);
if (metadata.size() == 0 || metadata.get(stream) == null) {
Expand All @@ -91,7 +92,7 @@ private List<Client.Broker> findBrokersForStream(String stream) {

List<Client.Broker> replicas = streamMetadata.getReplicas();
if ((replicas == null || replicas.isEmpty()) && streamMetadata.getLeader() == null) {
throw new IllegalStateException("Not node available to consume from stream " + stream);
throw new IllegalStateException("No node available to consume from stream " + stream);
}

List<Client.Broker> brokers;
Expand Down Expand Up @@ -178,6 +179,7 @@ private SubscriptionState(Client.ClientParameters clientParameters) {
if (streamSubscription != null) {
streamSubscription.offset = offset;
streamSubscription.messageHandler.handle(offset, message);
// FIXME set offset here as well, best effort to avoid duplicates
} else {
LOGGER.warn("Could not find stream subscription {}", subscriptionId);
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
// Copyright (c) 2020 VMware, Inc. or its affiliates. All rights reserved.
//
// This software, the RabbitMQ Stream Java client library, is dual-licensed under the
// Mozilla Public License 2.0 ("MPL"), and the Apache License version 2 ("ASL").
// For the MPL, please see LICENSE-MPL-RabbitMQ. For the ASL,
// please see LICENSE-APACHE2.
//
// This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND,
// either express or implied. See the LICENSE file for specific language governing
// rights and limitations of this software.
//
// If you have any questions regarding licensing, please contact us at
// [email protected].

package com.rabbitmq.stream.impl;

import com.rabbitmq.stream.Constants;
import com.rabbitmq.stream.OffsetSpecification;
import com.rabbitmq.stream.StreamDoesNotExistException;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.mockito.Mock;
import org.mockito.MockitoAnnotations;

import java.util.Arrays;
import java.util.Collections;

import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.mockito.Mockito.when;

public class DefaultClientSubscriptionsTest {

@Mock
StreamEnvironment environment;
@Mock
StreamConsumer consumer;
@Mock
Client locator;

DefaultClientSubscriptions clientSubscriptions;

@BeforeEach
void init() {
MockitoAnnotations.initMocks(this);
clientSubscriptions = new DefaultClientSubscriptions(environment);
}

@Test
void subscribeShouldThrowExceptionWhenNoMetadataForTheStream() {
when(environment.locator()).thenReturn(locator);
assertThatThrownBy(() -> clientSubscriptions.subscribe(consumer, "stream", OffsetSpecification.first(), (offset, message) -> {
})).isInstanceOf(StreamDoesNotExistException.class);
}

@Test
void subscribeShouldThrowExceptionWhenStreamDoesNotExist() {
when(environment.locator()).thenReturn(locator);
when(locator.metadata("stream"))
.thenReturn(Collections.singletonMap("stream",
new Client.StreamMetadata("stream", Constants.RESPONSE_CODE_STREAM_DOES_NOT_EXIST, null, null)));
assertThatThrownBy(() -> clientSubscriptions.subscribe(consumer, "stream", OffsetSpecification.first(), (offset, message) -> {
})).isInstanceOf(StreamDoesNotExistException.class);
}

@Test
void subscribeShouldThrowExceptionWhenMetadataResponseIsNotOk() {
when(environment.locator()).thenReturn(locator);
when(locator.metadata("stream"))
.thenReturn(Collections.singletonMap("stream",
new Client.StreamMetadata("stream", Constants.RESPONSE_CODE_ACCESS_REFUSED, null, null)));
assertThatThrownBy(() -> clientSubscriptions.subscribe(consumer, "stream", OffsetSpecification.first(), (offset, message) -> {
})).isInstanceOf(IllegalStateException.class);
}

@Test
void subscribeShouldThrowExceptionIfNodeAvailableForStream() {
when(environment.locator()).thenReturn(locator);
when(locator.metadata("stream"))
.thenReturn(Collections.singletonMap("stream",
new Client.StreamMetadata("stream", Constants.RESPONSE_CODE_OK, null, null)));
assertThatThrownBy(() -> clientSubscriptions.subscribe(consumer, "stream", OffsetSpecification.first(), (offset, message) -> {
})).isInstanceOf(IllegalStateException.class);
}

@Test
void findBrokersForStreamShouldReturnLeaderIfNoReplicas() {
when(environment.locator()).thenReturn(locator);
when(locator.metadata("stream"))
.thenReturn(Collections.singletonMap("stream",
new Client.StreamMetadata("stream", Constants.RESPONSE_CODE_OK, new Client.Broker("leader", -1), null)));
assertThat(clientSubscriptions.findBrokersForStream("stream"))
.hasSize(1)
.contains(new Client.Broker("leader", -1));
}

@Test
void findBrokersForStreamShouldReturnReplicasIfThereAreSome() {
when(environment.locator()).thenReturn(locator);
when(locator.metadata("stream"))
.thenReturn(Collections.singletonMap("stream",
new Client.StreamMetadata("stream", Constants.RESPONSE_CODE_OK, null, Arrays.asList(
new Client.Broker("replica1", -1), new Client.Broker("replica2", -1)
))));
assertThat(clientSubscriptions.findBrokersForStream("stream"))
.hasSize(2)
.contains(new Client.Broker("replica1", -1), new Client.Broker("replica2", -1));
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import java.util.UUID;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.IntStream;

import static org.assertj.core.api.Assertions.assertThat;
Expand Down Expand Up @@ -138,11 +139,13 @@ void consumerShouldKeepConsumingIfStreamBecomesUnavailable() throws Exception {
assertThat(publishLatch.await(10, TimeUnit.SECONDS)).isTrue();
producer.close();

AtomicInteger receivedMessageCount = new AtomicInteger(0);
CountDownLatch consumeLatch = new CountDownLatch(messageCount);
CountDownLatch consumeLatchSecondWave = new CountDownLatch(messageCount * 2);
StreamConsumer consumer = (StreamConsumer) environment.consumerBuilder()
.stream(s)
.messageHandler((offset, message) -> {
receivedMessageCount.incrementAndGet();
consumeLatch.countDown();
consumeLatchSecondWave.countDown();
})
Expand Down Expand Up @@ -174,6 +177,7 @@ void consumerShouldKeepConsumingIfStreamBecomesUnavailable() throws Exception {
producerSecondWave.close();

assertThat(consumeLatchSecondWave.await(10, TimeUnit.SECONDS)).isTrue();
assertThat(receivedMessageCount.get()).isEqualTo(messageCount * 2 + 1);
assertThat(consumer.isOpen()).isTrue();

consumer.close();
Expand Down