Add register analysis to repo analysis API #93955

DaveCTurner · 2023-02-21T08:56:24Z

In #93825 we introduced a CAS operation on snapshot repositories. This commit extends the repository analysis API to include some concurrent operations which atomically increment a register, retrying on collisions, and a check which ensures that the final value of the register is as expected.

In elastic#93825 we introduced a CAS operation on snapshot repositories. This commit extends the repository analysis API to include some concurrent operations which atomically increment a register, retrying on collisions, and a check which ensures that the final value of the register is as expected.

elasticsearchmachine · 2023-02-21T08:56:48Z

Hi @DaveCTurner, I've created a changelog YAML for you.

elasticsearchmachine · 2023-02-21T08:56:49Z

Pinging @elastic/es-distributed (Team:Distributed)

DaveCTurner · 2023-02-21T09:36:29Z

The test failures are legitimate and highlight an interesting question: there are situations where we might not even be able to read the current value (e.g. someone else holds the lock). How should we handle this? Do we need a specific exception to describe this case?

DaveCTurner · 2023-02-21T12:33:40Z

It seems there's some problems related to concurrently manipulating file locks, e.g. a6b4e67 would trip this assertion in the JDK:

[2023-02-21T13:29:58,265][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [yamlRestTest-0] fatal error in thread [elasticsearch[yamlRestTest-0][snapshot][T#1]], exiting
java.lang.AssertionError: null
        at sun.nio.ch.FileLockTable.removeKeyIfEmpty(FileLockTable.java:139) ~[?:?]
        at sun.nio.ch.FileLockTable.removeAll(FileLockTable.java:193) ~[?:?]
        at sun.nio.ch.FileChannelImpl.implCloseChannel(FileChannelImpl.java:188) ~[?:?]
        at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:112) ~[?:?]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer.compareAndExchangeRegister(FsBlobContainer.java:418) ~[elasticsearch-8.8.0-SNAPSHOT.jar:?]
        at org.elasticsearch.repositories.blobstore.testkit.RegisterAnalyzeAction$TransportAction$1Execution.doRun(RegisterAnalyzeAction.java:121) ~[?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958) ~[elasticsearch-8.8.0-SNAPSHOT.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.8.0-SNAPSHOT.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1589) ~[?:?]

I haven't spent much time tracking this down, but I suspect this might be tricky to avoid in general (e.g. allowing for things like mounting the same repository twice, aliasing via symlinks, etc.). Here I've implemented a simple solution that serialises all attempts to open or close a locked file.

DaveCTurner · 2023-02-21T14:04:43Z

Sorry for all the noise/iterations here @fcofdez, these new tests showed that the existing implementation needed quite a bit of work. Looks to be done now tho, so this is good to review.

fcofdez

This looks mostly good to me, I left a question about one of the invalid register transitions though.

fcofdez · 2023-02-22T17:18:03Z

server/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java

-                return FileChannel.open(path, StandardOpenOption.READ, StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE);
+    private record LockedFileChannel(FileChannel fileChannel, Closeable fileLock) implements Closeable {
+
+        // avoid concurrently opening/closing locked files


Can we add to the comment that we're serializing the LockedFileChannel creation due to a tripping assertion in the JDK?

Sure, done in 37e3761.

fcofdez · 2023-02-22T17:18:59Z

server/src/main/java/org/elasticsearch/common/blobstore/BlobContainer.java

     */
-    long compareAndExchangeRegister(String key, long expected, long updated) throws IOException;
+    long compareAndExchangeRegister(String key, long expected, long updated) throws IOException, ConcurrentRegisterOperationException;


nit: does it need to be a checked exception?

No, but I expect that callers will need to handle this exception very close to the call site (much like handling the case where the operation failed but managed to read the value) and forgetting to do this would be pretty bad, so I went this way. Perhaps it would be nicer to return an OptionalLong, we don't really need any details?

Perhaps it would be nicer to return an OptionalLong, we don't really need any details?

see ab1385b for a draft of that

fcofdez · 2023-02-22T17:37:27Z

...it/src/main/java/org/elasticsearch/repositories/blobstore/testkit/RegisterAnalyzeAction.java

+                            final var witness = blobContainer.compareAndExchangeRegister(registerName, currentValue, currentValue + 1);
+                            if (witness == currentValue) {
+                                listener.onResponse(null);
+                            } else if (witness < currentValue || witness >= request.getRequestCount()) {


witness >= request.getRequestCount() I think this is a valid scenario? i.e. another task runs > request.getRequestCount() updates while this task is waiting to be executed. Am I missing something here?

There are only request.getRequestCount() increment requests in total, and this one hasn't succeeded yet, so the current value must be smaller.

Right, for some reason I thought that the variable was created within the loop in https://github.com/elastic/elasticsearch/pull/93955/files#diff-2685c0b9e390123278fc8429f89d237daf9ed0d74ba41fd9ccc8318d077ca69cR454-R465 🤦

fcofdez

LGTM 👍

Further work towards the S3 compare-and-exchange implementation showed that we would like this API to permit async operations. This commit moves to an async API. Also, this change made it fairly awkward to use an exception to deliver to the caller the indication that the current value could not be read, so this commit adjusts things to use `OptionalLong` throughout as suggested in the discussion on elastic#93955.

Further work towards the S3 compare-and-exchange implementation showed that we would like this API to permit async operations. This commit moves to an async API. Also, this change made it fairly awkward to use an exception to deliver to the caller the indication that the current value could not be read, so this commit adjusts things to use `OptionalLong` throughout as suggested in the discussion on #93955.

DaveCTurner added >enhancement :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v8.8.0 labels Feb 21, 2023

DaveCTurner requested a review from fcofdez February 21, 2023 08:56

elasticsearchmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Feb 21, 2023

Update docs/changelog/93955.yaml

8d74253

DaveCTurner added 2 commits February 21, 2023 09:00

Comments

21934e2

Operator-only

60fda63

DaveCTurner added 5 commits February 21, 2023 09:56

Better handling of concurrent operations

c616ab1

Success IT can randomly trigger retries

1541584

Verify that register always increases up to max

a6b4e67

Avoid races within one node

ae2efe6

Avoid concurrently acquiring locks

2884210

Fix another test

40311d7

fcofdez reviewed Feb 22, 2023

View reviewed changes

DaveCTurner added 2 commits February 22, 2023 20:18

Merge branch 'main' into 2023-02-20-repo-analysis-register

d1f8a91

Better comment

37e3761

fcofdez approved these changes Feb 23, 2023

View reviewed changes

DaveCTurner merged commit eac8727 into elastic:main Feb 23, 2023

DaveCTurner deleted the 2023-02-20-repo-analysis-register branch February 23, 2023 09:21

DaveCTurner mentioned this pull request Feb 23, 2023

[CI] RepositoryAnalysisFailureIT testFailsIfRegisterHoldsSpuriousValue failing #94067

Closed

DaveCTurner mentioned this pull request Feb 23, 2023

Async blob-store compare-and-exchange API #94092

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add register analysis to repo analysis API #93955

Add register analysis to repo analysis API #93955

Uh oh!

DaveCTurner commented Feb 21, 2023

Uh oh!

elasticsearchmachine commented Feb 21, 2023

Uh oh!

elasticsearchmachine commented Feb 21, 2023

Uh oh!

DaveCTurner commented Feb 21, 2023

Uh oh!

DaveCTurner commented Feb 21, 2023

Uh oh!

DaveCTurner commented Feb 21, 2023

Uh oh!

fcofdez left a comment

Uh oh!

fcofdez Feb 22, 2023

Uh oh!

DaveCTurner Feb 22, 2023

Uh oh!

fcofdez Feb 22, 2023

Uh oh!

DaveCTurner Feb 22, 2023

Uh oh!

DaveCTurner Feb 22, 2023

Uh oh!

fcofdez Feb 22, 2023

Uh oh!

DaveCTurner Feb 22, 2023

Uh oh!

fcofdez Feb 23, 2023

Uh oh!

fcofdez left a comment

Uh oh!

Uh oh!

Add register analysis to repo analysis API #93955

Add register analysis to repo analysis API #93955

Uh oh!

Conversation

DaveCTurner commented Feb 21, 2023

Uh oh!

elasticsearchmachine commented Feb 21, 2023

Uh oh!

elasticsearchmachine commented Feb 21, 2023

Uh oh!

DaveCTurner commented Feb 21, 2023

Uh oh!

DaveCTurner commented Feb 21, 2023

Uh oh!

DaveCTurner commented Feb 21, 2023

Uh oh!

fcofdez left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fcofdez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!