gRPC: hang on first request when using `GrpcStorageOptions.setAttemptDirectPath(true)` and `GOOGLE_CLOUD_ENABLE_DIRECT_PATH_XDS=true`

[BenWhitehead]

Affected version(s): 2.14.0+

There is a noticeable hang (several seconds) for the first request from the client when using directPath XDS. After the initial hang it doesn't seem to be an issue.

[BenWhitehead]

Minimized Repro (using com.google.cloud:google-cloud-storage:2.17.2)

package x;

import com.google.cloud.storage.Blob;
import com.google.cloud.storage.BlobId;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import com.google.common.base.Stopwatch;

final class Main {
  public static void main(String[] args)
      throws Exception {
    try (Storage s = StorageOptions.grpc().setAttemptDirectPath(true).build().getService()) {
      BlobId id = BlobId.fromGsUtilUri(args[0]);
      doGet(s, id);
      doGet(s, id);
      doGet(s, id);
    }
  }

  private static void doGet(Storage s, BlobId id) {
    Stopwatch sw = Stopwatch.createStarted();
    Blob blob = s.get(id);
    Stopwatch stop = sw.stop();
    System.out.println("Fetching object metadata took " + stop);
  }
}

Results

Fetching object metadata took 10.36 s
Fetching object metadata took 43.23 ms
Fetching object metadata took 36.63 ms

[BenWhitehead]

Repro in repo form: https://github.com/BenWhitehead/dp-hang-debug

[BenWhitehead]

Fixed in grpc 1.54.0