RATIS-2514. Fix flaky TestReadOnlyRequestWithGrpc.testReadAfterWrite.

[slfan1989]

What changes were proposed in this pull request?

This pull request fixes a flaky assertion in TestReadOnlyRequestWithGrpc.testReadAfterWrite.

The previous test sent multiple async writes without waiting for them to complete, then issued a linearizable read and a read-after-write request concurrently. It asserted that the read-after-write result should be greater than or equal to the concurrent linearizable read result.

That assumption is unreliable because the two reads may have different linearization points and may observe different subsets of concurrent writes.

This patch keeps the original async write workload and both read paths, but makes the test deterministic by:

• waiting for the async writes to complete before issuing the reads;
• verifying that both linearizable read and read-after-write observe the completed prior writes;
• waiting for followers to catch up before cluster shutdown to avoid shutdown-time leak detection races.

What is the link to the Apache JIRA

RATIS-2514. Fix flaky TestReadOnlyRequestWithGrpc.testReadAfterWrite.

How was this patch tested?

Tested with:

./mvnw -pl ratis-test -am -Pgrpc-tests -Dtest=TestReadOnlyRequestWithGrpc#testReadAfterWrite test

Result:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.ratis.grpc.TestReadOnlyRequestWithGrpc
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.746 s - in org.apache.ratis.grpc.TestReadOnlyRequestWithGrpc
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------

[szetszwo]

@slfan1989 , this problem probably has already be fixed in the master branch. Since the difference between two branches are huge. I suggest doing a merge/rebase first.

• RATIS-1931_grpc-zero-copy 238 commits ahead of and 350 commits behind master.

[szetszwo]

The sendReadAfterWrite feature is for two async calls, (1) writeAsync and then (2) readAfterWriteAsync such that (2) must be able see the change by (1). So, we cannot call get() here. Otherwise, the test becomes useless. For exampe,

0. initial: X = 1
1. writeAsync: set X = 3
2. readAfterWriteAsync: must see X = 3 (it is a bug if it can see X = 1)

This may be a bug already fixed (and moved to LinearizableReadTests) in the master branch.