Remove flakiness from url_info tests...

Remove flakiness from url_info tests

This test was testing to be sure that timing was
correctly monitored during the lifetime of 
these UrlHostInfo instances.  The new code defensively
checks values, and when tehy appear to be out of
spec, it verifies that the test bot is indeed
running slowly, and exits.  This will usually
run the test as expected, but gracefully
exit when the bot is just too slow (or
has too much variance in execution speed)
to run the test carefully.

r=phajdan.jr
BUG=55169
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=60962

[Paweł Hajdan Jr., 2010-09-28 08:20:01]

Thanks for working on this. I think it needs much more work though.

At some point I was thinking about mocking time completely for this test. It
would be less "realistic", but much more solid. What do you think?

http://codereview.chromium.org/3431028/diff/5001/6001
File chrome/browser/net/url_info.cc (right):

http://codereview.chromium.org/3431028/diff/5001/6001#newcode75
chrome/browser/net/url_info.cc:75: kCacheExpirationDuration = time;
Can we fix this confusion while we're here? Let's make this look like a variable
(cache_expiration_duration_) and not a constant. Moreover, can we move the
implementation to the header file guarded with #ifdef UNIT_TEST? This will
ensure it's only used by tests, as it sounds really dangerous to change it in
production.

http://codereview.chromium.org/3431028/diff/5001/6003
File chrome/browser/net/url_info_unittest.cc (right):

http://codereview.chromium.org/3431028/diff/5001/6003#newcode56
chrome/browser/net/url_info_unittest.cc:56: return;  // Lets punt here, the test
bot is too slow.
So it's still timing dependent? I guess the flakiness is still going to be
there.

I'd suggest to rather remove the test than to keep a potential for flakiness.

http://codereview.chromium.org/3431028/diff/5001/6003#newcode70
chrome/browser/net/url_info_unittest.cc:70: PlatformThread::Sleep(25);
The Sleep is still there, suggesting we could do better.

http://codereview.chromium.org/3431028/diff/5001/6003#newcode81
chrome/browser/net/url_info_unittest.cc:81:
PlatformThread::Sleep(kMockExpirationTime + 20);
Adding a Sleep to fight flakiness sounds suspicious to me.

[jar (doing other things), 2010-09-28 16:23:24]

I don't believe it is reasonable to mock time.  The problem is that with "slow"
bots, any thread can AND WILL suspend for seemingly arbitrary lengths of time. 
A Real Time OS would offer guarantees against this, and any ordinary OS (Linux
or windows) will VERY rarely starve threads for extended periods of time (over a
second or two).  Sadly, the bots, presumably running virtual machines, regularly
starve threads for a LONG time (exceeding 1 second).  IF we try to "mock" time,
we'd still have the problem that some threads won't even wake to call for a time
value.  If we put in enough logic to try to ensure that each thread makes a time
call before going forward, we'll have much more test code that is probably
unreadable.  We would probably swamp the tested code with an order of magnitude
or 2 excess complexity.

Bottom line: I believe I removed all flakiness by this change.  There is no
longer a chance of a "random failure."  Any failure we observe should (unless my
logic is incorrect) be the result of a genuine failure.

http://codereview.chromium.org/3431028/diff/5001/6001
File chrome/browser/net/url_info.cc (right):

http://codereview.chromium.org/3431028/diff/5001/6001#newcode75
chrome/browser/net/url_info.cc:75: kCacheExpirationDuration = time;
On 2010/09/28 08:20:01, Paweł Hajdan Jr. wrote:
> Can we fix this confusion while we're here? Let's make this look like a
variable
> (cache_expiration_duration_) and not a constant. Moreover, can we move the
> implementation to the header file guarded with #ifdef UNIT_TEST? This will
> ensure it's only used by tests, as it sounds really dangerous to change it in
> production.

Done.

http://codereview.chromium.org/3431028/diff/5001/6003
File chrome/browser/net/url_info_unittest.cc (right):

http://codereview.chromium.org/3431028/diff/5001/6003#newcode56
chrome/browser/net/url_info_unittest.cc:56: return;  // Lets punt here, the test
bot is too slow.
There is no "flakiness," as defined to be a "random failure even when code is
correct." The test now observes something that could be misconstrued as a
failure, checks to see if the environment has precluded such testing (by
stalling on a thread too long), and if so, abandons the test.  If the
environment does not look bad, then the test results are validated.  Now, when
there is a bad result, it MUST have been a real failure, and can't be caused by
a flaky intermittent unjustified fault.

On 2010/09/28 08:20:01, Paweł Hajdan Jr. wrote:
> So it's still timing dependent? I guess the flakiness is still going to be
> there.
> 
> I'd suggest to rather remove the test than to keep a potential for flakiness.

http://codereview.chromium.org/3431028/diff/5001/6003#newcode70
chrome/browser/net/url_info_unittest.cc:70: PlatformThread::Sleep(25);
I don't believe we can do better.  The duration is 25ms, so I don't think it
will slow testing.

If you have a better suggestion, I'd like to understand it.

On 2010/09/28 08:20:01, Paweł Hajdan Jr. wrote:
> The Sleep is still there, suggesting we could do better.

http://codereview.chromium.org/3431028/diff/5001/6003#newcode81
chrome/browser/net/url_info_unittest.cc:81:
PlatformThread::Sleep(kMockExpirationTime + 20);
On 2010/09/28 08:20:01, Paweł Hajdan Jr. wrote:
> Adding a Sleep to fight flakiness sounds suspicious to me.

The sleep duration, even on a "slow" bot, is GUARANTEED to exceed the requested
duration (in this case, 320ms).  The expect that is checked below is tantamount
to asking the instance if it has now been more than 300ms since its last state
change.  As a result, this is fully deterministic, and there is on chance of
being flaky (false failure) unless logic in code is broken.

[Paweł Hajdan Jr., 2010-09-29 08:32:07]

I see. Just one remaining thing blocking this review.

Note there are two things there:

1) #ifdef UNIT_TEST
2) changing the name of the no-longer-constant

http://codereview.chromium.org/3431028/diff/5001/6001
File chrome/browser/net/url_info.cc (right):

http://codereview.chromium.org/3431028/diff/5001/6001#newcode75
chrome/browser/net/url_info.cc:75: kCacheExpirationDuration = time;
On 2010/09/28 16:23:24, jar wrote:
> On 2010/09/28 08:20:01, Paweł Hajdan Jr. wrote:
> > Can we fix this confusion while we're here? Let's make this look like a
> variable
> > (cache_expiration_duration_) and not a constant. Moreover, can we move the
> > implementation to the header file guarded with #ifdef UNIT_TEST? This will
> > ensure it's only used by tests, as it sounds really dangerous to change it
in
> > production.
> 
> Done.

Maybe I just don't see the new CL... could you make sure it's really done and
uploaded?

[jar (doing other things), 2010-09-29 16:30:36]

Responding to comments by Pawel.

Sorry the other upload didn't have the changes.  It pushed before the file save
was complete.


You're correct, given the that we changed the value for testing purposes, it was
not a constant (in the const or the style guide sense), and so I updated it name
accordingly.  

re: Putting inside #ifdef UNIT_TEST

It is not dangerous for anyone to call this function, but rather it added to
readability to assert that this was a constant.  Nothing would break if this
changed dynamically during the run of the program.  It is also nice that the
global optimizing compiler will automatically discard the accessor "set" method
(I believe), as it is never called (in chrome.exe).  Bottom line: I think it is
better, when possible, to avoid ifdef code unless there is a larger value add.

[Paweł Hajdan Jr., 2010-09-29 16:35:43]

LGTM