December 6, 2009 (Updated February 24, 2010)
Are test cases valuable or not? These quotes were found through Internet searches:
“Oracle is NOT MATH, a single contrary test case does not invalidate any general principle of Oracle performance.”
“Just because someone show a case where indexes do not need to be rebuilt, that DOES NOT mean that positive cases don’t exist!
I can write a test case to ‘prove’ that virtually any statement about Oracle is un-true. It’s easy.
People who believe that a single negative test case proves something is wrong join the ranks of the ‘deniers’, folks who cite ‘proof’ that the moon landing never happened, and that 911 was a government conspiracy.”
http://forums.oracle.com/forums/thread.jspa?messageID=3195898&tstart=0 (thread was taken offline by OTN, but still shows in search results)
“100% true? You are joking, right? There is NOTHING that is 100% true about database tuning . . . . YOU CANNOT PROVE ANYTHING ABOUT ORACLE PERFORMANCE. EVER. NO EQUATIONS, NO PROOFS, NO WAY, NO HOW. . . . Only fools or charlatans will claim that something about Oracle performance has been ‘proven wrong’. . . .
“It’s ridiculous to reverse engineer Oracle with test cases, when we can ask the people who hold the source code.”
“A test case is not the same as software testing!
The problem is that a single-user ‘test case’ on a PC is not a valid test, by any measure. . .
It does not accurately reproduce real-world behavior, especially in performance tuning, where slowdowns are only seen under heavy loads.”
“It baffles me why any practicng DBA would want to write a test case, when they have a real-world test database, full of real data and waiting to be used . . .”
(Added February 24, 2010):
If you search this site for the phrase test case, I think that it will be clear that test cases, when properly constructed, are extremely helpful for determining how things work – and how things should not work. Properly constructing a test case is critical to help eliminate false causation and false correlation. Employing the scientific method (secondary reference) is important when building test cases to help control false positives and false negatives. Keep in mind that if something is stated as an absolute (for example “the sun rises in the East” or “on Earth the sun always rises from the East“), it only requires a single negative test case to refute the absolute statement.