Thanks for stopping by my blog and sharing your great troubleshooting skills. I did not feel comfortable/confident with using the time= values in the Row Source Operation section of the TKPROF output for analysis purposes because of the apparent inconsistencies between the STAT lines and the aggregation of the PARSE, EXEC, and FETCH lines, but that does not mean that the time= values should not be used.

I seem to recall Jonathan Lewis telling me in the first half of 2008 on the OTN forums that the time= values in the STAT lines could be inaccurate. I do not recall if he stated that the TIME= values could not be trusted because not all of the code paths are instrumented, there are rounding errors when a code path is repeatedly executed in a loop (a NESTED LOOP join), or possibly another reason such as time occasionally being allocated to the wrong operation in the plan. Of course, I have been unable to find that thread on OTN, so I might just be remembering his statement incorrectly.

It is an interesting problem, so I hope that the OP returns to the Usenet thread.

It is good that you are reading these blog articles somewhat as personal challenges – that is how many of my articles are written. I do not always tell everything that I see to be wrong or right – partially due to time constraints, and partially to encourage other people to offer comments that provide much more complete (and accurate) answers than I could have offered. Fortunately, there are a number of very smart people who are willing to freely share their knowledge.

Good idea to check the estimated number of rows per operation in an execution plan – it could be that statistics have not been gathered in the last 8 years which will cause Oracle to greatly under-estimate the cardinality for the operations, or it could be a problem with non-mutually exclusive predicates (I think that Randolf described this as a potential problem). Randolf’s comment about the timing in the portion of the execution plan agrees with a comment that I thought about making, but his explanation is better than the one I considered offering.

Good point about questioning what would happen with the 64,806 rows – will 64,800 of those rows be thrown away, or will all of the rows end up on a report of some sort?

The Oracle 11.2.0.x documentation describes the “r=” entry in the Row Source Operation section of the TKPROF output. Quoting from the documentation:

Rows     Row Source Operation
-------  ---------------------------------------------------
      0  DELETE  (cr=43141 r=266947 w=25854 time=60235565 us)
  28144   HASH JOIN ANTI (cr=43057 r=262332 w=25854 time=48830056 us)
  51427    TABLE ACCESS FULL STATS$SQLTEXT (cr=3465 r=3463 w=0 time=865083 us)
 647529    INDEX FAST FULL SCAN STATS$SQL_SUMMARY_PK 
                      (cr=39592 r=39325 w=0 time=10522877 us) (object id 7409)

“In this sample TKPROF output, note the following under the Row Source Operation column:

•cr specifies consistent reads performed by the row source
•r specifies physical reads performed by the row source
•w specifies physical writes performed by the row source
•time specifies time in microseconds”

The original poster in the Usenet thread is running Oracle Database 9.2.0.6, so the “r=” entry on the STAT= lines in the trace file (the “Row Source Operation” lines in the TKPROF output) indicate the number of blocks that were physically read from disk. With your help, I think that we just found a documentation bug – something that was not updated in the documentation to reflect changes in Oracle’s tracing behavior. I am guessing that the STAT line trace file content changed starting with Oracle 10.1.0.x, but the documentation was never updated to reflect that change.

This is what the Row Source Operation section looks like in Oracle 10.2.0.x:

Rows     Row Source Operation
-------  ---------------------------------------------------
      1  NESTED LOOPS  (cr=7 pr=0 pw=0 time=160 us)
      2   TABLE ACCESS BY INDEX ROWID CURRENCY (cr=2 pr=0 pw=0 time=74 us)
      2    INDEX FULL SCAN SYS_C0018003 (cr=1 pr=0 pw=0 time=42 us)(object id 28289)
      1   TABLE ACCESS BY INDEX ROWID CUST_ORDER_CURR (cr=5 pr=0 pw=0 time=63 us)
      1    INDEX UNIQUE SCAN SYS_C0018028 (cr=4 pr=0 pw=0 time=37 us)(object id 28305)

Centinul answered your second question about how I calculated the average time for a block read. Interestingly, Randolf came up with entirely different average read times by using the “Row Source Operation” section from the TKPROF output. He calculated that the average read time for an index block was 13 ms, while the average read time for a table block was between 3 and 5 ms. Which calculation is more accurate, I do not know.

Thanks. I was wondering if that was somehow based on elapsed time figures of individual plan steps.

I’ve replied to the original thread on Usenet group – there are a couple of noteworthy oddities with this TKPROF output that might explain the issue with the missing 20 minutes.

@Narendra: The PFMQ_MESSAGESTATUS is actually queried for every iteration, since it is a child operation of the inner table of the NESTED LOOP operation. But as you can see from the TKPROF output it does not seem to be relevant in this particular case – it only takes 2 seconds to execute this subquery 64,000 times, so there are other areas to tackle first and no need to replace this subquery with the present execution plan.

See my comments on the original thread for more information.

Randolf

I believe the computation is as follows:

(total elapsed time – time spent on cpu) / # blocks read from disk

(580.31 – 14.56) / 231750 = 0.0024412 seconds / block

One question. How did you come to the conclusion the average time to read a block from disk was 0.0024412 seconds (2.4412ms) ?

Nice analysis. You covered almost everything that I could think of and then asked what else. That is not fair. Here is what I can think of:
1) To confirm the 3rd point (about row elimination) and 6th point in your analysis (about execution plan starting point), I would check what is the EXPLAIN PLAN outcome. Assuming this is on database version that gives predicate information in plan, the EXPLAIN PLAN will a) prove if expected plan is same as actual plan and b) tell clearly at what stages filter predicates are applied. Also, the ROWS column in EXPLAIN PLAN output shows expected rowcounts (per iteration) for a NESTED LOOP join which can be vital info as ROW SOURCE OPERATION reports row counts for all iterations combined (although in this case, it may not provide any new details)
2) I was actually a bit surprised to see following steps in ROW SOURCE
OPERATION

  64806       TABLE ACCESS BY INDEX ROWID PFMQ_MESSAGESTATUS (cr=325234 r=23150 w=0 time=224278563 us)
  64820        INDEX RANGE SCAN XPKPF_MESSAGESTATUS (cr=260414 r=15792 w=0 time=208616639 us)(object id 6515)
  64820         SORT AGGREGATE (cr=129644 r=116 w=0 time=1973822 us)
  64820          FIRST ROW  (cr=129644 r=116 w=0 time=1810738 us)
  64820           INDEX RANGE SCAN (MIN/MAX) XPKPF_MESSAGESTATUS (cr=129644 r=116 w=0 time=1756030 us)(object id 6515)

With the way query is written, I was expecting PFMQ_MESSAGESTATUS to be accessed more than once (for each iteraion). Is this some kind of optimizer improvement? Or is it because XPKPF_MESSAGESTATUS index includes both “statusrevisionnumber” and “ID” columns? Will there be any benifit in rewriting this query (using analytical function) in order to remove the correlated subquery ?
3) Finally, my most important question will be- What is done (by the “application”) with 64806 ordered set of rows that this query produce? In other words, is it absolutely essential to have this query being executed on its own?
I am not saying this is necessarily wrong but I would certainly validate the approach/design.

p.s. BTW, what does the “r=” component in ROW SOURCE OPERATION indicate? Physical Reads? I always seem to get “pr=” in 10.2.0.x versions and no “r=” in TkProf reports.