Thank you for the Metalink note. Interesting, so a session level 10046 trace at level 16 is equivalent to PLAN_STAT=>’ALL_EXECUTIONS’ with bind variables NOT captured and wait events NOT captured, although the Metalink note suggests that bind variables and wait events will be captured (I will have to double-check my trace file).

I am officially stumped – I cannot reproduce your TKPROF output using the 11.2.0.1 TKPROF on 64 bit Linux, 64 bit Windows, nor 32 bit Windows. I tried enabling a 10046 at level 4 (bind variables) and level 16 (this is an invalid level, as far as I am aware, and appears to be the same as a level 1 trace). I also tried setting the STATISTICS_LEVEL to ALL at the session level, just to see if it had any impact.

This is the output that I received on 32 bit Windows with the 11.2.0.1 version of TKPROF from the 10046 trace at level 4:

SELECT /*+ GATHER_PLAN_STATISTICS */
  COUNT(*)
FROM
  T1,
  T2
WHERE
  T1.C2=T2.C2
  AND T1.C2= :N1

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        4      0.00       0.00          0          0          0           0
Execute      4      0.00       0.00          0          0          0           0
Fetch        8      3.51       3.62          4      11120          0           4
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total       16      3.51       3.63          4      11120          0           4

Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 66  

Rows     Row Source Operation
-------  ---------------------------------------------------
      1  SORT AGGREGATE (cr=2780 pr=4 pw=0 time=0 us)
 990025   HASH JOIN  (cr=2780 pr=4 pw=0 time=2956504 us cost=789 size=2000000 card=250000)
    995    INDEX FAST FULL SCAN IND_T1_C1_C2 (cr=1390 pr=2 pw=0 time=199048 us cost=392 size=2000 card=500)(object id 70891)
    995    INDEX FAST FULL SCAN IND_T2_C1_C2 (cr=1390 pr=2 pw=0 time=172086 us cost=392 size=2000 card=500)(object id 70892)

********************************************************************************

This is the output that I received on 32 bit Windows with the 11.2.0.1 version of TKPROF from the 10046 trace at level 16:

SELECT /*+ GATHER_PLAN_STATISTICS */
  COUNT(*)
FROM
  T1,
  T2
WHERE
  T1.C2=T2.C2
  AND T1.C2= :N1

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        4      0.00       0.00          0          0          0           0
Execute      4      0.00       0.00          0          0          0           0
Fetch        8      2.90       2.94          0      11120          0           4
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total       16      2.90       2.94          0      11120          0           4

Misses in library cache during parse: 0
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 66  

Rows     Row Source Operation
-------  ---------------------------------------------------
      1  SORT AGGREGATE (cr=2780 pr=0 pw=0 time=0 us)
 990025   HASH JOIN  (cr=2780 pr=0 pw=0 time=2547184 us cost=789 size=2000000 card=250000)
    995    INDEX FAST FULL SCAN IND_T1_C1_C2 (cr=1390 pr=0 pw=0 time=144005 us cost=392 size=2000 card=500)(object id 70891)
    995    INDEX FAST FULL SCAN IND_T2_C1_C2 (cr=1390 pr=0 pw=0 time=153821 us cost=392 size=2000 card=500)(object id 70892)

********************************************************************************

We can achieve this by getting level 16 trace dump in 11gr2 also. Here is the test case that you provided

http://desaitaral.wordpress.com/2010/10/15/tkprof-in-11gr2-%e2%80%93-new-rows-columns/

C:\>tkprof or112_ora_9393_SQL_10046_ROW_TEST.trc or112_ora_9393_SQL_10046_ROW_TEST.txt
 
TKPROF: Release 11.2.0.1.0 - Development on Thu Oct 14 20:26:27 2010
 
Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

Inside the file:

SELECT /*+ GATHER_PLAN_STATISTICS */
  COUNT(*)
FROM
  T1,
  T2
WHERE
  T1.C2=T2.C2
  AND T1.C2= :N1
 
call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        4      0.00       0.00          0          0          0           0
Execute      4      0.00       0.01          0          0          0           0
Fetch        8      0.39       0.41          2      11120          0           4
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total       16      0.39       0.43          2      11120          0           4
 
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 68  
 
Rows     Row Source Operation
-------  ---------------------------------------------------
      1  SORT AGGREGATE (cr=2780 pr=2 pw=0 time=0 us)
 990025   HASH JOIN  (cr=2780 pr=2 pw=0 time=222451 us cost=759 size=2000000 card=250000)
    995    INDEX FAST FULL SCAN IND_T1_C1_C2 (cr=1390 pr=1 pw=0 time=27459 us cost=379 size=2000 card=500)(object id 72576)
    995    INDEX FAST FULL SCAN IND_T2_C1_C2 (cr=1390 pr=1 pw=0 time=17395 us cost=379 size=2000 card=500)(object id 72577)

One mystery still remaining.

PARSING IN CURSOR #13 len=100 dep=0 uid=68 oct=3 lid=68 tim=1286922845411268 hv=3718770855 ad='257788a80' sqlid='6z03777fugw57'
SELECT /*+ GATHER_PLAN_STATISTICS */
  COUNT(*)
FROM
  T1,
  T2
WHERE
  T1.C2=T2.C2
  AND T1.C2= :N1
END OF STMT
PARSE #13:c=0,e=422,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=0,tim=1286922845411267
BINDS #13:
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=03 fl2=1000000 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=2b9f6b961b38  bln=22  avl=02  flg=05
  value=5
EXEC #13:c=0,e=16909,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=765688747,tim=1286922845428289
FETCH #13:c=137979,e=153467,p=2,cr=2780,cu=0,mis=0,r=1,dep=0,og=1,plh=765688747,tim=1286922845581833
STAT #13 id=1 cnt=1 pid=0 pos=1 obj=0 op='SORT AGGREGATE (cr=2780 pr=2 pw=0 time=0 us)'
STAT #13 id=2 cnt=990025 pid=1 pos=1 obj=0 op='HASH JOIN  (cr=2780 pr=2 pw=0 time=222451 us cost=759 size=2000000 card=250000)'
STAT #13 id=3 cnt=995 pid=2 pos=1 obj=72576 op='INDEX FAST FULL SCAN IND_T1_C1_C2 (cr=1390 pr=1 pw=0 time=27459 us cost=379 size=2000 card=500)'
STAT #13 id=4 cnt=995 pid=2 pos=2 obj=72577 op='INDEX FAST FULL SCAN IND_T2_C1_C2 (cr=1390 pr=1 pw=0 time=17395 us cost=379 size=2000 card=500)'
FETCH #13:c=0,e=2,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=0,plh=765688747,tim=1286922845594968
=====================
 
*** 2010-10-12 18:34:19.469
CLOSE #13:c=0,e=19,dep=0,type=0,tim=1286922859469005
 
PARSING IN CURSOR #15 len=100 dep=0 uid=68 oct=3 lid=68 tim=1286922860187185 hv=3718770855 ad='257788a80' sqlid='6z03777fugw57'
SELECT /*+ GATHER_PLAN_STATISTICS */
  COUNT(*)
FROM
  T1,
  T2
WHERE
  T1.C2=T2.C2
  AND T1.C2= :N1
END OF STMT
PARSE #15:c=0,e=41,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=765688747,tim=1286922860187184
BINDS #15:
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=03 fl2=1000000 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=2b9f6bb525b0  bln=22  avl=02  flg=05
  value=1
EXEC #15:c=999,e=89,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=765688747,tim=1286922860187365
FETCH #15:c=118982,e=119348,p=0,cr=2780,cu=0,mis=0,r=1,dep=0,og=1,plh=765688747,tim=1286922860306760
STAT #15 id=1 cnt=1 pid=0 pos=1 obj=0 op='SORT AGGREGATE (cr=2780 pr=0 pw=0 time=0 us)'
STAT #15 id=2 cnt=998001 pid=1 pos=1 obj=0 op='HASH JOIN  (cr=2780 pr=0 pw=0 time=187133 us cost=759 size=2000000 card=250000)'
STAT #15 id=3 cnt=999 pid=2 pos=1 obj=72576 op='INDEX FAST FULL SCAN IND_T1_C1_C2 (cr=1390 pr=0 pw=0 time=18587 us cost=379 size=2000 card=500)'
STAT #15 id=4 cnt=999 pid=2 pos=2 obj=72577 op='INDEX FAST FULL SCAN IND_T2_C1_C2 (cr=1390 pr=0 pw=0 time=16217 us cost=379 size=2000 card=500)'
FETCH #15:c=0,e=1,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=0,plh=765688747,tim=1286922860307080
CLOSE #15:c=0,e=10,dep=0,type=0,tim=1286922860307402
=====================
 
PARSING IN CURSOR #15 len=100 dep=0 uid=68 oct=3 lid=68 tim=1286922860320484 hv=3718770855 ad='257788a80' sqlid='6z03777fugw57'
SELECT /*+ GATHER_PLAN_STATISTICS */
  COUNT(*)
FROM
  T1,
  T2
WHERE
  T1.C2=T2.C2
  AND T1.C2= :N1
END OF STMT
PARSE #15:c=0,e=16,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=765688747,tim=1286922860320483
BINDS #15:
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=03 fl2=1000000 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=2b9f6b8b54b0  bln=22  avl=02  flg=05
  value=95
EXEC #15:c=0,e=42,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=765688747,tim=1286922860320573
FETCH #15:c=102984,e=103706,p=0,cr=2780,cu=0,mis=0,r=1,dep=0,og=1,plh=765688747,tim=1286922860424305
STAT #15 id=1 cnt=1 pid=0 pos=1 obj=0 op='SORT AGGREGATE (cr=2780 pr=0 pw=0 time=0 us)'
STAT #15 id=2 cnt=819025 pid=1 pos=1 obj=0 op='HASH JOIN  (cr=2780 pr=0 pw=0 time=162422 us cost=759 size=2000000 card=250000)'
STAT #15 id=3 cnt=905 pid=2 pos=1 obj=72576 op='INDEX FAST FULL SCAN IND_T1_C1_C2 (cr=1390 pr=0 pw=0 time=17741 us cost=379 size=2000 card=500)'
STAT #15 id=4 cnt=905 pid=2 pos=2 obj=72577 op='INDEX FAST FULL SCAN IND_T2_C1_C2 (cr=1390 pr=0 pw=0 time=17402 us cost=379 size=2000 card=500)'
FETCH #15:c=0,e=2,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=0,plh=765688747,tim=1286922860424554
CLOSE #15:c=0,e=15,dep=0,type=1,tim=1286922860424806
=====================
 
PARSING IN CURSOR #13 len=100 dep=0 uid=68 oct=3 lid=68 tim=1286922860434659 hv=3718770855 ad='257788a80' sqlid='6z03777fugw57'
SELECT /*+ GATHER_PLAN_STATISTICS */
  COUNT(*)
FROM
  T1,
  T2
WHERE
  T1.C2=T2.C2
  AND T1.C2= :N1
END OF STMT
PARSE #13:c=0,e=9,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=765688747,tim=1286922860434658
BINDS #13:
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=03 fl2=1000000 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=2b9f6b92efe8  bln=22  avl=03  flg=05
  value=995
EXEC #13:c=0,e=65,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=765688747,tim=1286922860434773
FETCH #13:c=37994,e=38483,p=0,cr=2780,cu=0,mis=0,r=1,dep=0,og=1,plh=765688747,tim=1286922860473285
FETCH #13:c=0,e=1,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=0,plh=765688747,tim=1286922860473496
CLOSE #13:c=0,e=5,dep=0,type=3,tim=1286922860473722
=====================

There is a CLOSE entry after every execution, however the type= value differs. The first two executions have a type=0 entry, the third has a type=1 entry, and the fourth that does not have a set of STAT lines has a type=3 entry.

TYPE=0: cursor is actually closed.
TYPE=1: cursor is added to the session cursor cache without pushing an existing cursor out of the session cursor cache.
TYPE=2: cursor is added to the session cursor cache, in the process pushing an existing cursor out of the session cursor cache.
TYPE=3: cursor was already in the session cursor cache
(See Metalink/MOS Doc ID 39817.1 or page 284 of “Secrets of the Oracle Database”)

So, in summary, the average number of rows in the execution plan produced by TKPROF did not include the N1:=995 execution because the cursor was already in the session cursor cache, and the default behavior does not write the STAT lines on future executions when the cursor is held open.

One small mystery solved.

Interesting output, what were the parameters specified for TKPROF? I wonder if I made a mistake somewhere? I used TKPROF from the base release of Oracle Database 11.2.0.1 on 64 bit Linux during my testing and did not see the “Rows (1st)”, “Rows (avg)”, and “Rows (max)” columns in the TKPROF output. Thanks for giving me another item to check in about 9 hours.

Anyone else have a comment about the Centinul’s output? Anyone else able to reproduce Centinul’s output?

I see these additional columns on an 11.2.0.1 database as well:

TKPROF: Release 11.2.0.1.0 - Development on Thu Oct 14 08:22:25 2010

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

Trace file: test_ora_3896.trc
Sort options: default

SQL ID: d6vwqbw6r2ffk
Plan Hash: 1388734953
SELECT USER 
FROM
 DUAL


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch        2      0.00       0.00          0          0          0           1
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        4      0.00       0.00          0          0          0           1

Misses in library cache during parse: 0
Optimizer mode: ALL_ROWS
Parsing user id: 72  
Number of plan statistics captured: 1

Rows (1st) Rows (avg) Rows (max)  Row Source Operation
---------- ---------- ----------  ---------------------------------------------------
         1          1          1  FAST DUAL  (cr=0 pr=0 pw=0 time=0 us cost=2 size=0 card=1)


Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  SQL*Net message to client                       2        0.00          0.00
  SQL*Net message from client                     2        0.00          0.00
********************************************************************************

I have not yet manually reviewed the trace file to see if in fact the STAT lines for the fourth execution were not actually written to the trace file. In examining the test script, I included the following after each execution of the test SQL statement, which should have caused the cursor for the test SQL statement to close (the STAT lines are supposed to be written when the cursor is closed in Oracle Database releases prior to 11.1, but (by default) after the first execution starting with 11.1):

SELECT
  *
FROM
  TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST'));

The catch here with the above reasoning is that after the third execution and attempted close of the cursor, the SQL statement *should* have been added to the session cursor cache, if enabled, and the cursor would not actually be closed (11.1 and above will write a CLOSE entry to the trace file when a cursor is actually closed). That could explain why the fourth execution did not cause the STAT lines to be written to the trace file (with the default value of PLAN_STAT in effect) – because the cursor was still open from the previous (third) execution, the fourth execution was not considered a “new first execution” of the cursor.

It will be at least 10 hours before I will be able to review the raw trace file. I strongly suspect that Egor has correctly answered the riddle of why the “Rows (avg)” column in the TKPROF output excluded the case where the plan line identified by ID 2 returned 25 rows.

If someone would like to try a couple of additional experiments using Oracle Database 11.1.0.6 or higher, see if there is a CLOSE entry in the raw trace file after the first, second, third (there probably will not be one here), and fourth (there probably will not be one here) execution of the test SQL statement, and the STAT lines are written only after the first, second, and third executions. Then, execute:

ALTER SESSION SET SESSION_CACHED_CURSORS=0;

and repeat the test case script. Again process the trace file with TKPROF from Oracle Database 11.2.0.2.

This is a very good example of how a test case script can explain how/why something works as it does.