Thanks for the comments.

This is really cool. I came here from OTN and saw the struff that Charles is showing and then Mich’s stuff. They are great, A very good explanation and examples of treating tkprof output and aggregates.

Thanks,

Rick

Thanks for the tip on using PLAN_STAT.

Anyway I modified the query to use PLAN_STAT in trace

DBMS_MONITOR.SESSION_TRACE_ENABLE ( waits=>true, plan_stat=>’ALL_EXECUTIONS’ );

And that was cool as it showed the stat for 100 rows fetched. The number of physical reads (pr) from stats lines I worked out from trace file

Total pr from stat for ssdtest table scan

cat mydb_ora_9530_test_with_tdash_ssdtester_noindex.trc | grep STAT | grep TDASH | awk '{print $13}' | sed -e 's/pr=//g' | awk '{SUM+=$0}END {print "Total pr from stat lines is ",SUM}'
Total pr from stat lines is  194142802

SELECT *
FROM
 TDASH WHERE OBJECT_ID = :B1


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.01       0.00          2         47          0           0
Execute    100      0.00       0.00          6         54          0           0
Fetch      100     13.62    6423.32  194142802  194831012          0         100
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total      201     13.63    6423.34  194142810  194831113          0         100

Which is what it says in tkprof report for this loop section

and Parse and Execute come from

PARSING IN CURSOR #13 len=42 dep=1 uid=96 oct=3 lid=96 tim=1327941028415359 hv=605892470 ad='43fde4d40' sqlid='78kxqdhk1ubvq'
SELECT * FROM TDASH WHERE OBJECT_ID = :B1
END OF STMT
PARSE #13:c=12001,e=7676,p=2,cr=47,cu=0,mis=1,r=0,dep=1,og=1,plh=0,tim=1327941028415357

cat mydb_ora_9530_test_with_tdash_ssdtester_noindex.trc | grep 'EXEC #13'|grep -v p=0
EXEC #13:c=0,e=5333,p=6,cr=54,cu=0,mis=0,r=0,dep=1,og=1,plh=1148949653,tim=1327941028420736

2 physical blocks from PARSE and 6 from EXEC line.

Overall elapsed time

PARSING IN CURSOR #12 len=226 dep=2 uid=0 oct=3 lid=0 tim=1327941028408229 hv=1011610568 ad='443ad4ef8' sqlid='9tgj4g8y4rwy8'
CLOSE #3:c=0,e=10,dep=0,type=0,tim=1327947451920359

Time = select 1327947451920359 - 1327941028408229 from dual;

1327947451920359-1327941028408229
---------------------------------
                       6423512130 
= 6423.5 sec

and a silly one from tkprof last line

    6423  elapsed seconds in trace file.

And the number of direct path reads from trace

cat mydb_ora_9530_test_with_tdash_ssdtester_noindex.trc | grep 'direct path read'|wc -l
total = 1517298

cat mydb_ora_9530_test_with_tdash_ssdtester_noindex.trc | grep 'direct path read' | awk '{print $13}' |sed -e 's/cnt=//g' > 111

oracle@rhes564:/u01/app/oracle/diag/rdbms/mydb/mydb/trace% cat 111|awk '{SUM+=$0}END {print SUM}'
total blocks = 193906261
Waited = cat mydb_ora_9530_test_with_tdash_ssdtester_noindex.trc | grep 'direct path read' | awk '{print $7}' > 000
cat 000 | awk '{SUM+=$0}END {print SUM/1000000}'
6054.28 sec

See what tkprof says:

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  Disk file operations I/O                        3        0.00          0.00
  db file sequential read                         2        0.00          0.00
  direct path read                          1517298        0.07       6054.28
  asynch descriptor resize                      196        0.00          0.00
********************************************************************************

Now working out the total CPU time:

Total CPU time

cat mydb_ora_9530_test_with_tdash_ssdtester_noindex.trc | grep 'EXEC #' | cut -d "=" -f2 | cut -d "," -f1 | awk '{SUM+=$0}END {print "CPU in sec is ",SUM/1000000}'

CPU in sec is  18.2971

What does tkprof says on overall

OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute      2      4.65       4.79        317      48643          0           2
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        3      4.65       4.79        317      48643          0           2


OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse       10      0.01       0.00          2         47          0           0
Execute    130      0.00       0.01          6         55          2           1
Fetch      141     13.62    6423.32  194142810  194831113          0         141
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total      281     13.64    6423.35  194142818  194831215          2         142

And the sum = 4.65+ 13.64 = 18.29 sec

Which is what I get from the EXEC lines in trace file (CPU in sec is 18.2971) and it confirms that RECURSIVE matrices do not include NON-RECURSIVE stats.

It shows that if you suspect tkprof is not giving the correct info, the best place to look is the trace file and if you have a little shell routine you can get every detail from trace file.

Regards,

Mich

Thanks for showing how to use Unix/Linux OS tools to parse the raw 10046 trace files.

Oracle Database 11.1.0.6 and later by default will output the STAT lines (the Row Source Operation execution plan) in a trace file only after the first execution of the SQL statement (can you see where those lines appeared in your trace, it should be shortly after the first EXEC). Earlier versions would only output the STAT lines when the cursor was closed. That likely explains why the TKPROF Row Source Operation execution plan shows cr=1948310 pr=1941430 and not cr=194831114 pr=194142806.

You can control the frequency at which the STAT lines are written with the help of the new PLAN_STAT parameter for the DBMS_MONITOR.SESSION_TRACE_ENABLE call, but there does not appear to be an option to revert back to the pre-11.1.0.6 STAT writing method:
http://docs.oracle.com/cd/B28359_01/appdev.111/b28419/d_monitor.htm#i1003679

Thanks for details. I saw the OP questions in OTN and wondered about the points.

I do agree with your points that the trace file is best to analyze for details. Also if one uses sed and awk with trace one can pretty get the results quickly including sums!

For example I was looking at the number of waits due to direct path reads for a full table scan of order of 1,726,920 rows. What I got from tkprof report was:

SELECT *
FROM
 TDASH WHERE OBJECT_ID = :B1


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          1         48          0           0
Execute    100      0.00       0.00          3         54          0           0
Fetch      100     11.99   11808.99  194142802  194831012          0         100
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total      201     12.00   11809.00  194142806  194831114          0         100

Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 93     (recursive depth: 1)

Rows     Row Source Operation
-------  ---------------------------------------------------
      1  TABLE ACCESS FULL TDASH (cr=1948310 pr=1941430 pw=0 time=0 us cost=526948 size=8092 card=1)


Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  Disk file operations I/O                        3        0.00          0.00
  db file sequential read                         2        0.01          0.02
  direct path read                          1468497        0.66      11397.13
  asynch descriptor resize                      196        0.00          0.00
********************************************************************************

Now let me double check that “Total Waited” for direct path read from the trace file itself
A “typical wait” for direct path in trace shows:

WAIT #13: nam='direct path read' ela= 7509 file number=13 first dba=544512 block cnt=128 obj#=75856 tim=1327880589449318

The block size count may vary from 16 to 120, 126 and 128.

So roughly we expect 7509 microseconds elapsed time for each wait

cat mydb_ora_30587_test_with_tdash_tester_noindex.trc | grep 'direct path read' | wc -l
1468497 

OK we have 1468497 waits in this case that agrees with tkprof report

Get the elapsed time

cat mydb_ora_30587_test_with_tdash_tester_noindex.trc | grep 'direct path read' | awk '{print $7}' > 000

Sum the elapsed time

cat ooo | awk '{SUM+=$0}END {print SUM/1000000}'

11397.1 sec

So that is what tkprof is showing

Sum the number of blocks read

cat mydb_ora_30587_test_with_tdash_tester_noindex.trc | grep 'direct path' | grep 75856 | awk '{print $13}' | sed -e 's/cnt=//g' > 111
cat 111|awk '{SUM+=$0}END {print SUM}'
187631839

So we read 187,631,839 blocks via direct path read. the stat shows

Rows     Row Source Operation
-------  ---------------------------------------------------
      1  TABLE ACCESS FULL TDASH (cr=1948310 pr=1941430 pw=0 time=0 us cost=526948 size=8092 card=1)

Now the way I interpret this is that each row resulted in 1948310 buffer reads that included 1,941,430 physical block reads. In my case I read the first 100 rows from the table and hence would expect 194,831,000 buffers reads and 194,143,000 physical block reads. Let us check against tkprof report. It shows 194142806 physical block reads and 194831114 buffer reads as below.

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          1         48          0           0
Execute    100      0.00       0.00          3         54          0           0
Fetch      100     11.99   11808.99  194142802  194831012          0         100
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total      201     12.00   11809.00  194142806  194831114          0         100

So in summary it is really a bit of intution I guess.

I found this article very useful. Thanks for sharing your findings Charles

Regards,

Mich