So if I run the whole thing (UPDATES/DELETES/INSERTS) and look at the number of recursive statement for HDD I get

OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse    52982      0.23       0.47          3         19          0           0
Execute 105787    141.17    4308.08    3564902   12708368   43266845     3729575
Fetch    21254      0.22       3.72       4380      29023          0     1733335
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total   180023    141.63    4312.28    3569285   12737410   43266845     5462910

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file sequential read                   3260950        1.57       3660.65
  db file scattered read                      55729        0.57        244.64

    4321  elapsed seconds in trace file.

That 4,321 elapsed seconds includes 4,312.28 sec from recursive statements plus 9.15 sec for non-recursive statements (not shown). These recursive statements occurred because of dynamic storage extension. Dynamic storage extension occurred in our case because table testwrites and its index had to extend beyond their allocated space (that is, new extents were allocated). The wait times included 3,660.65 sec for db file sequential reads and 244.64 sec waits for db file scattered reads.

Now this summary seems to be correct. However, it typically does not answer (at least directly) the throughput of writes. Not suprising noone waits for writes. So I wanted to devise a mechanisom for measuring writes (exclusing ORION or other scripts). I thought abou it and decided to use AWR for this purpose. Went back to my original script and used checkpoint prior and following PL/SQL block and with snapshot

ALTER SYSTEM CHECKPOINT;
EXEC DBMS_WORKLOAD_REPOSITORY.CREATE_SNAPSHOT();
DECLARE
        type ObjIdArray is table of tdash.object_id%TYPE index by binary_integer;
        l_ids objIdArray;
        CURSOR c IS SELECT object_id FROM tdash;
BEGIN
        OPEN c;
        LOOP
                BEGIN
                        FETCH c BULK COLLECT INTO l_ids LIMIT 100;

                        FORALL rs in 1 .. l_ids.COUNT
                                UPDATE testwrites
                                  SET PADDING1 =  RPAD('y',4000,'y')
                                WHERE object_id = l_ids(rs);
                        commit;
                        FORALL rs in 1 .. l_ids.COUNT
                                DELETE FROM testwrites
                                WHERE object_id = l_ids(rs);
                        commit;
                        FORALL rs in 1 .. l_ids.COUNT
                                INSERT INTO testwrites
                                SELECT * FROM tdash t WHERE t.object_id = l_ids(rs);
                        commit;
                        EXIT WHEN C%NOTFOUND;
                EXCEPTION
                  WHEN NO_DATA_FOUND THEN
                    NULL;
                  WHEN OTHERS THEN
                    DBMS_OUTPUT.PUT_LINE('Transaction failed');
                END;
        END LOOP;
        CLOSE c;
END;
/
ALTER SYSTEM CHECKPOINT;
EXEC DBMS_WORKLOAD_REPOSITORY.CREATE_SNAPSHOT();

Now I can work out the volume of redos and writes generated (another topic for itself). Also for clarity with SSD tests redo files were on SSD. With HDD tests redo files were put on HDD. So after a bit of head scratching with tkprof output, I decided to look at the overall statistics for PL/SQL block. just compare the SQL execute times pretty much as they are taken at the highest level (like outside of the loop).

Cheers,

Mich

FYI, My version of tkprof is

TKPROF: Release 11.2.0.1.0 

SQL ID: 1ahnbczb20gdx
Plan Hash: 1751280057
UPDATE T2 SET PADDING1 = RPAD('y',4000,'y') 
WHERE
 OBJECT_ID = :B1 


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute    190      1.35       1.42        348      52552     165435       19000
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total      191      1.35       1.42        348      52552     165435       19000

Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 64     (recursive depth: 1)
Number of plan statistics captured: 1

Rows (1st) Rows (avg) Rows (max)  Row Source Operation
---------- ---------- ----------  ---------------------------------------------------
         0          0          0  UPDATE  T2 (cr=332 pr=30 pw=0 time=19223 us)
       100        100        100   INDEX UNIQUE SCAN IND_T2_OBJ (cr=105 pr=8 pw=0 time=1372 us cost=1 size=2015 card=1)(object id 103855)
         0          0          0  UPDATE  T2 (cr=288 pr=2 pw=0 time=8629 us)
       100        100        100   INDEX UNIQUE SCAN IND_T2_OBJ (cr=105 pr=0 pw=0 time=430 us cost=1 size=2015 card=1)(object id 103855)
         0          0          0  UPDATE  T2 (cr=294 pr=2 pw=0 time=8961 us)
       100        100        100   INDEX UNIQUE SCAN IND_T2_OBJ (cr=105 pr=0 pw=0 time=438 us cost=1 size=2015 card=1)(object id 103855)
...
         0          0          0  UPDATE  T2 (cr=294 pr=0 pw=0 time=6507 us)
       100        100        100   INDEX UNIQUE SCAN IND_T2_OBJ (cr=111 pr=0 pw=0 time=440 us cost=1 size=2015 card=1)(object id 103855)
         0          0          0  UPDATE  T2 (cr=291 pr=0 pw=0 time=6051 us)
       100        100        100   INDEX UNIQUE SCAN IND_T2_OBJ (cr=105 pr=0 pw=0 time=430 us cost=1 size=2015 card=1)(object id 103855)


Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file scattered read                         40        0.00          0.03
  db file sequential read                        32        0.00          0.01

And the output from my program:

Statement Depth 1 (Trigger Code)
Cursor 910   Ver 1   Parse at 0.187068 SQL_ID='1ahnbczb20gdx'  (TD Prev 0.000996)  Similar Cnt 1
|PARSEs       1|CPU S    0.000000|CLOCK S    0.000162|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|EXECs      600|CPU S    3.603618|CLOCK S    3.683344|ROWs    50000|PHY RD BLKs       912|CON RD BLKs (Mem)    144826|CUR RD BLKs (Mem)    426496|SHARED POOL MISs      1|
|FETCHs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
  CPU S 16.91%  CLOCK S 15.45%
|                 +++||                 +++|
UPDATE T2 SET PADDING1 = RPAD('y',4000,'y') WHERE OBJECT_ID = :B1

       (Rows 0)   UPDATE  T2 (cr=332 pr=30 pw=0 time=19223 us)
     (Rows 100)    INDEX UNIQUE SCAN IND_T2_OBJ (cr=105 pr=8 pw=0 time=1372 us cost=1 size=2015 card=1)

       (Rows 0)   UPDATE  T2 (cr=288 pr=2 pw=0 time=8629 us)
     (Rows 100)    INDEX UNIQUE SCAN IND_T2_OBJ (cr=105 pr=0 pw=0 time=430 us cost=1 size=2015 card=1)

       (Rows 0)   UPDATE  T2 (cr=294 pr=2 pw=0 time=8961 us)
     (Rows 100)    INDEX UNIQUE SCAN IND_T2_OBJ (cr=105 pr=0 pw=0 time=438 us cost=1 size=2015 card=1)
...
       (Rows 0)   UPDATE  T2 (cr=105 pr=0 pw=0 time=221 us)
       (Rows 0)    INDEX UNIQUE SCAN IND_T2_OBJ (cr=105 pr=0 pw=0 time=174 us cost=1 size=2015 card=1)

       (Rows 0)   UPDATE  T2 (cr=105 pr=0 pw=0 time=193 us)
       (Rows 0)    INDEX UNIQUE SCAN IND_T2_OBJ (cr=105 pr=0 pw=0 time=153 us cost=1 size=2015 card=1)

Notice now that the elapsed time is just less than half of what my program outputs, while the numbers were very close when a full table scan was the only option. The same is true for the CPU time. I might have to experiment a bit more to see what is happening.

Sorry for the delay in responding.

It appears that what you are seeing is a potential bug in the version of tkprof that you are using, but the potential bug is unconfirmed at the moment. The version of tkprof is probably prior to 11.1.0.7. The potential bug appears to be related to attempting to subtract recursive child statistics from the associated from the parent UPDATE SQL statement statistics (the update statement has a dep=1, while the recursive child statistics will have a dep=2, great-grandchild will have a cursive dep=3, etc.).

I set up a test (60,000 rows in T1 from ALL_OBJECTS, 50,000 in T2) with Oracle Database 11.2.0.2, and the 32 bit Oracle Client 11.2.0.1 on Windows with patch 10155837 (fixes ODBC update problems related to bind variables), and found that that version of tkprof apparently correctly associates the parent and child statistics (I wonder about the grandchildren statitsics).

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute    600    197.38     197.60        452   50299237     426582       50000
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total      601    197.38     197.60        452   50299237     426582       50000
 
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 64     (recursive depth: 1)
Number of plan statistics captured: 1
 
Rows (1st) Rows (avg) Rows (max)  Row Source Operation
---------- ---------- ----------  ---------------------------------------------------
         0          0          0  UPDATE  T2 (cr=80409 pr=436 pw=0 time=357525 us)
       100        100        100   TABLE ACCESS FULL T2 (cr=80182 pr=420 pw=0 time=337295 us cost=116 size=16120 card=8)
         0          0          0  UPDATE  T2 (cr=83795 pr=0 pw=0 time=344133 us)
       100        100        100   TABLE ACCESS FULL T2 (cr=83598 pr=0 pw=0 time=332180 us cost=116 size=16120 card=8)
         0          0          0  UPDATE  T2 (cr=83790 pr=2 pw=0 time=303134 us)
       100        100        100   TABLE ACCESS FULL T2 (cr=83596 pr=0 pw=0 time=288815 us cost=116 size=16120 card=8)
         0          0          0  UPDATE  T2 (cr=83789 pr=2 pw=0 time=283738 us)
       100        100        100   TABLE ACCESS FULL T2 (cr=83595 pr=0 pw=0 time=271190 us cost=116 size=16120 card=8)
...
         0          0          0  UPDATE  T2 (cr=87900 pr=0 pw=0 time=375297 us)
         0          0          0   TABLE ACCESS FULL T2 (cr=87900 pr=0 pw=0 time=375015 us cost=116 size=16120 card=8)


Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file sequential read                        36        0.00          0.01
  db file scattered read                         31        0.00          0.01

For comparison, my program (Hyper-Extended Oracle Performance Monitor), does not subtract child recursive statistics from the parent statistics, so double-counting of statistics is very likely when dep=1 (or greater) statements are found in the trace file. My program output:

Statement Depth 1 (Trigger Code)
Cursor 220   Ver 1   Parse at 0.254283 SQL_ID='1ahnbczb20gdx'  (TD Prev 0.000877)  Similar Cnt 1
|PARSEs       1|CPU S    0.000000|CLOCK S    0.000169|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|EXECs      600|CPU S  197.403668|CLOCK S  197.659011|ROWs    50000|PHY RD BLKs       774|CON RD BLKs (Mem)  50299674|CUR RD BLKs (Mem)    426582|SHARED POOL MISs      1|
|FETCHs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
  CPU S 25.08%  CLOCK S 25.01%
|               +++++||               +++++|
UPDATE T2 SET PADDING1 = RPAD('y',4000,'y') WHERE OBJECT_ID = :B1

       (Rows 0)   UPDATE  T2 (cr=80409 pr=436 pw=0 time=357525 us)
     (Rows 100)    TABLE ACCESS FULL T2 (cr=80182 pr=420 pw=0 time=337295 us cost=116 size=16120 card=8)

       (Rows 0)   UPDATE  T2 (cr=83795 pr=0 pw=0 time=344133 us)
     (Rows 100)    TABLE ACCESS FULL T2 (cr=83598 pr=0 pw=0 time=332180 us cost=116 size=16120 card=8)

       (Rows 0)   UPDATE  T2 (cr=83790 pr=2 pw=0 time=303134 us)
     (Rows 100)    TABLE ACCESS FULL T2 (cr=83596 pr=0 pw=0 time=288815 us cost=116 size=16120 card=8)

       (Rows 0)   UPDATE  T2 (cr=83789 pr=2 pw=0 time=283738 us)
     (Rows 100)    TABLE ACCESS FULL T2 (cr=83595 pr=0 pw=0 time=271190 us cost=116 size=16120 card=8)

One of the outputs of my program is also an Excel spreadsheet with the wait events for each cursor. That spreadsheet confirms the tkprof output, that there were 31 db file scattered read waits (for a total of 416 blocks, all close to the start of the trace – Oracle was trying to fill the empty buffer cache) and 36 db file sequential read waits (36 total blocks). Notice also that the CPU time and elapsed times differ, in addition to the number of physical blocks read and consistent gets.

Which version of Oracle11g R2 Database are you using (11.2.0.1, 11.2.0.2, 11.2.0.3)?

Which version of the Oracle client are you using? You can determine the Oracle client by using the tnsping utility immedaitely after using tkprof:

C:\Oracle\diag\rdbms\or1122p\or1122p\trace>tnsping
 
TNS Ping Utility for 32-bit Windows: Version 11.2.0.1.0 - Production on 20-MAY-2012 11:00:31
 
Copyright (c) 1997, 2010, Oracle.  All rights reserved.
 
TNS-03502: Insufficient arguments.  Usage:  tnsping <address> [<count>]

db file sequential reads are single block IO’s. they are not 32 they are 1. db file scattered reads are the multi-block IO’s, they range from 2-7 blocks

Could this be exlained due to recursive sql

Briefly:

This is Oracle11g R2 on 8K block

I am running the following simple OLTP type tests. Selecting 1.7 million rows into an associateive array and updating another table where object_id match (via index key)

Table tdash has 1.7 Million rows based on all_objects + padding1 varchar(4000) + padding2 varchar(4000). There is a unique index on object_id
Table testwrites was created as CREATE TABLE testwrites AS SELECT * FROM tdash WHERE ROWNUM <=1000000. A unque index was created on object_id of testwrites table.

The Pl/SQL block

EXEC DBMS_MONITOR.SESSION_TRACE_ENABLE ( waits=>true, plan_stat=>'ALL_EXECUTIONS' );
DECLARE
        type ObjIdArray is table of tdash.object_id%TYPE index by binary_integer;
        l_ids objIdArray;
        CURSOR c IS SELECT object_id FROM tdash;
BEGIN
        OPEN c;
        LOOP
                BEGIN
                        FETCH c BULK COLLECT INTO l_ids LIMIT 100;

                        FORALL rs in 1 .. l_ids.COUNT
                                UPDATE testwrites
                                  SET PADDING1 =  RPAD('y',4000,'y')
                                WHERE object_id = l_ids(rs);
                        commit;
                        FORALL rs in 1 .. l_ids.COUNT
                                DELETE FROM testwrites
                                WHERE object_id = l_ids(rs);
                        commit;
                        FORALL rs in 1 .. l_ids.COUNT
                                INSERT INTO testwrites
                                SELECT * FROM tdash t WHERE t.object_id = l_ids(rs);
                        commit;
                        EXIT WHEN C%NOTFOUND;
                EXCEPTION
                  WHEN NO_DATA_FOUND THEN
                    NULL;
                  WHEN OTHERS THEN
                    DBMS_OUTPUT.PUT_LINE('Transaction failed');
                END;
        END LOOP;
        CLOSE c;
END;

TKPROF output for update only shows:

UPDATE TESTWRITES SET PADDING1 = RPAD('y',4000,'y')
WHERE
OBJECT_ID = :B1
 
 
call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        4      0.00       0.00          0          0          0           0
Execute  17293     21.62    1245.41    1064667    2411398    6280588     1000000
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total    17297     21.62    1245.41    1064667    2411398    6280588     1000000
 
Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 93     (recursive depth: 1)
 
Rows     Row Source Operation
-------  ---------------------------------------------------
      0  UPDATE  TESTWRITES (cr=214 pr=427 pw=0 time=0 us)
    100   INDEX UNIQUE SCAN TESTWRITES_PK (cr=110 pr=3 pw=0 time=0 us cost=2 size=4007 card=1)(object id 92380)
 
 
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  Disk file operations I/O                        2        0.00          0.00
  db file sequential read                   2160263        1.57       2304.22
  db file scattered read                      27769        0.26         99.78
  
*******************************************************************************
 
SQL ID: 8ggw94h7mvxd7
Plan Hash: 0
COMMIT
 
 
call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse    51879      0.16       0.35          0          0          0           0
Execute  51879      1.82       6.29          0          0      37301           0
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total   103758      1.98       6.65          0          0      37301           0
 
Misses in library cache during parse: 0
Parsing user id: 93     (recursive depth: 1)
 
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  log file switch completion                      1        0.05          0.05
  db file sequential read                    121686        1.39        162.95
  log buffer space                                2        0.98          0.99
  log file switch (private strand flush incomplete)
                                                  1        1.47          1.47
********************************************************************************

Now why my elapsed times for updates from wait events 2304.22+ 99.78 = 2404 sec is HIGHER than my elapsed time of 1245.21 sec? I don’t think it can be explained purely due to different clock granularities. Additionally,

Also I have “db file sequential read” = 2,160,263 physical IOs and from the rsaw trace file one physical IO in this case is 32 blocks. I also have “db file scattered read” = 27,769 physical IOs. In contrast the number of my disk reads is 1,064,667 blocks which is far less that sum of physical IOs. Something is not right somewhere.

Thanks,

Mich

I have been struggling to find time lately to write anything on this blog (I am suffering a bit from lack of ideas), or even to look through message threads on OTN or read blogs.

That is an interesting problem that you mention. If you look in the raw trace file, do you see any waits associated with CURSOR #0? Another thought is that SQL statements at recursive dep=1 or dep=2 might be affecting the tkprof output.

I will hopefully have some time later today – it might make for an interesting blog article to explore an example of the wait time being greater than the elapsed time. How large is the trace file?

I have an output from tkprof that Elapsed time from wait events is higher than “elapsed” in tkprof output. However, I do not know where in your blog I can post it. So aplogies for this. A comment has been it is due to using different clock granularities. Please advise if you are interested in me posting and where.

Thanks,

Mich