To N or Not to N, is SYS_OP_C2C the Question; Oracle NVARCHAR Slow?

February 8, 2015

I was recently contacted about an odd performance issue with a software product named TDM, where certain SQL statements submitted to an Oracle 9.2.x database required roughly 30 seconds to execute when a nearly instantaneous response was expected.  The person who reported the issue to me provided a sample SQL statement that appeared something similar to the following:

SELECT
  DATA
FROM 
  MVIS_DATA 
WHERE 
  TOOLID = N'112' 
  AND DATATYPE = 0 
  AND COMMISSIONID IS NULL 
  AND OPERATIONID IS NULL 
  AND COMMISSIONLISTPOS IS NULL;

In the SQL statement, notice the N character that is immediately before ‘112’ in the SQL statement.  The person indicated that the SQL statement executed quickly if that N character were removed from the SQL statement.  At this time the developer of that application is unwilling to release a bug fix to remove the N character from this (and likely other) SQL statements.

I did not initially have the table datatype descriptions (retrieved with DESC MVIS_DATA), so I made a couple of guesses about the datatypes.  What if the TOOLID column was defined as a number, and is it the primary key column for the table (indicating that there must be an index on that column)?  It might be the case that the developer of the application decided that in all SQL statements that are submitted with literal values (rather than using bind variables), that all numbers would be submitted in single quotes.  I created a testing table for a mock up in Oracle Database 11.2.0.2:

CREATE TABLE MVIS_DATA_NUM (
  TOOLID NUMBER,
  DATATYPE NUMBER,
  COMMISSIONID NUMBER,
  OPERATIONID NUMBER,
  COMMISSIONLISTPOS NUMBER,
  DATA VARCHAR2(100),
  PRIMARY KEY (TOOLID));
 
INSERT INTO
  MVIS_DATA_NUM
SELECT
  ROWNUM TOOLID,
  MOD(ROWNUM,2) DATATYPE,
  NULL COMMISSIONID,
  DECODE(MOD(ROWNUM,2),0,NULL,MOD(ROWNUM,2)) OPERATIONID,
  DECODE(MOD(ROWNUM,2),0,NULL,MOD(ROWNUM,2)) COMMISSIONLISTPOS,
  LPAD('A',100,'A') DATA
FROM
  DUAL
CONNECT BY
  LEVEL<=100000;
 
COMMIT;
 
EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'MVIS_DATA_NUM',CASCADE=>TRUE,NO_INVALIDATE=>FALSE)

With the testing table created with 100,000 rows, and statistics gathered for the table and primary key index, I then tried executing a query and retrieving the execution plan for that query so that I could determine if the Predicate Information section of the execution plan provided any clues.  I executed the following, the first SQL statement retrieved one row, and the second SQL statement retrieved the execution plan for the first SQL statement:

SET LINESIZE 140
SET PAGESIZE 1000
 
SELECT
  DATA
FROM 
  MVIS_DATA_NUM
WHERE 
  TOOLID = N'112' 
  AND DATATYPE = 0 
  AND COMMISSIONID IS NULL 
  AND OPERATIONID IS NULL 
  AND COMMISSIONLISTPOS IS NULL;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

The execution plan output is as follows:

SQL_ID  gzzrppktqkbmu, child number 0
-------------------------------------
SELECT   DATA FROM   MVIS_DATA_NUM WHERE   TOOLID = N'112'   AND
DATATYPE = 0   AND COMMISSIONID IS NULL   AND OPERATIONID IS NULL   AND
COMMISSIONLISTPOS IS NULL
 
Plan hash value: 1080991
 
---------------------------------------------------------------------------------------------
| Id  | Operation                   | Name          | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |               |       |       |     2 (100)|          |
|*  1 |  TABLE ACCESS BY INDEX ROWID| MVIS_DATA_NUM |     1 |   113 |     2   (0)| 00:00:01 |
|*  2 |   INDEX UNIQUE SCAN         | SYS_C0050817  |     1 |       |     1   (0)| 00:00:01 |
---------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter(("OPERATIONID" IS NULL AND "COMMISSIONLISTPOS" IS NULL AND
              "DATATYPE"=0 AND "COMMISSIONID" IS NULL))
   2 - access("TOOLID"=112)

Nothing too unusual in the above execution plan, the N’112′ portion of the SQL statement was simply changed by the query optimizer to the number 112, which permitted the use of the table’s primary key index.  Obviously, Oracle Database 11.2.0.2 contains a few optimizations that are not available in Oracle Database 9.2.x, so maybe the outcome would be different in Oracle Database 9.2.x.  However, when a number value is compared to a character (for example VARCHAR2) value, Oracle Database will attempt to implicitly convert the character value to a number value when performing the comparison, so the outcome should be the same on Oracle Database 9.2.x.

What if that TOOLID column were defined as VARCHAR?  Below is another test table with that column defined as VARCHAR2:

CREATE TABLE MVIS_DATA (
  TOOLID VARCHAR2(15),
  DATATYPE NUMBER,
  COMMISSIONID NUMBER,
  OPERATIONID NUMBER,
  COMMISSIONLISTPOS NUMBER,
  DATA VARCHAR2(100),
  PRIMARY KEY (TOOLID));
 
INSERT INTO
  MVIS_DATA
SELECT
  TO_CHAR(ROWNUM) TOOLID,
  MOD(ROWNUM,2) DATATYPE,
  NULL COMMISSIONID,
  DECODE(MOD(ROWNUM,2),0,NULL,MOD(ROWNUM,2)) OPERATIONID,
  DECODE(MOD(ROWNUM,2),0,NULL,MOD(ROWNUM,2)) COMMISSIONLISTPOS,
  LPAD('A',100,'A') DATA
FROM
  DUAL
CONNECT BY
  LEVEL<=100000;
 
COMMIT;

EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'MVIS_DATA',CASCADE=>TRUE,NO_INVALIDATE=>FALSE)

With the new test table created, let’s try the SQL statement again.  A 10053 trace file will be enabled in the event that you are interested in examining any potential automatic transformations of the SQL statement:

SET LINESIZE 140
SET PAGESIZE 1000
 
ALTER SESSION SET TRACEFILE_IDENTIFIER = 'SQL_10053V';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';
 
SELECT
  DATA
FROM 
  MVIS_DATA 
WHERE 
  TOOLID = N'112' 
  AND DATATYPE = 0 
  AND COMMISSIONID IS NULL 
  AND OPERATIONID IS NULL 
  AND COMMISSIONLISTPOS IS NULL;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));
 
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT OFF';

The first of the above SQL statements output one row.  Here is the execution plan that was output:

SQL_ID  5pkwzs079jwu2, child number 0
-------------------------------------
SELECT   DATA FROM   MVIS_DATA WHERE   TOOLID = N'112'   AND DATATYPE =
0   AND COMMISSIONID IS NULL   AND OPERATIONID IS NULL   AND
COMMISSIONLISTPOS IS NULL
 
Plan hash value: 353063534
 
-------------------------------------------------------------------------------
| Id  | Operation         | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |           |       |       |   227 (100)|          |
|*  1 |  TABLE ACCESS FULL| MVIS_DATA |   122 | 13908 |   227   (3)| 00:00:01 |
-------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter(("OPERATIONID" IS NULL AND "COMMISSIONLISTPOS" IS NULL
              AND "DATATYPE"=0 AND SYS_OP_C2C("TOOLID")=U'112' AND "COMMISSIONID" IS
              NULL))

Notice the SYS_OP_C2C function in the Predicate Information section of the execution plan, that is a clue that there might be a performance problem lurking.  Also notice that the INDEX UNIQUE SCAN operation was replaced with a TABLE ACCESS FULL operation, that is also a clue that a performance problem may be lurking.  This section of the execution plan also indicates that the N’112′ portion of the SQL statement was changed to U’112′.  Consulting the 10053 trace file indicates that the query optimizer rewrote the submitted SQL statement to the following:

SELECT
  "MVIS_DATA"."DATA" "DATA"
FROM
  "TESTUSER"."MVIS_DATA" "MVIS_DATA"
WHERE
  SYS_OP_C2C("MVIS_DATA"."TOOLID")=U'112'
  AND "MVIS_DATA"."DATATYPE"=0
  AND "MVIS_DATA"."COMMISSIONID" IS NULL
  AND "MVIS_DATA"."OPERATIONID" IS NULL
  AND "MVIS_DATA"."COMMISSIONLISTPOS" IS NULL

SYS_OP_C2C is an internal characterset conversion function.

What happens to the execution plan if the N character is removed from the SQL statement?

SELECT
  DATA
FROM 
  MVIS_DATA 
WHERE 
  TOOLID = '112' 
  AND DATATYPE = 0 
  AND COMMISSIONID IS NULL 
  AND OPERATIONID IS NULL 
  AND COMMISSIONLISTPOS IS NULL;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

Below is the execution plan that was output:

SQL_ID  d70jxj3ypy60g, child number 0
-------------------------------------
SELECT   DATA FROM   MVIS_DATA WHERE   TOOLID = '112'   AND DATATYPE =
0   AND COMMISSIONID IS NULL   AND OPERATIONID IS NULL   AND
COMMISSIONLISTPOS IS NULL
 
Plan hash value: 1051843381
 
--------------------------------------------------------------------------------------------
| Id  | Operation                   | Name         | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |              |       |       |     2 (100)|          |
|*  1 |  TABLE ACCESS BY INDEX ROWID| MVIS_DATA    |     1 |   114 |     2   (0)| 00:00:01 |
|*  2 |   INDEX UNIQUE SCAN         | SYS_C0050814 |     1 |       |     1   (0)| 00:00:01 |
--------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter(("OPERATIONID" IS NULL AND "COMMISSIONLISTPOS" IS NULL AND
              "DATATYPE"=0 AND "COMMISSIONID" IS NULL))
   2 - access("TOOLID"='112')

Notice that the SYS_OP_C2C function does not appear in the Predicate Information section of the execution plan this time, and that the primary key index is used, rather than requiring a full table scan.  Unfortunately, the DBMS_XPLAN.DISPLAY_CURSOR function does not exist in Oracle Database 9.2.0.x, otherwise the reason for the performance problem may have been much more readily apparent to the person who reported the issue to me.

So, what is the purpose of that N character in the SQL statement?  I recall seeing SQL statements similar to this one in the past, which converts a character string to a date:

SELECT DATE'2015-02-08' FROM DUAL;
 
DATE'2015
---------
08-FEB-15

After a fair amount of digging through the Oracle documentation, I located the following note about that N character:

“The TO_NCHAR function converts the data at run time, while the N function converts the data at compilation time.”

Interesting.  That quote suggests that the author of the SQL statement may have been trying to convert ‘112’ to a NVARCHAR2 (or NCHAR).  Time for another test, the below script creates a table with the TOOLID column defined as NVARCHAR2, populates the table with 100,000 rows, and then collects statistics on the table and its primary key index:

CREATE TABLE MVIS_DATA_N (
  TOOLID NVARCHAR2(15),
  DATATYPE NUMBER,
  COMMISSIONID NUMBER,
  OPERATIONID NUMBER,
  COMMISSIONLISTPOS NUMBER,
  DATA VARCHAR2(100),
  PRIMARY KEY (TOOLID));
 
INSERT INTO
  MVIS_DATA_N
SELECT
  TO_CHAR(ROWNUM) TOOLID,
  MOD(ROWNUM,2) DATATYPE,
  NULL COMMISSIONID,
  DECODE(MOD(ROWNUM,2),0,NULL,MOD(ROWNUM,2)) OPERATIONID,
  DECODE(MOD(ROWNUM,2),0,NULL,MOD(ROWNUM,2)) COMMISSIONLISTPOS,
  LPAD('A',100,'A') DATA
FROM
  DUAL
CONNECT BY
  LEVEL<=100000;
 
COMMIT;
 
EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'MVIS_DATA_N',CASCADE=>TRUE,NO_INVALIDATE=>FALSE)

Executing the query against this table also returns one row:

SELECT
  DATA
FROM 
  MVIS_DATA_N 
WHERE 
  TOOLID = N'112' 
  AND DATATYPE = 0 
  AND COMMISSIONID IS NULL 
  AND OPERATIONID IS NULL 
  AND COMMISSIONLISTPOS IS NULL;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

The execution plan follows:

SQL_ID  1yuzz9rqkvnpv, child number 0
-------------------------------------
SELECT   DATA FROM   MVIS_DATA_N WHERE   TOOLID = N'112'   AND DATATYPE
= 0   AND COMMISSIONID IS NULL   AND OPERATIONID IS NULL   AND
COMMISSIONLISTPOS IS NULL
 
Plan hash value: 1044325464
 
--------------------------------------------------------------------------------------------
| Id  | Operation                   | Name         | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |              |       |       |     2 (100)|          |
|*  1 |  TABLE ACCESS BY INDEX ROWID| MVIS_DATA_N  |     1 |   119 |     2   (0)| 00:00:01 |
|*  2 |   INDEX UNIQUE SCAN         | SYS_C0050815 |     1 |       |     1   (0)| 00:00:01 |
--------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter(("OPERATIONID" IS NULL AND "COMMISSIONLISTPOS" IS NULL AND
              "DATATYPE"=0 AND "COMMISSIONID" IS NULL))
   2 - access("TOOLID"=U'112')

Notice in the above Predicate Information section that the SYS_OP_C2C function does not appear, and the N’112′ portion of the SQL statement was still changed to U’112′.  The execution plan also shows that the primary key index was used, while a full table scan was required when the TOOLID column was defined as a VARCHAR2.

The person who reported the issue to me later provide the output of DESC MVIS_DATA, which indicated that the TOOLID column was in fact defined as a VARCHAR2 column.  If this person were running a more recent version of Oracle Database, he might be able to create a function based index that uses the SYS_OP_C2C function on the TOOLID column.  Such an index might look something like this:

CREATE INDEX IND_TOOLID_FIX ON MVIS_DATA (SYS_OP_C2C("TOOLID"));

Gathering statistics on the table and its indexes, executing the original SQL statement, and outputting the execution plan:

EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'MVIS_DATA',CASCADE=>TRUE,NO_INVALIDATE=>FALSE)
 
SELECT
  DATA
FROM 
  MVIS_DATA 
WHERE 
  TOOLID = N'112' 
  AND DATATYPE = 0 
  AND COMMISSIONID IS NULL 
  AND OPERATIONID IS NULL 
  AND COMMISSIONLISTPOS IS NULL;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

The first of the above queries output the expected one row, while the second query output the following execution plan:

SQL_ID  5pkwzs079jwu2, child number 1
-------------------------------------
SELECT   DATA FROM   MVIS_DATA WHERE   TOOLID = N'112'   AND DATATYPE =
0   AND COMMISSIONID IS NULL   AND OPERATIONID IS NULL   AND
COMMISSIONLISTPOS IS NULL
 
Plan hash value: 1497912695
 
----------------------------------------------------------------------------------------------
| Id  | Operation                   | Name           | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |                |       |       |     2 (100)|          |
|*  1 |  TABLE ACCESS BY INDEX ROWID| MVIS_DATA      |     1 |   125 |     2   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | IND_TOOLID_FIX |     1 |       |     1   (0)| 00:00:01 |
----------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter(("OPERATIONID" IS NULL AND "COMMISSIONLISTPOS" IS NULL AND "DATATYPE"=0
              AND "COMMISSIONID" IS NULL))
   2 - access("MVIS_DATA"."SYS_NC00007$"=U'112')

In the Predicate Information section of the execution plan, notice the absence of the SYS_OP_C2C function on the TOOLID column – those values are pre-computed in the virtual column (SYS_NC00007$) created for the function based index.  An index range scan is reported in the execution plan, rather than an index unique scan (the function based index, when created, was not declared as unique), so the SQL statement should execute much faster than the roughly 30 seconds required by the SQL statement without the function based index.

So, what are the options that were mentioned above?

• Have the application programmer fix the SQL statements.
• Upgrade to a version of Oracle Database (if that version is supported by the application) that supports the SYS_OP_C2C function, and create a function based index using that function.
• If the TOOLID column only contains numbers, that column should be defined as NUMBER, rather than VARCHAR2.  Consider redefining that column as a NUMBER datatype.
• If that N character is always used when this column is referenced, that column probably should be defined as NVARCHAR2 rather than VARCHAR2.  Consider redefining that column as a NVARCHAR2 datatype.
• Consider that the application is working as designed, and that it is nice to receive 30 second breaks every now and then.
• Take a DUMP and share it with the application developer.  On second thought, such an approach may have limited success in helping to solve this problem.

Here is a little experiment with the DUMP function, which reveals Oracle’s internal representation of data – refer to the Internal datatypes and datatype codes in the Oracle documentation for help in decoding the Typ= values.

SELECT DUMP(112) A FROM DUAL;
 
A
---------------------
Typ=2 Len=3: 194,2,13
 
/* ------------------- */
SELECT DUMP(TO_CHAR(112)) A FROM DUAL;
 
A
---------------------
Typ=1 Len=3: 49,49,50
 
/* ------------------- */
SELECT DUMP('112') A FROM DUAL;
 
A
----------------------
Typ=96 Len=3: 49,49,50
 
/* ------------------- */
SELECT DUMP(N'112') A FROM DUAL;
 
A
----------------------------
Typ=96 Len=6: 0,49,0,49,0,50
 
/* ------------------- */
SELECT DUMP(SYS_OP_C2C('112'))  A FROM DUAL;
 
A
----------------------------
Typ=96 Len=6: 0,49,0,49,0,50
 
/* ------------------- */
SELECT DUMP(TO_NCHAR('112')) A FROM DUAL;
 
A
---------------------------
Typ=1 Len=6: 0,49,0,49,0,50

It is possibly interesting to note that the internal representation for N’112′ is CHAR (or NCHAR), while the internal representation for TO_NCHAR(‘112’) (and TO_NCHAR(112)) is VARCHAR2 (or NVARCHAR2).

—

This blog’s statistics indicate that the search engine search term Oracle NVARCHAR slow resulted in two page views of this blog yesterday.  I can’t help but wonder if the person who performed that search might have been helped by some of the above analysis.

FIRST_ROWS_n Optimizer Mode – What is Wrong with this Statement?

June 8, 2014

It has been nearly two years since I last wrote a review of an Oracle Database related book, although I have recently written reviews of two Microsoft Exchange Server 2013 books and a handful of security cameras in the last two years.  My copy of the second edition of the “Troubleshooting Oracle Performance” book arrived yesterday, so I have been spending some time reading the new edition and comparing it with a PDF version of the first edition.  My initial impressions of the second edition, based on the first 30 pages, are almost entirely positive, although I did find one glitch in the second edition so far.

At the top of page 27 is the following caution, which did not appear in the first edition of the book:

“As the previous example shows, the optimizer_mode column doesn’t show the right value for child number 1. In fact, the column shows FIRST_ROWS instead of FIRST_ROWS_1. The same behavior can be observed with FIRST_ROWS_10, FIRST_ROWS_100, and FIRST_ROWS_1000 as well.  This fact leads to the potential problem that even though the execution environment is different, the SQL engine doesn’t distinguish that difference. As a result, a child cursor might be incorrectly shared.”

So, what is wrong with the above statement?  The above statement, if true, would seem to indicate a bug in Oracle Database, one that has potentially serious performance side-effects; the opening paragraph of the “Troubleshooting Oracle Performance” book states, “Performance is not merely optional, though; it is a key property of an application,” so as an extension, possibly a performance problem should be addressed in a similar fashion as a bug.  I wrote the following into my notes about the above quoted caution section from the book:

This statement might be incorrect, the OPTIMIZER_ENV_HASH_VALUE column in V$SQL should show a different value if the OPTIMIZER_MODE changes from FIRST_ROWS_1 to FIRST_ROWS_1000 and the same SQL statement is executed a second time.

So, I set up a test case to determine whether the above caution quote from the book is correct, or if what I wrote into my notes about the book is accurate.  Actually, both of the above seemingly mutually exclusive statements are correct based on the results of my test case script.  So, how is that possible?  The key is whether or not the SQL statement’s cursor is flushed out of the library cache, or if the child cursor is somehow marked as unshareable (possibly due to statistics collection) between the two executions of the same SQL statement with different FIRST_ROWS_n OPTIMIZER_MODE parameter values.  The  OPTIMIZER_ENV_HASH_VALUE column in V$SQL will be different (at least in Oracle Database 11.2.0.1) if the OPTIMIZER_MODE changes from FIRST_ROWS_1 to FIRST_ROWS_1000 and a hard parse is required, but the change of the OPTIMIZER_MODE from the first value to the second is NOT sufficient to force that hard parse.

The statement found in the “Troubleshooting Oracle Performance” book is correct, so what is wrong with the statement?  If performance problems are considered bugs, then this particular issue seems to point to a bug in Oracle Database, where an opportunity to re-optimize a SQL statement is missed.  Possibly equally important to recognize is that the OPTIMIZER_ENV_HASH_VALUE that is found in V$SQL is NOT used to determine if a SQL statement must be re-optimized (at least in Oracle Database 11.2.0.1) because the changed OPTIMIZER_MODE resulted in a different OPTIMIZER_ENV_HASH_VALUE when a hard parse happens.

So, how is this seemingly minor hard parsing issue a potential problem?  Consider a case where Oracle’s query optimizer should predict that the cardinality of an operation will be 990 rows.  With the OPTIMIZER_MODE set to FIRST_ROWS_1000, the query optimizer will optimize the SQL statement just as if the OPTIMIZER_MODE were set to ALL_ROWS because the predicted cardinality is less than 1000.  So, the optimizer might correctly select to perform a full table scan; while with the OPTIMIZER_MODE set to FIRST_ROWS_1, the predicted cardinality for the same operation might be just 2 (or 1), thus leading to a possibly inefficient index access path if one exists.

If the quote on page 27 does not contain the glitch that I found in the first 30 pages of the book, where is the glitch?  I actually found two glitches in the first 27 pages of the book, but I will not mention those glitches at this time.

—

It might be interesting to see if later releases of Oracle Database actually do force a hard parse for a SQL statement when the OPTIMIZER_MODE changes from FIRST_ROWS_1 to FIRST_ROWS_1000. (or if two sessions have the different FIRST_ROWS_n OPTIMIZER_MODE settings, and each execute the same SQL statement). Here is the test script that I constructed:

DROP TABLE T1 PURGE;
 
SET LINESIZE 140
SET PAGESIZE 1000
SET TRIMSPOOL ON
 
SELECT
  VERSION
FROM
  V$INSTANCE;
 
CREATE TABLE T1 AS
SELECT
  ROWNUM C1,
  MOD(ROWNUM,500) C2
FROM
  DUAL
CONNECT BY
  LEVEL<=10000;
 
EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T1')
 
ALTER SYSTEM FLUSH SHARED_POOL;
 
ALTER SESSION SET OPTIMIZER_MODE=FIRST_ROWS_1;
 
SELECT C1 FROM T1 WHERE C2=2;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));
 
SELECT
  SQL_ID,
  CHILD_NUMBER,
  OPTIMIZER_MODE,
  OPTIMIZER_ENV_HASH_VALUE,
  PLAN_HASH_VALUE
FROM
  V$SQL
WHERE
  SQL_TEXT='SELECT C1 FROM T1 WHERE C2=2';
 
ALTER SESSION SET OPTIMIZER_MODE=FIRST_ROWS_1000;
 
SELECT C1 FROM T1 WHERE C2=2;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));
 
SELECT
  SQL_ID,
  CHILD_NUMBER,
  OPTIMIZER_MODE,
  OPTIMIZER_ENV_HASH_VALUE,
  PLAN_HASH_VALUE
FROM
  V$SQL
WHERE
  SQL_TEXT='SELECT C1 FROM T1 WHERE C2=2';
 
SELECT
  CHILD_NUMBER,
  NAME,
  VALUE
FROM
  V$SQL_OPTIMIZER_ENV
WHERE
  SQL_ID='bqx2tj39jw1f5'
  AND NAME='optimizer_mode'
ORDER BY
  NAME;
 
ALTER SYSTEM FLUSH SHARED_POOL;
 
ALTER SESSION SET OPTIMIZER_MODE=FIRST_ROWS_1000;
 
SELECT C1 FROM T1 WHERE C2=2;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));
 
SELECT
  SQL_ID,
  CHILD_NUMBER,
  OPTIMIZER_MODE,
  OPTIMIZER_ENV_HASH_VALUE,
  PLAN_HASH_VALUE
FROM
  V$SQL
WHERE
  SQL_TEXT='SELECT C1 FROM T1 WHERE C2=2';
 
ALTER SESSION SET OPTIMIZER_MODE=FIRST_ROWS_1;
 
SELECT C1 FROM T1 WHERE C2=2;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));
 
SELECT
  SQL_ID,
  CHILD_NUMBER,
  OPTIMIZER_MODE,
  OPTIMIZER_ENV_HASH_VALUE,
  PLAN_HASH_VALUE
FROM
  V$SQL
WHERE
  SQL_TEXT='SELECT C1 FROM T1 WHERE C2=2';
 
SELECT
  CHILD_NUMBER,
  NAME,
  VALUE
FROM
  V$SQL_OPTIMIZER_ENV
WHERE
  SQL_ID='bqx2tj39jw1f5'
  AND NAME='optimizer_mode'
ORDER BY
  NAME;

The script is broken into two halves, with an ALTER SYSTEM FLUSH SHARED_POOL; separating the two halves of the script.  The execution plan is displayed after each execution of the test SQL statement to show the optimizer’s predicted cardinality for the TABLE ACCESS FULL operation as well as the calculated cost and estimated number of bytes returned from that operation.  The query optimizer’s calculated cost for an operation could cause the execution plan to change, although such a change could not happen in this test case script.

Below is the output that I received on Oracle Database 11.2.0.1 for the first half of the script.  Note that I have removed excessive blank lines and the output of the test SQL statement.  Notice that the OPTIMIZER_ENV_HASH_VALUE is displayed as 1002285490 when starting with the FIRST_ROWS_1 OPTIMIZER_MODE:

SQL> ALTER SYSTEM FLUSH SHARED_POOL;
 
System altered.
 
SQL> ALTER SESSION SET OPTIMIZER_MODE=FIRST_ROWS_1;
 
Session altered.
 
SQL> SELECT C1 FROM T1 WHERE C2=2;
 
20 rows selected.
 
SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));
 
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
SQL_ID  bqx2tj39jw1f5, child number 0
-------------------------------------
SELECT C1 FROM T1 WHERE C2=2
 
Plan hash value: 3617692013
 
--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |     2 (100)|          |
|*  1 |  TABLE ACCESS FULL| T1   |     2 |    16 |     2   (0)| 00:00:01 |
--------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C2"=2)
 
SQL> SELECT
  2    SQL_ID,
  3    CHILD_NUMBER,
  4    OPTIMIZER_MODE,
  5    OPTIMIZER_ENV_HASH_VALUE,
  6    PLAN_HASH_VALUE
  7  FROM
  8    V$SQL
  9  WHERE
 10    SQL_TEXT='SELECT C1 FROM T1 WHERE C2=2';
 
SQL_ID        CHILD_NUMBER OPTIMIZER_ OPTIMIZER_ENV_HASH_VALUE PLAN_HASH_VALUE
------------- ------------ ---------- ------------------------ ---------------
bqx2tj39jw1f5            0 FIRST_ROWS               1002285490      3617692013
 
SQL> ALTER SESSION SET OPTIMIZER_MODE=FIRST_ROWS_1000;
 
Session altered.
 
SQL> SELECT C1 FROM T1 WHERE C2=2;
 
20 rows selected.
 
SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));
 
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
SQL_ID  bqx2tj39jw1f5, child number 0
-------------------------------------
SELECT C1 FROM T1 WHERE C2=2
 
Plan hash value: 3617692013
 
--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |     2 (100)|          |
|*  1 |  TABLE ACCESS FULL| T1   |     2 |    16 |     2   (0)| 00:00:01 |
--------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C2"=2)
 
SQL> SELECT
  2    SQL_ID,
  3    CHILD_NUMBER,
  4    OPTIMIZER_MODE,
  5    OPTIMIZER_ENV_HASH_VALUE,
  6    PLAN_HASH_VALUE
  7  FROM
  8    V$SQL
  9  WHERE
 10    SQL_TEXT='SELECT C1 FROM T1 WHERE C2=2';
 
SQL_ID        CHILD_NUMBER OPTIMIZER_ OPTIMIZER_ENV_HASH_VALUE PLAN_HASH_VALUE
------------- ------------ ---------- ------------------------ ---------------
bqx2tj39jw1f5            0 FIRST_ROWS               1002285490      3617692013
 
SQL>
SQL> SELECT
  2    CHILD_NUMBER,
  3    NAME,
  4    VALUE
  5  FROM
  6    V$SQL_OPTIMIZER_ENV
  7  WHERE
  8    SQL_ID='bqx2tj39jw1f5'
  9    AND NAME='optimizer_mode'
 10  ORDER BY
 11    NAME;
 
CHILD_NUMBER NAME                                     VALUE
------------ ---------------------------------------- -------------------------
           0 optimizer_mode                           first_rows_1

Below is the output that I received on Oracle Database 11.2.0.1 for the second half of the script.  Note that I have removed excessive blank lines and the output of the test SQL statement.  Notice that the OPTIMIZER_ENV_HASH_VALUE is displayed as 4271299772 (rather than 1002285490 as was seen above) when starting with the FIRST_ROWS_1000 OPTIMIZER_MODE:

SQL> ALTER SYSTEM FLUSH SHARED_POOL;
 
System altered.
 
SQL> ALTER SESSION SET OPTIMIZER_MODE=FIRST_ROWS_1000;
 
Session altered.
 
SQL> SELECT C1 FROM T1 WHERE C2=2;
 
20 rows selected.
 
SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));
 
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
SQL_ID  bqx2tj39jw1f5, child number 0
-------------------------------------
SELECT C1 FROM T1 WHERE C2=2
 
Plan hash value: 3617692013
 
--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |     3 (100)|          |
|*  1 |  TABLE ACCESS FULL| T1   |    20 |   160 |     3   (0)| 00:00:01 |
--------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C2"=2)
 
SQL> SELECT
  2    SQL_ID,
  3    CHILD_NUMBER,
  4    OPTIMIZER_MODE,
  5    OPTIMIZER_ENV_HASH_VALUE,
  6    PLAN_HASH_VALUE
  7  FROM
  8    V$SQL
  9  WHERE
 10    SQL_TEXT='SELECT C1 FROM T1 WHERE C2=2';
 
SQL_ID        CHILD_NUMBER OPTIMIZER_ OPTIMIZER_ENV_HASH_VALUE PLAN_HASH_VALUE
------------- ------------ ---------- ------------------------ ---------------
bqx2tj39jw1f5            0 FIRST_ROWS               4271299772      3617692013
 
SQL> ALTER SESSION SET OPTIMIZER_MODE=FIRST_ROWS_1;
 
Session altered.
 
SQL> SELECT C1 FROM T1 WHERE C2=2;
 
20 rows selected.
 
SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));
 
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
SQL_ID  bqx2tj39jw1f5, child number 0
-------------------------------------
SELECT C1 FROM T1 WHERE C2=2
 
Plan hash value: 3617692013
 
--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |     3 (100)|          |
|*  1 |  TABLE ACCESS FULL| T1   |    20 |   160 |     3   (0)| 00:00:01 |
--------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C2"=2)
 
SQL> SELECT
  2    SQL_ID,
  3    CHILD_NUMBER,
  4    OPTIMIZER_MODE,
  5    OPTIMIZER_ENV_HASH_VALUE,
  6    PLAN_HASH_VALUE
  7  FROM
  8    V$SQL
  9  WHERE
 10    SQL_TEXT='SELECT C1 FROM T1 WHERE C2=2';
 
SQL_ID        CHILD_NUMBER OPTIMIZER_ OPTIMIZER_ENV_HASH_VALUE PLAN_HASH_VALUE
------------- ------------ ---------- ------------------------ ---------------
bqx2tj39jw1f5            0 FIRST_ROWS               4271299772      3617692013
 
SQL> SELECT
  2    CHILD_NUMBER,
  3    NAME,
  4    VALUE
  5  FROM
  6    V$SQL_OPTIMIZER_ENV
  7  WHERE
  8    SQL_ID='bqx2tj39jw1f5'
  9    AND NAME='optimizer_mode'
 10  ORDER BY
 11    NAME;
 
CHILD_NUMBER NAME                                     VALUE
------------ ---------------------------------------- -------------------------
           0 optimizer_mode                           first_rows_1000

As repeatedly demonstrated in the “Troubleshooting Oracle Performance” book, testing theories is important.  In this case, I learned something new when what was mentioned in the book did not agree with what I recalled as being Oracle Database behavior.  I anticipate that the learning (or re-learning) process will continue as I quietly question the statements found in the book.  There is enough new material in the second edition of the book to make it a compulsive buy for people who already own the first edition (I own the print hard copy and the companion PDF versions of the first edition).

Parallel Parking – What is causing the Query to be Executed in Parallel?

July 31, 2012

I encountered an interesting OTN thread that caused me to stop and think for a couple of minutes.  The OP (original poster) in the OTN thread wondered why parallel execution was used for a query when the parallel degree for the table accessed by the query and its indexes were specified to have a parallel degree of 1.  For some reason, while the OP provided DBMS_XPLAN generated execution plan output, the Note section from the output was not included in the OTN post.  With parallel execution enabled in the session (ALTER SESSION ENABLE PARALLEL QUERY;) the execution plan showed a cost of 2,025.  With parallel execution disabled in the session (ALTER SESSION DISABLE PARALLEL QUERY;) the execution plan showed a cost of 36,504.  From this, we can determine that the parallel degree for the query is 20 (36504 / 2025 / 0.9 = 20.03).  The table is partitioned on a date column, apparently partitioned by month.

Interesting?

Could the parallel execution be caused by the PARALLEL_AUTOMATIC_TUNING parameter’s value?  The DBMS_XPLAN output indicates that the OP is using Oracle Database 11.2.0.2, where that parameter is deprecated.  What about the PARALLEL_DEGREE_POLICY parameter?  When that parameter is set to AUTO, the query optimizer is free to plan a parallel execution for a query, even when the tables (and their indexes) accessed by the query are set to a parallel degree of 1.

A table creation script for a couple of tests (note that the table created in the test case is using INTERVAL partitioning using an approach very similar to that found on page 576 in the book Expert Oracle Database Architecture: Oracle Database Programming 9i, 10g, and 11g Techniques and Solutions, Second Edition except that I have a DATE column in the table rather than a TIMESTAMP column):

DROP TABLE T1 PURGE;

CREATE TABLE T1 (
  C1 NUMBER,
  C2 VARCHAR2(10),
  C3 DATE,
  C4 VARCHAR2(20))
  PARTITION BY RANGE(C3)
  INTERVAL(NUMTOYMINTERVAL(1, 'MONTH'))
  ( PARTITION p0 VALUES LESS THAN (TO_DATE('1-1-2007', 'DD-MM-YYYY')));

INSERT INTO
  T1
SELECT
  ROWNUM,
  RPAD(ROWNUM,10,'A'),
  TRUNC(SYSDATE)+MOD(ROWNUM-1,3000),
  RPAD('A',20,'A')
FROM
  (SELECT
    1 C1
  FROM
    DUAL
  CONNECT BY
    LEVEL<100000),
  (SELECT
    1 C1
  FROM
    DUAL
  CONNECT BY
    LEVEL<1000);

COMMIT; 

Let’s see, a Cartesian join between a rowsource with 1,000 rows and a rowsource with 100,000 rows produces (on 11.2.0.1, 11.2.0.2, and 11.2.0.3):

99899001 rows created. 

Interesting, I might have to investigate that oddity at a later time.  I was expecting 1,000 * 100,000 = 100,000,000 rows.

Let’s try a simple test:

select count(*) from t1;

SET LINESIZE 140
SET PAGESIZE 1000

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

SQL_ID  5bc0v4my7dvr5, child number 0
-------------------------------------
select count(*) from t1

Plan hash value: 2705263620

-------------------------------------------------------------------------------------
| Id  | Operation            | Name | Rows  | Cost (%CPU)| Time     | Pstart| Pstop |
-------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT     |      |       | 59271 (100)|          |       |       |
|   1 |  SORT AGGREGATE      |      |     1 |            |          |       |       |
|   2 |   PARTITION RANGE ALL|      |   115M| 59271   (2)| 00:07:55 |     1 |1048575|
|   3 |    TABLE ACCESS FULL | T1   |   115M| 59271   (2)| 00:07:55 |     1 |1048575|
-------------------------------------------------------------------------------------

Note
-----
   - dynamic sampling used for this statement (level=2)

1,048,575 partitions… I was expecting about 100 partitions because there should be 3,000 distinct dates in column C3.  Maybe I should find some time to investigate?  Moving on…

To test the theory that the unexpected parallel execution might be caused by a non-default value for the PARALLEL_DEGREE_POLICY parameter, we need to first use the DBMS_RESOURCE_MANAGER.CALIBRATE_IO procedure to indicate performance metrics for the server.  To save time, I found a blog article (by another OakTable Network member) that shows a back-door approach that does not require the CALIBRATE_IO procedure (the usual warnings apply here – we are technically modifying objects in the SYS schema).  The back-door approach, when connected as the SYS user (note that the script first selects from RESOURCE_IO_CALIBRATE$ so that if values are currently present, those values may be restored after testing completes):

SELECT
  *
FROM
  RESOURCE_IO_CALIBRATE$;

DELETE FROM
  RESOURCE_IO_CALIBRATE$;

INSERT INTO
  RESOURCE_IO_CALIBRATE$
VALUES(
  CURRENT_TIMESTAMP,
  CURRENT_TIMESTAMP,
  0,
  0,
  200,
  0,
  0);

COMMIT; 

I will start by posting results that I obtained from Oracle Database 11.2.0.3.

Let’s try the previous select from T1 again (note that I am first setting the PARALLEL_DEGREE_POLICY parameter at the session level to its default value):

ALTER SESSION SET PARALLEL_DEGREE_POLICY=MANUAL;

select count(*) from t1;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR('5bc0v4my7dvr5',NULL,'TYPICAL'));

SQL_ID  5bc0v4my7dvr5, child number 0
-------------------------------------
select count(*) from t1

Plan hash value: 2705263620

-------------------------------------------------------------------------------------
| Id  | Operation            | Name | Rows  | Cost (%CPU)| Time     | Pstart| Pstop |
-------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT     |      |       | 59271 (100)|          |       |       |
|   1 |  SORT AGGREGATE      |      |     1 |            |          |       |       |
|   2 |   PARTITION RANGE ALL|      |   115M| 59271   (2)| 00:00:03 |     1 |1048575|
|   3 |    TABLE ACCESS FULL | T1   |   115M| 59271   (2)| 00:00:03 |     1 |1048575|
-------------------------------------------------------------------------------------

Note
-----
   - dynamic sampling used for this statement (level=2)

Let’s try again with a modified value for that parameter:

ALTER SESSION SET PARALLEL_DEGREE_POLICY=AUTO;

select count(*) from t1;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR('5bc0v4my7dvr5',NULL,'TYPICAL'));

SQL_ID  5bc0v4my7dvr5, child number 2
-------------------------------------
select count(*) from t1

Plan hash value: 974985148

------------------------------------------------------------------------------------------------------------------------
| Id  | Operation              | Name     | Rows  | Cost (%CPU)| Time     | Pstart| Pstop |    TQ  |IN-OUT| PQ Distrib |
------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT       |          |       | 32790 (100)|          |       |       |        |      |            |
|   1 |  SORT AGGREGATE        |          |     1 |            |          |       |       |        |      |            |
|   2 |   PX COORDINATOR       |          |       |            |          |       |       |        |      |            |
|   3 |    PX SEND QC (RANDOM) | :TQ10000 |     1 |            |          |       |       |  Q1,00 | P->S | QC (RAND)  |
|   4 |     SORT AGGREGATE     |          |     1 |            |          |       |       |  Q1,00 | PCWP |            |
|   5 |      PX BLOCK ITERATOR |          |   115M| 32790   (2)| 00:00:02 |     1 |1048575|  Q1,00 | PCWC |            |
|*  6 |       TABLE ACCESS FULL| T1       |   115M| 32790   (2)| 00:00:02 |     1 |1048575|  Q1,00 | PCWP |            |
------------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   6 - access(:Z>=:Z AND :Z<=:Z)

Note
-----
   - dynamic sampling used for this statement (level=2)
   - automatic DOP: Computed Degree of Parallelism is 2 

In the above, the Note section indicates that the automatic degree of parallelism was set to 2, and that dynamic sampling was performed at level 2.  I have yet to collect statistics on table T1, so that explains why dynamic sampling was used when the query was executed with and without parallel execution.  If you take a close look at the OTN thread that is linked to at the start of this blog article, you will see that the dynamic sampling level was set to 6, but only for the parallel execution.  The simple answer for this change in behavior is that it is expected, as described in this article by Maria Colgan (another OakTable Network member).  We can confirm that the cost calculation is working as expected: 59271 / 32790 / 0.9 = 2.01.

Let’s try executing the query of table T1 using a WHERE clause that is similar to what is found in the OTN thread:

ALTER SESSION SET PARALLEL_DEGREE_POLICY=MANUAL;

select count(*) from t1
WHERE C3 BETWEEN to_date('01/08/2012', 'dd/mm/yyyy') and to_date('14/10/2012', 'dd/mm/yyyy');

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL +OUTLINE'));

SQL_ID  1ggk1hy0r22kt, child number 0
-------------------------------------
select count(*) from t1 WHERE C3 BETWEEN to_date('01/08/2012',
'dd/mm/yyyy') and to_date('14/10/2012', 'dd/mm/yyyy')

Plan hash value: 2744578615

--------------------------------------------------------------------------------------------------
| Id  | Operation                 | Name | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
--------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT          |      |       |       |   235K(100)|          |       |       |
|   1 |  SORT AGGREGATE           |      |     1 |     9 |            |          |       |       |
|   2 |   PARTITION RANGE ITERATOR|      |  2286K|    19M|   235K(100)| 00:00:10 |    69 |    71 |
|*  3 |    TABLE ACCESS FULL      | T1   |  2286K|    19M|   235K(100)| 00:00:10 |    69 |    71 |
--------------------------------------------------------------------------------------------------

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('11.2.0.3')
      DB_VERSION('11.2.0.3')
      ALL_ROWS
      OUTLINE_LEAF(@"SEL$1")
      FULL(@"SEL$1" "T1"@"SEL$1")
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------
   3 - filter(("C3">=TO_DATE(' 2012-08-01 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND
              "C3"<=TO_DATE(' 2012-10-14 00:00:00', 'syyyy-mm-dd hh24:mi:ss')))

Note
-----
   - dynamic sampling used for this statement (level=2) 

Let’s try again, this time with the non-default value for the PARALLEL_DEGREE_POLICY:

ALTER SESSION SET PARALLEL_DEGREE_POLICY=AUTO;

select count(*) from t1
WHERE C3 BETWEEN to_date('01/08/2012', 'dd/mm/yyyy') and to_date('14/10/2012', 'dd/mm/yyyy');

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL +OUTLINE'));

SQL_ID  1ggk1hy0r22kt, child number 1
-------------------------------------
select count(*) from t1 WHERE C3 BETWEEN to_date('01/08/2012',
'dd/mm/yyyy') and to_date('14/10/2012', 'dd/mm/yyyy')

Plan hash value: 2946076746

--------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation              | Name     | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |    TQ  |IN-OUT| PQ Distrib |
--------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT       |          |       |       |   835K(100)|          |       |       |        |      |            |
|   1 |  SORT AGGREGATE        |          |     1 |     9 |            |          |       |       |        |      |            |
|   2 |   PX COORDINATOR       |          |       |       |            |          |       |       |        |      |            |
|   3 |    PX SEND QC (RANDOM) | :TQ10000 |     1 |     9 |            |          |       |       |  Q1,00 | P->S | QC (RAND)  |
|   4 |     SORT AGGREGATE     |          |     1 |     9 |            |          |       |       |  Q1,00 | PCWP |            |
|   5 |      PX BLOCK ITERATOR |          |   112M|   968M|   835K(100)| 00:00:33 |    69 |    71 |  Q1,00 | PCWC |            |
|*  6 |       TABLE ACCESS FULL| T1       |   112M|   968M|   835K(100)| 00:00:33 |    69 |    71 |  Q1,00 | PCWP |            |
--------------------------------------------------------------------------------------------------------------------------------

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('11.2.0.3')
      DB_VERSION('11.2.0.3')
      ALL_ROWS
      SHARED(16)
      OUTLINE_LEAF(@"SEL$1")
      FULL(@"SEL$1" "T1"@"SEL$1")
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------
   6 - access(:Z>=:Z AND :Z<=:Z)
       filter(("C3">=TO_DATE(' 2012-08-01 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "C3"<=TO_DATE(' 2012-10-14
              00:00:00', 'syyyy-mm-dd hh24:mi:ss')))

Note
-----
   - dynamic sampling used for this statement (level=2)
   - automatic DOP: Computed Degree of Parallelism is 16 because of degree limit 

Notice anything strange… really strange about the above output?  The new version of the query is limited to just 3 partitions, yet the calculated cost of the non-parallel execution is approximately 235,000 (much higher than the calculated cost of selecting from the full table without the WHERE clause) – that is just one reason why relying on cost for tuning performance purposes (as suggested by a couple of books) is potentially misleading.  But wait, the plan using parallel execution with an automatic degree of parallelism of 16 has a plan cost of approximately 835,000 and the optimizer still selected that plan!  I strongly suspect that a query optimizer bug caused a miscalculation of the plan cost; I could not reproduce the odd cost calculation on Oracle Database 11.2.0.1 or 11.2.0.2.  Still suspicious of the cost in the execution plan, at the end of testing I dropped and recreated the table.  After recreating the test table, the following execution plan was generated with a less crazy cost calculation:

SQL_ID  1ggk1hy0r22kt, child number 1
-------------------------------------
select count(*) from t1 WHERE C3 BETWEEN to_date('01/08/2012',
'dd/mm/yyyy') and to_date('14/10/2012', 'dd/mm/yyyy')

Plan hash value: 2946076746

--------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation              | Name     | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |    TQ  |IN-OUT| PQ Distrib |
--------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT       |          |       |       | 23822 (100)|          |       |       |        |      |            |
|   1 |  SORT AGGREGATE        |          |     1 |     9 |            |          |       |       |        |      |            |
|   2 |   PX COORDINATOR       |          |       |       |            |          |       |       |        |      |            |
|   3 |    PX SEND QC (RANDOM) | :TQ10000 |     1 |     9 |            |          |       |       |  Q1,00 | P->S | QC (RAND)  |
|   4 |     SORT AGGREGATE     |          |     1 |     9 |            |          |       |       |  Q1,00 | PCWP |            |
|   5 |      PX BLOCK ITERATOR |          |  2286K|    19M| 23822 (100)| 00:00:01 |    69 |    71 |  Q1,00 | PCWC |            |
|*  6 |       TABLE ACCESS FULL| T1       |  2286K|    19M| 23822 (100)| 00:00:01 |    69 |    71 |  Q1,00 | PCWP |            |
--------------------------------------------------------------------------------------------------------------------------------

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('11.2.0.3')
      DB_VERSION('11.2.0.3')
      ALL_ROWS
      SHARED(11)
      OUTLINE_LEAF(@"SEL$1")
      FULL(@"SEL$1" "T1"@"SEL$1")
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------
   6 - access(:Z>=:Z AND :Z<=:Z)
       filter(("C3">=TO_DATE(' 2012-08-01 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND "C3"<=TO_DATE(' 2012-10-14
              00:00:00', 'syyyy-mm-dd hh24:mi:ss')))

Note
-----
   - dynamic sampling used for this statement (level=2)
   - automatic DOP: Computed Degree of Parallelism is 11 

Let’s double-check the calculated cost in the parallel execution plan: 235,000 / 23822 / 0.9 = 10.96 – that calculated cost is consistent with the Computed Degree of Parallelism.

I am curious about whether or not dynamic sampling might affect just the parallel execution.  Let’s collect statistics on table T1 with 100% sampling (note that it could take a long time for this statistics collection to complete):

EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T1',ESTIMATE_PERCENT=>NULL)

Let’s try our query again (note that I have forced a hard parse of the SQL statement by changing the capitalization of a couple parts of the SQL statement):

ALTER SESSION SET PARALLEL_DEGREE_POLICY=MANUAL;

SELECT COUNT(*) from t1
WHERE C3 BETWEEN to_date('01/08/2012', 'dd/mm/yyyy') and to_date('14/10/2012', 'dd/mm/yyyy');

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL +OUTLINE'));

SQL_ID  2q2uz15vyzqsy, child number 1
-------------------------------------
SELECT COUNT(*) from t1 WHERE C3 BETWEEN to_date('01/08/2012',
'dd/mm/yyyy') and to_date('14/10/2012', 'dd/mm/yyyy')

Plan hash value: 2744578615

--------------------------------------------------------------------------------------------------
| Id  | Operation                 | Name | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
--------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT          |      |       |       |  1816 (100)|          |       |       |
|   1 |  SORT AGGREGATE           |      |     1 |     8 |            |          |       |       |
|   2 |   PARTITION RANGE ITERATOR|      |  2531K|    19M|  1816   (3)| 00:00:01 |    69 |    71 |
|*  3 |    TABLE ACCESS FULL      | T1   |  2531K|    19M|  1816   (3)| 00:00:01 |    69 |    71 |
--------------------------------------------------------------------------------------------------

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('11.2.0.3')
      DB_VERSION('11.2.0.3')
      ALL_ROWS
      OUTLINE_LEAF(@"SEL$1")
      FULL(@"SEL$1" "T1"@"SEL$1")
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------
   3 - filter(("C3"<=TO_DATE(' 2012-10-14 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND
              "C3">=TO_DATE(' 2012-08-01 00:00:00', 'syyyy-mm-dd hh24:mi:ss')))

Notice in the above that the execution plan for the non-parallel execution has a calculated cost that decreased from 235,000 to just 1,816, even though the same Pstart (69) and Pstop (71) columns are listed, and approximately the same estimated number of rows (~2,286,000 vs ~2,531,000) are expected to be returned.  So, if you are attempting to performance tune by attempting to simply reduce the calculated cost of a query (not a valid approach), you might be lead to believe that collecting statistics improved the performance of this query by a factor of 129.4 (the performance did improve a little due to dynamic sampling no longer being required, but the performance did not improve nearly that much).

Let’s try the query again with a modified value for the PARALLEL_DEGREE_POLICY parameter:

ALTER SESSION SET PARALLEL_DEGREE_POLICY=AUTO;

select COUNT(*) from t1
WHERE C3 BETWEEN to_date('01/08/2012', 'dd/mm/yyyy') and to_date('14/10/2012', 'dd/mm/yyyy');

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL +OUTLINE'));

SQL_ID  2q2uz15vyzqsy, child number 0
-------------------------------------
SELECT COUNT(*) from t1 WHERE C3 BETWEEN to_date('01/08/2012',
'dd/mm/yyyy') and to_date('14/10/2012', 'dd/mm/yyyy')

Plan hash value: 2744578615

--------------------------------------------------------------------------------------------------
| Id  | Operation                 | Name | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
--------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT          |      |       |       |  1816 (100)|          |       |       |
|   1 |  SORT AGGREGATE           |      |     1 |     8 |            |          |       |       |
|   2 |   PARTITION RANGE ITERATOR|      |  2531K|    19M|  1816   (3)| 00:00:01 |    69 |    71 |
|*  3 |    TABLE ACCESS FULL      | T1   |  2531K|    19M|  1816   (3)| 00:00:01 |    69 |    71 |
--------------------------------------------------------------------------------------------------

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('11.2.0.3')
      DB_VERSION('11.2.0.3')
      ALL_ROWS
      NO_PARALLEL
      OUTLINE_LEAF(@"SEL$1")
      FULL(@"SEL$1" "T1"@"SEL$1")
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------
   3 - filter(("C3"<=TO_DATE(' 2012-10-14 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND
              "C3">=TO_DATE(' 2012-08-01 00:00:00', 'syyyy-mm-dd hh24:mi:ss')))

Note
-----
   - automatic DOP: Computed Degree of Parallelism is 1 because of parallel threshold

The Note section indicates that parallel execution was considered, but not used.  So, after collecting statistics, the parallel degree for this query decreased from 16 (or 11 in the non-bug affected execution plan) to 1 – a non-parallel execution.

The above output was generated with Oracle Database 11.2.0.3 using the following system (CPU) statistics:

COLUMN PNAME FORMAT A14
COLUMN PVAL2 FORMAT A20

SELECT
  PNAME,
  PVAL1,
  PVAL2
FROM
  SYS.AUX_STATS$;

PNAME               PVAL1 PVAL2
-------------- ---------- ----------------
STATUS                    COMPLETED
DSTART                    07-11-2012 15:11
DSTOP                     07-11-2012 15:11
FLAGS                   1
CPUSPEEDNW     1720.20725
IOSEEKTIM              10
IOTFRSPEED           4096
SREADTIM                8
MREADTIM               10
CPUSPEED             2664
MBRC                   16
MAXTHR           19181568
SLAVETHR

—

While experimenting with Oracle Database 11.2.0.2, I used the following system (CPU) statistics:

COLUMN PNAME FORMAT A14
COLUMN PVAL2 FORMAT A20

SELECT
  PNAME,
  PVAL1,
  PVAL2
FROM
  SYS.AUX_STATS$;

PNAME               PVAL1 PVAL2
-------------- ---------- ----------------
STATUS                    COMPLETED
DSTART                    02-29-2012 19:07
DSTOP                     02-29-2012 19:07
FLAGS                   1
CPUSPEEDNW     2116.57559
IOSEEKTIM              10
IOTFRSPEED           4096
SREADTIM               .4
MREADTIM               .8
CPUSPEED             1922
MBRC                   16
MAXTHR          155910144
SLAVETHR

11.2.0.2 showed the following execution plan after collecting statistics for table T1 and setting the PARALLEL_DEGREE_POLICY parameter to AUTO:

SQL_ID  81dw7f2yz0jxp, child number 1
-------------------------------------
select COUNT(*) from t1 WHERE C3 BETWEEN to_date('01/08/2012',
'dd/mm/yyyy') and to_date('14/10/2012', 'dd/mm/yyyy')

Plan hash value: 2744578615

--------------------------------------------------------------------------------------------------
| Id  | Operation                 | Name | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
--------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT          |      |       |       |  4082 (100)|          |       |       |
|   1 |  SORT AGGREGATE           |      |     1 |     8 |            |          |       |       |
|   2 |   PARTITION RANGE ITERATOR|      |  2531K|    19M|  4082  (29)| 00:00:01 |    69 |    71 |
|*  3 |    TABLE ACCESS FULL      | T1   |  2531K|    19M|  4082  (29)| 00:00:01 |    69 |    71 |
--------------------------------------------------------------------------------------------------

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('11.2.0.2')
      DB_VERSION('11.2.0.2')
      ALL_ROWS
      NO_PARALLEL
      OUTLINE_LEAF(@"SEL$1")
      FULL(@"SEL$1" "T1"@"SEL$1")
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------
   3 - filter(("C3"<=TO_DATE(' 2012-10-14 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND
              "C3">=TO_DATE(' 2012-08-01 00:00:00', 'syyyy-mm-dd hh24:mi:ss')))

Note
-----
   - automatic DOP: Computed Degree of Parallelism is 1 because of parallel threshold

In the above execution plan, notice that the calculated cost is 4,082 (this is higher than the calculated cost returned by Oracle Database 11.2.0.3).  The SREADTIM and MREADTIM system statistics were manually set to very small numbers – due to a bug in Oracle Database 11.2.0.1 and 11.2.0.2 these system statistics could be set to much large values, such as 2,491 and 9,163, respectively.  What might happen to the calculated cost if we were to increase the SREADTIM and MREADTIM system statistics by a factor of 10?  Such a change would indicate to the query optimizer that each disk access will required 10 times longer to complete.  Should the calculated cost of this query increase or decrease after changing the system statistics?

First, let’s manually alter the value of the system statistics:

EXEC DBMS_STATS.SET_SYSTEM_STATS('SREADTIM',4)
EXEC DBMS_STATS.SET_SYSTEM_STATS('MREADTIM',8)

Next, execute the query and display the execution plan (notice that capitalization of the word SELECT was changed to force a hard parse of the query):

ALTER SESSION SET PARALLEL_DEGREE_POLICY=AUTO;

SELECT COUNT(*) from t1
WHERE C3 BETWEEN to_date('01/08/2012', 'dd/mm/yyyy') and to_date('14/10/2012', 'dd/mm/yyyy');

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL +OUTLINE'));

SQL_ID  2q2uz15vyzqsy, child number 1
-------------------------------------
SELECT COUNT(*) from t1 WHERE C3 BETWEEN to_date('01/08/2012',
'dd/mm/yyyy') and to_date('14/10/2012', 'dd/mm/yyyy')

Plan hash value: 2744578615

--------------------------------------------------------------------------------------------------
| Id  | Operation                 | Name | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
--------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT          |      |       |       |  3020 (100)|          |       |       |
|   1 |  SORT AGGREGATE           |      |     1 |     8 |            |          |       |       |
|   2 |   PARTITION RANGE ITERATOR|      |  2531K|    19M|  3020   (4)| 00:00:01 |    69 |    71 |
|*  3 |    TABLE ACCESS FULL      | T1   |  2531K|    19M|  3020   (4)| 00:00:01 |    69 |    71 |
--------------------------------------------------------------------------------------------------

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('11.2.0.2')
      DB_VERSION('11.2.0.2')
      ALL_ROWS
      NO_PARALLEL
      OUTLINE_LEAF(@"SEL$1")
      FULL(@"SEL$1" "T1"@"SEL$1")
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------
   3 - filter(("C3"<=TO_DATE(' 2012-10-14 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND
              "C3">=TO_DATE(' 2012-08-01 00:00:00', 'syyyy-mm-dd hh24:mi:ss')))

Note
-----
   - automatic DOP: Computed Degree of Parallelism is 1 because of parallel threshold

Notice the changes in the execution plan?  The calculated cost shown in the execution plan decreased from 4,082 to 3,020, and the %CPU for the third line in the execution plan decreased from 29 to 4.  Why did the calculated cost decrease when the optimizer is informed that disk accesses require 10 times as long to complete?  For the answer, take a look at this blog article written by Randolf Geist (another OakTable Network member).

—

OK, so I drifted a bit from the focus of the OTN message thread that provoked this article.  What do you think caused parallel execution to be used when the OP executed the query found in the OTN thread?

Parallel Execution Challenge – It’s About Time

July 8, 2012

As I was reading a book, I saw a comment essentially stating that with multiple CPUs/cores, parallel execution will be faster than serial execution of the same query.  The book provided a test case – I decided to change around the test case a bit so that the results were a bit more fair, and then I performed a bit of testing using the sample database data that Oracle Corporation provides. 

My testing was performed in Oracle Database 11.2.0.1 on 64 bit Windows with a quad core CPU that supports hyperthreading, and two SSD drives in a RAID 0 array (with all scripts executed on the server).  So, is parallel query always faster than serial execution when the server has multiple CPUs/cores?  But wait, that is not the interesting question… not the point of this parallel execution challenge, but still an interesting question.  Here is the test script, which will be executed three times, with the results of the first execution thrown out so that we may eliminate the side effects of the hard parse:

ALTER SESSION SET CURRENT_SCHEMA=SH;
SET AUTOTRACE TRACEONLY STATISTICS EXPLAIN
SET TIMING ON
SET LINESIZE 145
SET PAGESIZE 1000

ALTER SYSTEM FLUSH BUFFER_CACHE;
SELECT /*+ PARALLEL(2) */ S.PROD_ID, S.CUST_ID, S.TIME_ID FROM SH.SALES S ORDER BY S.AMOUNT_SOLD DESC;

ALTER SYSTEM FLUSH BUFFER_CACHE;
SELECT S.PROD_ID, S.CUST_ID, S.TIME_ID FROM SH.SALES S ORDER BY S.AMOUNT_SOLD DESC; 

Notice in the above that the test case performs the parallel execution first, with a degree of parallelism of 2, and then executes the same query without using parallel execution.

Let’s look at the output for the parallel execution:

918843 rows selected.

Elapsed: 00:00:11.97

Execution Plan
----------------------------------------------------------
Plan hash value: 2055439529

-----------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation               | Name     | Rows  | Bytes |TempSpc| Cost (%CPU)| Time     | Pstart| Pstop |    TQ  |IN-OUT| PQ Distrib |
-----------------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT        |          |   918K|    19M|       |   308  (14)| 00:00:04 |       |       |        |      |            |
|   1 |  PX COORDINATOR         |          |       |       |       |            |          |       |       |        |      |            |
|   2 |   PX SEND QC (ORDER)    | :TQ10001 |   918K|    19M|       |   308  (14)| 00:00:04 |       |       |  Q1,01 | P->S | QC (ORDER) |
|   3 |    SORT ORDER BY        |          |   918K|    19M|    31M|   308  (14)| 00:00:04 |       |       |  Q1,01 | PCWP |            |
|   4 |     PX RECEIVE          |          |   918K|    19M|       |   277   (4)| 00:00:04 |       |       |  Q1,01 | PCWP |            |
|   5 |      PX SEND RANGE      | :TQ10000 |   918K|    19M|       |   277   (4)| 00:00:04 |       |       |  Q1,00 | P->P | RANGE      |
|   6 |       PX BLOCK ITERATOR |          |   918K|    19M|       |   277   (4)| 00:00:04 |     1 |    28 |  Q1,00 | PCWC |            |
|   7 |        TABLE ACCESS FULL| SALES    |   918K|    19M|       |   277   (4)| 00:00:04 |     1 |    28 |  Q1,00 | PCWP |            |
-----------------------------------------------------------------------------------------------------------------------------------------

Note
-----
   - Degree of Parallelism is 2 because of hint

Statistics
----------------------------------------------------------
         12  recursive calls
          0  db block gets
       1802  consistent gets
       1647  physical reads
          0  redo size
   22116448  bytes sent via SQL*Net to client
    1102982  bytes received via SQL*Net from client
      61258  SQL*Net roundtrips to/from client
          3  sorts (memory)
          0  sorts (disk)
     918843  rows processed 

918,843 rows selected (not displayed in SQL*Plus due to the AUTOTRACE setting), 1,647 blocks read from disk, 1,802 consistent gets, an estimated cost of 308, and the execution completed in 11.97 seconds.

Let’s take a look at the serial execution output:

918843 rows selected.

Elapsed: 00:00:08.84

Execution Plan
----------------------------------------------------------
Plan hash value: 3803407550

------------------------------------------------------------------------------------------------------
| Id  | Operation            | Name  | Rows  | Bytes |TempSpc| Cost (%CPU)| Time     | Pstart| Pstop |
------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT     |       |   918K|    19M|       |  6637   (2)| 00:01:20 |       |       |
|   1 |  SORT ORDER BY       |       |   918K|    19M|    31M|  6637   (2)| 00:01:20 |       |       |
|   2 |   PARTITION RANGE ALL|       |   918K|    19M|       |   499   (4)| 00:00:06 |     1 |    28 |
|   3 |    TABLE ACCESS FULL | SALES |   918K|    19M|       |   499   (4)| 00:00:06 |     1 |    28 |
------------------------------------------------------------------------------------------------------

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
       1718  consistent gets
       1647  physical reads
          0  redo size
   22116448  bytes sent via SQL*Net to client
    1102982  bytes received via SQL*Net from client
      61258  SQL*Net roundtrips to/from client
          1  sorts (memory)
          0  sorts (disk)
     918843  rows processed 

918,843 rows selected (not displayed in SQL*Plus due to the AUTOTRACE setting), 1,647 blocks read from disk, 1,718 consistent gets, an estimated cost of 6,637, and the execution completed in 8.84 seconds.

* The first part of the challenge is to describe how the cost is derived in the above execution plan for the parallel execution.  Here is the easy part of the cost calculation:

cost of serial full table scan / degree of parallel / 0.90
499 / 2 / 0.90 = 277.22

The above calculation is easy to understand, and is explained in this article.  The SORT ORDER BY operation in the parallel query execution plan added 31 to the calculated cost (308 – 277), while the same operation in the serial execution added 6,138 to the calculated cost (6637 – 499).

The first challenge is to describe why the calculated cost of the parallel execution plan is so small compared to the serial execution.

—

The timing output indicated that the serial execution required 3.13 fewer seconds than the parallel execution.  Let’s look at the results of the third execution of this test case script:

918843 rows selected.

Elapsed: 00:00:10.66

Execution Plan
----------------------------------------------------------
Plan hash value: 2055439529

-----------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation               | Name     | Rows  | Bytes |TempSpc| Cost (%CPU)| Time     | Pstart| Pstop |    TQ  |IN-OUT| PQ Distrib |
-----------------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT        |          |   918K|    19M|       |   308  (14)| 00:00:04 |       |       |        |      |            |
|   1 |  PX COORDINATOR         |          |       |       |       |            |          |       |       |        |      |            |
|   2 |   PX SEND QC (ORDER)    | :TQ10001 |   918K|    19M|       |   308  (14)| 00:00:04 |       |       |  Q1,01 | P->S | QC (ORDER) |
|   3 |    SORT ORDER BY        |          |   918K|    19M|    31M|   308  (14)| 00:00:04 |       |       |  Q1,01 | PCWP |            |
|   4 |     PX RECEIVE          |          |   918K|    19M|       |   277   (4)| 00:00:04 |       |       |  Q1,01 | PCWP |            |
|   5 |      PX SEND RANGE      | :TQ10000 |   918K|    19M|       |   277   (4)| 00:00:04 |       |       |  Q1,00 | P->P |      RANGE |
|   6 |       PX BLOCK ITERATOR |          |   918K|    19M|       |   277   (4)| 00:00:04 |     1 |    28 |  Q1,00 | PCWC |            |
|   7 |        TABLE ACCESS FULL| SALES    |   918K|    19M|       |   277   (4)| 00:00:04 |     1 |    28 |  Q1,00 | PCWP |            |
-----------------------------------------------------------------------------------------------------------------------------------------

Note
-----
   - Degree of Parallelism is 2 because of hint

Statistics
----------------------------------------------------------
         12  recursive calls
          0  db block gets
       1802  consistent gets
       1647  physical reads
          0  redo size
   22116448  bytes sent via SQL*Net to client
    1102982  bytes received via SQL*Net from client
      61258  SQL*Net roundtrips to/from client
          3  sorts (memory)
          0  sorts (disk)
     918843  rows processed 

The above shows the results of the parallel execution.  918,843 rows selected (not displayed in SQL*Plus due to the AUTOTRACE setting), 1,647 blocks read from disk, 1,802 consistent gets, an estimated cost of 308, and completed in 10.66 seconds.  Notice that with the exception of the elapsed time, the statistics are identical to what we saw during the previous execution of the test script.

918843 rows selected.

Elapsed: 00:00:08.48

Execution Plan
----------------------------------------------------------
Plan hash value: 3803407550

------------------------------------------------------------------------------------------------------
| Id  | Operation            | Name  | Rows  | Bytes |TempSpc| Cost (%CPU)| Time     | Pstart| Pstop |
------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT     |       |   918K|    19M|       |  6637   (2)| 00:01:20 |       |       |
|   1 |  SORT ORDER BY       |       |   918K|    19M|    31M|  6637   (2)| 00:01:20 |       |       |
|   2 |   PARTITION RANGE ALL|       |   918K|    19M|       |   499   (4)| 00:00:06 |     1 |    28 |
|   3 |    TABLE ACCESS FULL | SALES |   918K|    19M|       |   499   (4)| 00:00:06 |     1 |    28 |
------------------------------------------------------------------------------------------------------

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
       1718  consistent gets
       1647  physical reads
          0  redo size
   22116448  bytes sent via SQL*Net to client
    1102982  bytes received via SQL*Net from client
      61258  SQL*Net roundtrips to/from client
          1  sorts (memory)
          0  sorts (disk)
     918843  rows processed 

The above shows the results of the serial execution. 918,843 rows selected (not displayed in SQL*Plus due to the AUTOTRACE setting), 1,647 blocks read from disk, 1,718 consistent gets, an estimated cost of 6,637, and completed in 8.48 seconds. Notice that with the exception of the elapsed time, the statistics are identical to what we saw during the previous execution of the test script.

The second challenge is to describe why the serial execution completed at least 25% faster than the parallel execution of the same query – this result is counter to what the calculated cost of the execution plans suggest, and counter to what the book claimed.

—

Maybe we need to look at the result from a different angle.  The book did not use SET TIMING ON as my test case above did, so let’s change the test case again.  This time, we will use the GATHER_PLAN_STATISTICS hint so that the runtime engine will collect the timing and various other statistics, and then use DBMS_XPLAN to display the actual execution plan.  To eliminate the effects of hard parsing, the script will be executed three times with the first execution thrown out.  The new script:

ALTER SESSION SET CURRENT_SCHEMA=SH;
SET AUTOTRACE TRACEONLY STATISTICS
SET TIMING OFF
SET LINESIZE 145
SET PAGESIZE 1000

ALTER SYSTEM FLUSH BUFFER_CACHE;
SELECT /*+ Q1 PARALLEL(2) GATHER_PLAN_STATISTICS */ S.PROD_ID, S.CUST_ID, S.TIME_ID FROM SH.SALES S ORDER BY S.AMOUNT_SOLD DESC;

ALTER SYSTEM FLUSH BUFFER_CACHE;
SELECT /*+ Q2 GATHER_PLAN_STATISTICS */ S.PROD_ID, S.CUST_ID, S.TIME_ID FROM SH.SALES S ORDER BY S.AMOUNT_SOLD DESC;

SET AUTOTRACE OFF 

SELECT
  SQL_ID,
  CHILD_NUMBER,
  SUBSTR(SQL_TEXT,1,30) SQL_TEXT
FROM
  V$SQL
WHERE
  SQL_TEXT LIKE 'SELECT /*+ Q%'
ORDER BY
  3,
  1,
  2; 

The last of the above SQL statements simply retrieves the SQL_ID and CHILD_NUMBER for the two previous queries.  For my results, I received the following output from the last of the above SQL statements:

SQL_ID        CHILD_NUMBER SQL_TEXT
------------- ------------ ------------------------------
5p47nbddfn0k7            0 SELECT /*+ Q1 PARALLEL(2) GATH
8k6bdqr09xc8u            0 SELECT /*+ Q2 GATHER_PLAN_STAT 

To display the execution plans with the runtime statistics, we are able to use the following two statements:

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR('5p47nbddfn0k7',0,'ALLSTATS LAST +COST'));

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR('8k6bdqr09xc8u',0,'ALLSTATS LAST +COST')); 

Here are the autotrace statistics for the parallel execution:

Statistics
----------------------------------------------------------
         12  recursive calls
          0  db block gets
       1802  consistent gets
       1647  physical reads
          0  redo size
   22116448  bytes sent via SQL*Net to client
    1102982  bytes received via SQL*Net from client
      61258  SQL*Net roundtrips to/from client
          3  sorts (memory)
          0  sorts (disk)
     918843  rows processed 

The DBMS_XPLAN generated execution plan for the parallel execution follows:

SQL_ID  5p47nbddfn0k7, child number 0
-------------------------------------
SELECT /*+ Q1 PARALLEL(2) GATHER_PLAN_STATISTICS */ S.PROD_ID,
S.CUST_ID, S.TIME_ID FROM SH.SALES S ORDER BY S.AMOUNT_SOLD DESC

Plan hash value: 2055439529

-----------------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation               | Name     | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers | Reads  |  OMem |  1Mem | Used-Mem |
-----------------------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT        |          |      1 |        |   308 (100)|    918K|00:00:00.70 |     116 |     28 |       |       |          |
|   1 |  PX COORDINATOR         |          |      1 |        |            |    918K|00:00:00.70 |     116 |     28 |       |       |          |
|   2 |   PX SEND QC (ORDER)    | :TQ10001 |      0 |    918K|   308  (14)|      0 |00:00:00.01 |       0 |      0 |       |       |          |
|   3 |    SORT ORDER BY        |          |      0 |    918K|   308  (14)|      0 |00:00:00.01 |       0 |      0 |    83M|  3142K|   37M (0)|
|   4 |     PX RECEIVE          |          |      0 |    918K|   277   (4)|      0 |00:00:00.01 |       0 |      0 |       |       |          |
|   5 |      PX SEND RANGE      | :TQ10000 |      0 |    918K|   277   (4)|      0 |00:00:00.01 |       0 |      0 |       |       |          |
|   6 |       PX BLOCK ITERATOR |          |      0 |    918K|   277   (4)|      0 |00:00:00.01 |       0 |      0 |       |       |          |
|*  7 |        TABLE ACCESS FULL| SALES    |      0 |    918K|   277   (4)|      0 |00:00:00.01 |       0 |      0 |       |       |          |
-----------------------------------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   7 - access(:Z>=:Z AND :Z<=:Z)

Note
-----
   - Degree of Parallelism is 2 because of hint 

Notice that the execution plan claims that the query completed in 0.70 seconds, that the calculated cost of the execution plan is (still) 308, that 28 blocks were read from disk, and 116 consistent gets were performed.

Here are the autotrace statistics for the serial execution:

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
       1718  consistent gets
       1647  physical reads
          0  redo size
   22116448  bytes sent via SQL*Net to client
    1102982  bytes received via SQL*Net from client
      61258  SQL*Net roundtrips to/from client
          1  sorts (memory)
          0  sorts (disk)
     918843  rows processed 

The DBMS_XPLAN generated execution plan for the serial execution plan follows:

SQL_ID  8k6bdqr09xc8u, child number 0
-------------------------------------
SELECT /*+ Q2 GATHER_PLAN_STATISTICS */ S.PROD_ID, S.CUST_ID, S.TIME_ID
FROM SH.SALES S ORDER BY S.AMOUNT_SOLD DESC

Plan hash value: 3803407550

-----------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation            | Name  | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers | Reads  |  OMem |  1Mem | Used-Mem |
-----------------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT     |       |      1 |        |  6637 (100)|    918K|00:00:00.93 |    1718 |   1647 |       |       |          |
|   1 |  SORT ORDER BY       |       |      1 |    918K|  6637   (2)|    918K|00:00:00.93 |    1718 |   1647 |    42M|  2308K|   37M (0)|
|   2 |   PARTITION RANGE ALL|       |      1 |    918K|   499   (4)|    918K|00:00:00.31 |    1718 |   1647 |       |       |          |
|   3 |    TABLE ACCESS FULL | SALES |     28 |    918K|   499   (4)|    918K|00:00:00.22 |    1718 |   1647 |       |       |          |
----------------------------------------------------------------------------------------------------------------------------------------- 

Notice that the execution plan claims that the query completed in 0.93 seconds, that the calculated cost of the execution plan is (still) 6,637, that 1,647 blocks were read from disk, and 1,718 consistent gets were performed.

The third challenge is to describe why DBMS_XPLAN claims that the parallel execution completed roughly 32.9% faster than the serial execution, when the timing output in SQL*Plus indicated that the reverse was true.  Bonus: Why do the Buffers and Reads statistics in the top row of the execution plan for the serial execution plan exactly match the autotrace generated consistent gets and physical reads statistics, yet the same is not true for the execution plan for the parallel execution?  Double Bonus: the execution plan claims that the query completed in 0.93 seconds, yet the timing information in SQL*Plus indicated that the query required an additional 7.55 seconds, why is there so much difference between the reported times (keep in mind that all testing was performed on the server, and the output was discarded without formatting and without displaying on screen)?

—

If cost is time, and time is money, what is the exchange rate for parallel execution?  😉

—

I have not fully worked the answers to the above challenge questions, and I will wait a couple of days before providing any feedback regarding proposed answers supplied by readers.  I strongly suspect that some readers know without a doubt the answers to all of the above questions.

Different Execution Plans when Converting SQL from SELECT to UPDATE … IN SELECT, ROWNUM Related Bug in 11.2.0.2

June 28, 2012 (Modified June 28, 2012)

Sometimes it is possible to build a very effective test case to reproduce a particular problem.  Sometimes the test case drifts a bit offtrack, in the process revealing other information that is not directly related to the original problem that the test case attempted to simulate.  You will see several examples of test cases scripts used in various articles on this blog, including one that attempts to describe how to build useful test cases.  It is not uncommon to see test cases used to demonstrate various types of behavior, and there are several well known people in the Oracle Database community that use test cases to not only demonstrate problems, but also to teach methods of avoiding those problems.

The FAQ (Frequently Asked Questions) for the SQL and PL/SQL OTN forum includes links to two OTN threads that everyone posting to the forum about performance problems should read carefully:

When your query takes too long…
How to post a SQL statement tuning request

A handful of the good test cases have been posted to the OTN forums over the last couple of years that have then sparked an interesting article on this blog.  Take, for example, this OTN thread that provided a test case script to show that a poor execution plan was generated by the Oracle Database optimizer when ROWNUM was included in the WHERE clause.  Prior to Oracle Database 11.2.0.1, the cardinality calculation for steps in the execution plan that could not possibly be short-circuited early (such as a hash join with a subsequent ORDER BY) were affected by the ROWNUM predicate in the WHERE clause (of a parent query block).  That test case resulted in at least one article on this blog, which showed that with a bit of modification to the SQL statement, that it was possible to use the ROW_NUMBER analytic function as a replacement for ROWNUM, thus avoiding the automatic switch to a FIRST_ROWS(n) optimizer mode for the SQL statement.  This particular problem was determined to be a bug, and corrected with the release of Oracle Database 11.2.0.1.

Fast forward two and a half years, and we see another interesting, and oddly related test case in a thread on the OTN forum.  The OP (original poster) in the OTN forum thread generated the test case, posting his results from Oracle Database 10.2.0.5 (Standard Edition), while I posted my observed results from Oracle Database 11.2.0.2 in an attempt to find a more efficient method for updating rows that match specific criteria (other people helped also, possibly executing the test case script in other Oracle Database release versions).

What do we learn from the OTN test case script?  Here is a start:

• When hinting a SQL statement to generate an expected execution plan, it is important to be as specific as possible with the hints, without boxing the SQL statement into a corner as the data volume increases in the various tables accessed by the SQL statement.  The hints also must appear in the correct section of a complex SQL statement with one or more inline views or subqueries.
• If you use ROWNUM in the WHERE clause of a SQL statement, and then convert that SELECT statement to a UPDATE tablex WHERE coly IN (SELECT…) statement, you may find that the execution plan for the SELECT portion of the UPDATE statement changes dramatically from what was seen when just the SELECT statement was executed.  With Oracle Database 11.2.0.2, you may find that the first execution is efficient, and on the second execution find that cardinality feedback triggered a new hard parse of the SQL statement, resulting in a significantly less efficient execution plan (or possibly significantly more efficient execution plan).
• AUTOTRACE in SQL*Plus lies about the execution plan sometimes, such as when cardinality feedback triggers the generation of a new execution plan.
• Using a UNION ALL in an inline view will likely not permit the Oracle runtime engine to short-circuit an operation in the inline view when a ROWNUM condition is present in the parent SQL statement, even if the query optimizer’s generated execution plan indicates that this short-circuit will happen.  If possible, rewrite the SQL statement to avoid the UNION ALL operation, if both sections of the UNION ALL are accessing the same tables.
• The GATHER_PLAN_STATISTICS hint, when added to a SQL statement, permits the use of DBMS_XPLAN.DISPLAY_CURSOR with the ALLSTATS LAST format parameter.  An execution plan generated with that format parameter will indicate whether or not short-circuiting of steps in an execution plan took place.  This short-circuiting will be indicated in the A-Rows column, the Starts column, and/or the Predicate Information section of the execution plan.
• Within an inline view, an index-driven nested loop join that permits the elimination of an ORDER BY clause can be short-circuited by a ROWNUM condition in the parent portion of the SQL statement, while the same is not true for hash or sort-merge joins.
• The ROWNUM cardinality calculation bug is still present in Oracle Database 11.2.0.2 if the inline view contains a UNION ALL clause – this bug is most apparent when cardinality feedback triggers the generation of a new execution plan on the second execution of the SQL statement.
• In Oracle Database 11.2.0.2, query blocks in a SQL statement that cannot be short-circuited by a parent query block’s ROWNUM predicate are still optimized with an automatic FIRST_ROWS(n) optimizer mode (where n is the number associated with the parent operation’s ROWNUM predicate).  Through an apparent bug in the query optimizer, this automatic optimization may cause odd cost calculation problems, such as dropping the cost of a full table scan from 35.01 to 2.02 (with the cost of the corresponding hash join dropping from 56.76 to just 23.83); at the same time, due to cardinality feedback, the cost of an efficient index access path with a nested loops join (that may be short-circuited by the runtime engine) at the same point in the execution plan may increase from a cost of 41.32 to 2947.89.  As a result of the costing change, the nested loop join with index access path is likely to be replaced with a hash join and full table scan.

What else may we learn from the latest OTN test case thread?

—

Late Addition to the Article (June 28, 2012)

A side-by-side comparison of a portion of the 10053 trace file (showing the possible bug related to ROWNUM and cardinality feedback) that was generated by the following script (using the OP’s sample data):

DROP INDEX IND_DTL_TEST;

SET AUTOTRACE OFF
SET LINESIZE 140
SET PAGESIZE 1000

ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';
ALTER SESSION SET TRACEFILE_IDENTIFIER='TESTING 1';

select /*+ GATHER_PLAN_STATISTICS */
  dtlid
from
  (select
     dtlid
   from
     (select
        dtlid,
        process_date
      from
        dtl
      where
        hdrid is null
        and process_date < (sysdate + 30)
        and process_ind = 'NEW'
      UNION ALL
      select
        dtl.dtlid,
        dtl.process_date
      from
        dtl,
        hdr
      where
        dtl.hdrid = hdr.hdrid
        and dtl.process_date < (sysdate + 30)
        and dtl.process_ind = 'NEW'
        and hdr_ind in (0, 2))
   order by
     dtlid,
     process_date)
where
  rownum <= 10;

SELECT * FROM TABLE (DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST'));

ALTER SESSION SET TRACEFILE_IDENTIFIER='TESTING 2';

select /*+ GATHER_PLAN_STATISTICS */
  dtlid
from
  (select
     dtlid
   from
     (select
        dtlid,
        process_date
      from
        dtl
      where
        hdrid is null
        and process_date < (sysdate + 30)
        and process_ind = 'NEW'
      UNION ALL
      select
        dtl.dtlid,
        dtl.process_date
      from
        dtl,
        hdr
      where
        dtl.hdrid = hdr.hdrid
        and dtl.process_date < (sysdate + 30)
        and dtl.process_ind = 'NEW'
        and hdr_ind in (0, 2))
   order by
     dtlid,
     process_date)
where
  rownum <= 10;

SELECT * FROM TABLE (DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST'));

ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT OFF'; 

The generated execution plan, before cardinality feedback:

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  4dwv2ktktv1ck, child number 1
-------------------------------------
select /*+ GATHER_PLAN_STATISTICS */   dtlid from   (select      dtlid
  from      (select         dtlid,         process_date       from
   dtl       where         hdrid is null         and process_date <
(sysdate + 30)         and process_ind = 'NEW'       UNION ALL
select         dtl.dtlid,         dtl.process_date       from
dtl,         hdr       where         dtl.hdrid = hdr.hdrid         and
dtl.process_date < (sysdate + 30)         and dtl.process_ind = 'NEW'
      and hdr_ind in (0, 2))    order by      dtlid,      process_date)
where   rownum <= 10

Plan hash value: 2201656677

------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                          | Name     | Starts | E-Rows | A-Rows |   A-Time   | Buffers |  OMem |  1Mem | Used-Mem |
------------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                   |          |      1 |        |     10 |00:00:00.05 |   31500 |       |       |          |
|*  1 |  COUNT STOPKEY                     |          |      1 |        |     10 |00:00:00.05 |   31500 |       |       |          |
|   2 |   VIEW                             |          |      1 |     22 |     10 |00:00:00.05 |   31500 |       |       |          |
|*  3 |    SORT ORDER BY STOPKEY           |          |      1 |     22 |     10 |00:00:00.05 |   31500 |  2048 |  2048 | 2048  (0)|
|   4 |     VIEW                           |          |      1 |     22 |   1367 |00:00:00.04 |   31500 |       |       |          |
|   5 |      UNION-ALL                     |          |      1 |        |   1367 |00:00:00.04 |   31500 |       |       |          |
|*  6 |       TABLE ACCESS FULL            | DTL      |      1 |     11 |    160 |00:00:00.01 |    1117 |       |       |          |
|   7 |       NESTED LOOPS                 |          |      1 |        |   1207 |00:00:00.07 |   30383 |       |       |          |
|   8 |        NESTED LOOPS                |          |      1 |     11 |   2838 |00:00:00.04 |   27566 |       |       |          |
|   9 |         TABLE ACCESS BY INDEX ROWID| DTL      |      1 |     32 |  15034 |00:00:00.02 |   15068 |       |       |          |
|* 10 |          INDEX RANGE SCAN          | DTL_IDX1 |      1 |        |  15034 |00:00:00.01 |      56 |       |       |          |
|* 11 |         INDEX UNIQUE SCAN          | HDR_PK   |  15034 |      1 |   2838 |00:00:00.01 |   12498 |       |       |          |
|* 12 |        TABLE ACCESS BY INDEX ROWID | HDR      |   2838 |      1 |   1207 |00:00:00.01 |    2817 |       |       |          |
------------------------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter(ROWNUM<=10)
   3 - filter(ROWNUM<=10)
   6 - filter(("HDRID" IS NULL AND "PROCESS_IND"='NEW' AND "PROCESS_DATE"<SYSDATE@!+30))
  10 - access("DTL"."PROCESS_IND"='NEW' AND "DTL"."PROCESS_DATE"<SYSDATE@!+30)
  11 - access("DTL"."HDRID"="HDR"."HDRID")
  12 - filter(("HDR_IND"=0 OR "HDR_IND"=2)) 

After cardinality feedback:

SQL_ID  4dwv2ktktv1ck, child number 2
-------------------------------------
select /*+ GATHER_PLAN_STATISTICS */   dtlid from   (select      dtlid
  from      (select         dtlid,         process_date       from
   dtl       where         hdrid is null         and process_date <
(sysdate + 30)         and process_ind = 'NEW'       UNION ALL
select         dtl.dtlid,         dtl.process_date       from
dtl,         hdr       where         dtl.hdrid = hdr.hdrid         and
dtl.process_date < (sysdate + 30)         and dtl.process_ind = 'NEW'
      and hdr_ind in (0, 2))    order by      dtlid,      process_date)
where   rownum <= 10

Plan hash value: 986254152

---------------------------------------------------------------------------------------------------------------------
| Id  | Operation               | Name | Starts | E-Rows | A-Rows |   A-Time   | Buffers |  OMem |  1Mem | Used-Mem |
---------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT        |      |      1 |        |     10 |00:00:00.01 |    2437 |       |       |          |
|*  1 |  COUNT STOPKEY          |      |      1 |        |     10 |00:00:00.01 |    2437 |       |       |          |
|   2 |   VIEW                  |      |      1 |   2104 |     10 |00:00:00.01 |    2437 |       |       |          |
|*  3 |    SORT ORDER BY STOPKEY|      |      1 |   2104 |     10 |00:00:00.01 |    2437 |  2048 |  2048 | 2048  (0)|
|   4 |     VIEW                |      |      1 |   2104 |   1367 |00:00:00.01 |    2437 |       |       |          |
|   5 |      UNION-ALL          |      |      1 |        |   1367 |00:00:00.01 |    2437 |       |       |          |
|*  6 |       TABLE ACCESS FULL | DTL  |      1 |    160 |    160 |00:00:00.01 |    1117 |       |       |          |
|*  7 |       HASH JOIN         |      |      1 |   1944 |   1207 |00:00:00.01 |    1320 |  1517K|  1517K| 1398K (0)|
|*  8 |        TABLE ACCESS FULL| HDR  |      1 |   6667 |   4285 |00:00:00.01 |     203 |       |       |          |
|*  9 |        TABLE ACCESS FULL| DTL  |      1 |  15034 |  15034 |00:00:00.01 |    1117 |       |       |          |
---------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter(ROWNUM<=10)
   3 - filter(ROWNUM<=10)
   6 - filter(("HDRID" IS NULL AND "PROCESS_IND"='NEW' AND "PROCESS_DATE"<SYSDATE@!+30))
   7 - access("DTL"."HDRID"="HDR"."HDRID")
   8 - filter(("HDR_IND"=0 OR "HDR_IND"=2))
   9 - filter(("DTL"."PROCESS_IND"='NEW' AND "DTL"."PROCESS_DATE"<SYSDATE@!+30))

Note
-----
   - cardinality feedback used for this statement 

In the above example, cardinality feedback helped to reduce the execution time.  Notice in the first plan that the E-Rows column shows that the ROWNUM predicate was pushed too far into the inline view’s cardinality calculations.  So, cardinality feedback helped in the above example.  Now, let’s look at what happens to my optimized version of the SQL statement that removes the UNION ALL:

select
  dtlid
from
  (select
     dtlid
   from
     (select /*+ INDEX(DTL) INDEX(HRD) USE_NL(DTL HDR) */
        dtl.dtlid,
        dtl.process_date
      from
        dtl,
        hdr
      where
        dtl.hdrid = hdr.hdrid(+)
        and dtl.process_date < (sysdate + 30)
        and dtl.process_ind = 'NEW'
        and (
            (hdr_ind in (0, 2) and dtl.hdrid IS NOT NULL)
            OR (dtl.hdrid IS NULL)))
   order by
     dtlid,
     process_date) 

The execution plan before cardinality feedback:

SQL_ID  5d8128vjhm1j0, child number 0
-------------------------------------
select /*+ GATHER_PLAN_STATISTICS */   dtlid from   (select      dtlid
  from      (select         dtl.dtlid,         dtl.process_date
from         dtl,         hdr       where         dtl.hdrid =
hdr.hdrid(+)         and dtl.process_date < (sysdate + 30)         and
dtl.process_ind = 'NEW'         and (             (hdr_ind in (0, 2)
and dtl.hdrid IS NOT NULL)             OR (dtl.hdrid IS NULL)))
order by      dtlid,      process_date) where   rownum <= 10

Plan hash value: 1187173211

----------------------------------------------------------------------------------------------------------
| Id  | Operation                       | Name         | Starts | E-Rows | A-Rows |   A-Time   | Buffers |
----------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                |              |      1 |        |     10 |00:00:00.01 |      80 |
|*  1 |  COUNT STOPKEY                  |              |      1 |        |     10 |00:00:00.01 |      80 |
|   2 |   VIEW                          |              |      1 |     11 |     10 |00:00:00.01 |      80 |
|*  3 |    FILTER                       |              |      1 |        |     10 |00:00:00.01 |      80 |
|   4 |     NESTED LOOPS OUTER          |              |      1 |     11 |     19 |00:00:00.01 |      80 |
|   5 |      TABLE ACCESS BY INDEX ROWID| DTL          |      1 |  19635 |     19 |00:00:00.01 |      52 |
|*  6 |       INDEX FULL SCAN           | IND_DTL_TEST |      1 |     16 |     19 |00:00:00.01 |      49 |
|   7 |      TABLE ACCESS BY INDEX ROWID| HDR          |     19 |      1 |     18 |00:00:00.01 |      28 |
|*  8 |       INDEX UNIQUE SCAN         | HDR_PK       |     19 |      1 |     18 |00:00:00.01 |      10 |
----------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter(ROWNUM<=10)
   3 - filter(((INTERNAL_FUNCTION("HDR_IND") AND "DTL"."HDRID" IS NOT NULL) OR "DTL"."HDRID" IS
              NULL))
   6 - access("DTL"."PROCESS_IND"='NEW' AND "DTL"."PROCESS_DATE"<SYSDATE@!+30)
       filter(("DTL"."PROCESS_IND"='NEW' AND "DTL"."PROCESS_DATE"<SYSDATE@!+30))
   8 - access("DTL"."HDRID"="HDR"."HDRID") 

The execution plan after cardinality feedback:

SQL_ID  5d8128vjhm1j0, child number 1
-------------------------------------
select /*+ GATHER_PLAN_STATISTICS */   dtlid from   (select      dtlid
  from      (select         dtl.dtlid,         dtl.process_date
from         dtl,         hdr       where         dtl.hdrid =
hdr.hdrid(+)         and dtl.process_date < (sysdate + 30)         and
dtl.process_ind = 'NEW'         and (             (hdr_ind in (0, 2)
and dtl.hdrid IS NOT NULL)             OR (dtl.hdrid IS NULL)))
order by      dtlid,      process_date) where   rownum <= 10

Plan hash value: 672193791

--------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                        | Name   | Starts | E-Rows | A-Rows |   A-Time   | Buffers |  OMem |  1Mem | Used-Mem |
--------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                 |        |      1 |        |     10 |00:00:00.02 |    5315 |       |       |          |
|*  1 |  COUNT STOPKEY                   |        |      1 |        |     10 |00:00:00.02 |    5315 |       |       |          |
|   2 |   VIEW                           |        |      1 |      9 |     10 |00:00:00.02 |    5315 |       |       |          |
|*  3 |    SORT ORDER BY STOPKEY         |        |      1 |      9 |     10 |00:00:00.02 |    5315 |  2048 |  2048 | 2048  (0)|
|*  4 |     FILTER                       |        |      1 |        |   1374 |00:00:00.02 |    5315 |       |       |          |
|   5 |      NESTED LOOPS OUTER          |        |      1 |      9 |  15115 |00:00:00.02 |    5315 |       |       |          |
|*  6 |       TABLE ACCESS FULL          | DTL    |      1 |     19 |  15115 |00:00:00.01 |    1117 |       |       |          |
|   7 |       TABLE ACCESS BY INDEX ROWID| HDR    |  15115 |      1 |   2853 |00:00:00.01 |    4198 |       |       |          |
|*  8 |        INDEX UNIQUE SCAN         | HDR_PK |  15115 |      1 |   2853 |00:00:00.01 |    1345 |       |       |          |
--------------------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter(ROWNUM<=10)
   3 - filter(ROWNUM<=10)
   4 - filter(((INTERNAL_FUNCTION("HDR_IND") AND "DTL"."HDRID" IS NOT NULL) OR "DTL"."HDRID" IS NULL))
   6 - filter(("DTL"."PROCESS_IND"='NEW' AND "DTL"."PROCESS_DATE"<SYSDATE@!+30))
   8 - access("DTL"."HDRID"="HDR"."HDRID")

Note
-----
   - cardinality feedback used for this statement 

In the above, notice that the reported A-Time increased from 0.01 seconds (the actual time could be must faster than that) to 0.02 seconds.  The number of consistent gets also increased from 80 to 5,315.  Cardinality feedback will NOT automatically change the execution plan back to the more efficient execution plan during later executions of the SQL statement.

Why is a Full Table Scan Selected – Will the Execution Plan Provide a Clue?

June 20, 2012

I have not had much time to respond in OTN threads recently, although I do still occasionally read threads on the forum.  I was a little surprised by one of the late responses in one of the recent threads, where one of the responders suggested actually testing the problem with the assistance of execution plans.  Additionally, that responder suggested that the earlier responses missed the target.  Ouch!

The OP did not provide DDL to create the table or indexes, or DML to populate the table.  The SQL statement provided by the OP was apparently looking for rows where the VARCHAR2 column A was blank (as in containing a zero length string).  If an index exists on column A, why would the optimizer select to perform a full table scan when attempting to return all rows with a zero length string in column A?  (Raise your hand if you know the answer.)

Let’s create a test table with four indexes so that we are able to test some of the theories (or possible causes) that were proposed in the thread:

DROP TABLE T1 PURGE;

CREATE TABLE T1 AS 
SELECT
  ROWNUM C1,
  MOD(ROWNUM,10) C2,
  TRUNC(ROWNUM/10000) C3,
  RPAD(TO_CHAR(ROWNUM),6,'A') C4,
  LPAD('A',200,'A') C5
FROM
  DUAL
CONNECT BY
  LEVEL<=100000;

ALTER TABLE T1 MODIFY (
  C1 NOT NULL,
  C2 NOT NULL,
  C3 NOT NULL);

CREATE INDEX IND_T1_C1 ON T1(C1);
CREATE INDEX IND_T1_C2 ON T1(C2);
CREATE INDEX IND_T1_C3 ON T1(C3);
CREATE INDEX IND_T1_C4 ON T1(C4);

EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T1',CASCADE=>TRUE,ESTIMATE_PERCENT=>NULL)

SET LINESIZE 140
SET PAGESIZE 1000 

The above script creates a table with 100,000 rows, with the first 3 columns declared as NOT NULL.  Column C1 contains the numbers 1 through 100,000.  Column C2 contains the numbers 0 through 9 in a repeating pattern.  Column C3 contains the numbers 0 through 10 with most of the rows containing the same value likely closely bunched together (a single row contains the number 10).  Column C4 is a simple VARCHAR2 column that is the number 1 through 100,000 padded to six characters using the letter A.  The statistics for the table and indexes were collected with a 100% sampling percentage.

Using the test table created by the above script, we might simulate the OP’s SQL statement using the following.  One of the last responders in the OTN thread recommended looking at the execution plan, so we will retrieve that also:

SELECT
  *
FROM
  T1
WHERE
  C4='';

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL')); 

Unfortunately, the two single quotes (”) in the SQL statement are not interpretted as a test for a zero length string as happens on other DB platforms, and as the OP likely intended (see this AskTom thread).  The Oracle Database 11.2.0.2 output of the above follows:

no rows selected

SQL>

SQL_ID  5tc2j4q52493b, child number 0
-------------------------------------
SELECT   * FROM   T1 WHERE   C4=''

Plan hash value: 3332582666

---------------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |       |       |     1 (100)|          |
|*  1 |  FILTER            |      |       |       |            |          |
|   2 |   TABLE ACCESS FULL| T1   |   100K|    20M|   252   (1)| 00:00:01 |
---------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter(NULL IS NOT NULL) 

So, the Predicate Information section shows NULL IS NOT NULL, which never has a result of TRUE.  Notice that the full table scan has a cost of 252, yet plan row 0, which is a grandparent of row 2, has a cost of 1 (costs for parent operations are supposed to include the calculated cost of child operations plus the calculated cost of the work performed at the parent operation).  (Raise your hand if you know the answer why plan row 0 has a cost of 1 when plan row 2 has a cost of 252.)  We will have to come back to this execution plan oddity later.

For a SQL statement like the following, which should return one row, you would expect the index on column C1 to be used:

SELECT
  C1,
  C2,
  C3,
  C4,
  SUBSTR(C5,1,10) C5
FROM
  T1
WHERE
  C1=9;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL')); 

Here is the output of the above:

        C1         C2         C3 C4     C5
---------- ---------- ---------- ------ ----------
         9          9          0 9AAAAA AAAAAAAAAA

SQL>
SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

PLAN_TABLE_OUTPUT
-----------------------------------------------------------------------------------------
SQL_ID  90d7zdbavdpuu, child number 0
-------------------------------------
SELECT   C1,   C2,   C3,   C4,   SUBSTR(C5,1,10) C5 FROM   T1 WHERE
C1=9

Plan hash value: 683303157

-----------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |       |       |     2 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T1        |     1 |   219 |     2   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | IND_T1_C1 |     1 |       |     1   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C1"=9) 

So, an index range scan was performed to retrieve the one row from table T1.  (Raise your hand if you know the answer why an index range scan operation was selected when every value in column C1 is unique and an equality predicate was used in the WHERE clause.)

For the following SQL statement, where an index on column C2 would likely have a large clustering factor value, should an index be used to retrieve 10% of the rows from the table?

SELECT
  C1,
  C2,
  C3,
  C4,
  SUBSTR(C5,1,10) C5
FROM
  T1
WHERE
  C2=9;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL')); 

Here is a portion of the output from the above SQL statement:

...
     99969          9          9 99969A AAAAAAAAAA
     99979          9          9 99979A AAAAAAAAAA
     99989          9          9 99989A AAAAAAAAAA
     99999          9          9 99999A AAAAAAAAAA

10000 rows selected.

SQL>
SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------
SQL_ID  459zvkh9p725k, child number 0
-------------------------------------
SELECT   C1,   C2,   C3,   C4,   SUBSTR(C5,1,10) C5 FROM   T1 WHERE
C2=9

Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |   252 (100)|          |
|*  1 |  TABLE ACCESS FULL| T1   | 10000 |  2138K|   252   (1)| 00:00:01 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C2"=9) 

The optimizer selected to use a full table scan in order to retrieve 10% of the rows from the table.  (Raise your hand if you know the answer why a full table scan was selected.)

A final SQL statement that also retrieves 10% of the rows from the table, this time using column C3:

SELECT
  C1,
  C2,
  C3,
  C4,
  SUBSTR(C5,1,10) C5
FROM
  T1
WHERE
  C3=9;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL')); 

For the above SQL statement, would you expect the optimizer to select the use of an index range scan operation, or a full table scan operation?  Here is the output that I received:

...
     99996          6          9 99996A AAAAAAAAAA
     99997          7          9 99997A AAAAAAAAAA
     99998          8          9 99998A AAAAAAAAAA
     99999          9          9 99999A AAAAAAAAAA

10000 rows selected.

SQL>
SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------
SQL_ID  4c1s2vujtc1ff, child number 0
-------------------------------------
SELECT   C1,   C2,   C3,   C4,   SUBSTR(C5,1,10) C5 FROM   T1 WHERE
C3=9

Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |   252 (100)|          |
|*  1 |  TABLE ACCESS FULL| T1   |  9091 |  1944K|   252   (1)| 00:00:01 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C3"=9) 

So, once again the optimizer selected to use a full table scan to facilitate the retrieval of 10% of the rows from the table.  But why, the clustering factor should be reasonably low.  (Raise your hand if you know the answer why a full table scan was selected.)

But wait… the optimizer from another 11.2.0.2 database instance decided differently:

...
     99996          6          9 99996A AAAAAAAAAA
     99997          7          9 99997A AAAAAAAAAA
     99998          8          9 99998A AAAAAAAAAA
     99999          9          9 99999A AAAAAAAAAA

10000 rows selected.

SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

PLAN_TABLE_OUTPUT
-----------------------------------------------------------------------------------------
SQL_ID  4c1s2vujtc1ff, child number 0
-------------------------------------
SELECT   C1,   C2,   C3,   C4,   SUBSTR(C5,1,10) C5 FROM   T1 WHERE
C3=9

Plan hash value: 1220227203

-----------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |       |       |   304 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T1        |  9091 |  1944K|   304   (1)| 00:00:02 |
|*  2 |   INDEX RANGE SCAN          | IND_T1_C3 |  9091 |       |    18   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C3"=9) 

Why did the optimizer in one of the 11.2.0.2 database instances select to use a full table scan operation, while the optimizer in the other database instance selected to use an index range scan operation?  (Raise your hand if you know the answer why there is a difference between the two execution plan results.)

—

If we take another look at the execution plan for the SQL statement that simulates the problem experienced by the OP, I wonder if we are able to determine why the parent operation has a lower calculated cost than its child (or grandchild’s) calculated cost?

SELECT /*+ GATHER_PLAN_STATISTICS */
  *
FROM
  T1
WHERE
  C4='';

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +COST')); 

Let’s take a look at the output of the above:

no rows selected

SQL>
SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +COST'));

PLAN_TABLE_OUTPUT
----------------------------------------------------------------------------------------
SQL_ID  63yz0tn58sz4v, child number 0
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS */   * FROM   T1 WHERE   C4=''

Plan hash value: 3332582666

----------------------------------------------------------------------------------------
| Id  | Operation          | Name | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   |
----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |      1 |        |     1 (100)|      0 |00:00:00.01 |
|*  1 |  FILTER            |      |      1 |        |            |      0 |00:00:00.01 |
|   2 |   TABLE ACCESS FULL| T1   |      0 |    100K|   252   (1)|      0 |00:00:00.01 |
----------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter(NULL IS NOT NULL) 

Do you see it?  Does it matter if a full table scan of table T1 was selected by the optimizer?   (Raise your hand if you know the answer.)

What if we force an index access path:

SELECT /*+ GATHER_PLAN_STATISTICS INDEX(T1) */
  *
FROM
  T1
WHERE
  C4='';

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +COST')); 

Here is the output of the above:

no rows selected

SQL>
SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +COST'));

PLAN_TABLE_OUTPUT
-------------------------------------------------------------------------------------------------------
SQL_ID  06ucccd8j8ss0, child number 0
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS INDEX(T1) */   * FROM   T1 WHERE
C4=''

Plan hash value: 1158503954

-------------------------------------------------------------------------------------------------------
| Id  | Operation                    | Name      | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   |
-------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |           |      1 |        |     1 (100)|      0 |00:00:00.01 |
|*  1 |  FILTER                      |           |      1 |        |            |      0 |00:00:00.01 |
|   2 |   TABLE ACCESS BY INDEX ROWID| T1        |      0 |    100K|  3323   (1)|      0 |00:00:00.01 |
|   3 |    INDEX FULL SCAN           | IND_T1_C3 |      0 |    100K|   196   (1)|      0 |00:00:00.01 |
-------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter(NULL IS NOT NULL) 

Is the above execution plan even legal because the SQL statement is essentially looking for NULL values?  (Raise your hand if you know the answer.)

—-

OK, put down your hand – the other people in the office are probably laughing at your hand-waving by now.  😉

I wonder if the OP now understands the problem with his SQL statement?

—–

Late Addition June 20, 2012:

Recall that the optimizer selected that a full table scan should be used for the initial SELECT statement.  Part 2 of a previous blog article on this blog pointed to an article on another blog, asking whether or not a couple of statements made on the other blog were true regarding whether or not NULL values were ever stored in a B*tree index structure.  One of my previous articles seemed to offer a counter-point, that in fact it is possible for a B*tree index to contain NULL values in certain situations.

What might happen if we swap in a bind variable in place of the literal in the initial SELECT statement?  If we set the bind variable to have a value of ” (the same value as the literal), will the optimizer select to use a full table scan operation or an index range scan operation?  Will the STARTS column in the execution plan contain 0 for one or more of the rows in the execution plan?  Let’s test:

VARIABLE V1 VARCHAR2
EXEC :V1:=''
 
SELECT /*+ GATHER_PLAN_STATISTICS */
  *
FROM
  T1
WHERE
  C4= :V1;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +COST')); 

Here is the output:

no rows selected
 
SQL>
SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +COST'));
 
PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------
SQL_ID  c4faj4cgbdrw1, child number 0
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS */   * FROM   T1 WHERE   C4= :V1
 
Plan hash value: 7035821
 
------------------------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   |
------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |      1 |        |     2 (100)|      0 |00:00:00.01 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T1        |      1 |      1 |     2   (0)|      0 |00:00:00.01 |
|*  2 |   INDEX RANGE SCAN          | IND_T1_C4 |      1 |      1 |     1   (0)|      0 |00:00:00.01 |
------------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C4"=:V1)

Notice that none of the rows in the execution plan have a STARTS value of 0, and that filter(NULL IS NOT NULL) does not appear in the Predicate Information section of the execution plan as had happened when we used a literal with the same value as the bind variable.  So, if the OP would like to see an index access path in the execution plan, perhaps he should use bind variables rather than literals?  Is the above execution plan more efficient or less efficient than the execution plan with the full table scan operation that was seen when a literal was passed in the SQL statement?

Let’s insert a row into table T4 with our bind variable value being placed in column C4:

INSERT INTO T1 VALUES(
  -1,
  -1,
  -1,
  :V1,
  'A');
 
COMMIT;
 
SELECT /*+ GATHER_PLAN_STATISTICS  FIND_ME */
  *
FROM
  T1
WHERE
  C4= :V1;
  
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +COST'));

Will a row be returned?  Will the execution plan change?  Here is the output for the above script:

no rows selected
 
SQL>
SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +COST'));
 
PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------
SQL_ID  csmkq5csd6f0a, child number 0
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS  FIND_ME */   * FROM   T1 WHERE   C4=
:V1
 
Plan hash value: 7035821
 
------------------------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   |
------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |      1 |        |     2 (100)|      0 |00:00:00.01 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T1        |      1 |      1 |     2   (0)|      0 |00:00:00.01 |
|*  2 |   INDEX RANGE SCAN          | IND_T1_C4 |      1 |      1 |     1   (0)|      0 |00:00:00.01 |
------------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C4"=:V1)

No rows selected, even though a row was just inserted into the table with the same value  as what is in bind variable V1.  Let’s DUMP the values of a couple of columns from the row that was just inserted:

COLUMN DUMP_C1 FORMAT A25
COLUMN DUMP_C4 FORMAT A25
 
SELECT
  C1,
  DUMP(C1) DUMP_C1,
  C4,
  DUMP(C4) DUMP_C4
FROM
  T1
WHERE
  C1=-1;
 
        C1 DUMP_C1                   C4     DUMP_C4
---------- ------------------------- ------ -------------------------
        -1 Typ=2 Len=3: 62,100,102          NULL

The above output shows that column C4 of this row contains a NULL value.  I suggest that it might not matter whether or not the execution plan shows an index access path or a full table scan for this SQL statement – if the SQL statement is answering a different question than what the OP expects, then it does not matter whether or not the SQL statement executes efficiently.

However, as a bonus we were able to see the optimizer using an index access path to check for a NULL value.

Execution Plan Changes when the OPTIMIZER_FEATURES_ENABLED Parameter is Changed, But Why?

May 30, 2012

A question appeared on the OTN Database forums yesterday that has yet to receive an answer.  The question was essentially, why did the execution plan change when the OPTIMIZER_FEATURES_ENABLED parameter was adjusted from 11.2.0.2 to 11.1.0.7, and why did the execution performance improve as a result of making the change?  A DBMS_XPLAN generated execution plan where TYPICAL +OUTLINE were apparently specified for the format parameters was provided for the execution with the OPTIMIZER_FEATURES_ENABLED parameter set to 11.2.0.2, while the format parameter was apparently set to ALLSTATS LAST for the 11.1.0.7 test execution.  Why did the performance improve when the optimizer’s parameter was modified?

We do not have access to the SQL statement, but we do have a few clues.  Take a look at the Predicate Information section, you will see the following:

1 - filter(ROWNUM<=1000)

So, ROWNUM<=1000 was specified in the SQL statement.  As mentioned in one of my other articles, there was a bug in the optimizer’s code prior to Oracle Database 11.2.0.1 related to the appearance of ROWNUM in the WHERE clause.  Would changing the  OPTIMIZER_FEATURES_ENABLED parameter to 11.1.0.7 reinstate the bug (this would not happen based on testing that I performed some time ago)?  The OP stated that the OPTIMIZER_MODE parameter is set to FIRST_ROWS_10, could specifying ROWNUM<=1000 cause the optimizer to behave as if FIRST_ROWS_1000 was set for the optimizer mode, behaving differently depending on the value of the OPTIMIZER_FEATURES_ENABLED parameter?

What other clues did the OP provide for potential readers of the OTN thread?  Let’s take a look at a portion of the outline sections supplied with the execution plans to see if there are any clues.  With OPTIMIZER_FEATURES_ENABLED set to 11.2.0.2:

OPTIMIZER_FEATURES_ENABLE('11.2.0.2')
DB_VERSION('11.2.0.2')
OPT_PARAM('_push_join_union_view' 'false')
OPT_PARAM('_optim_peek_user_binds' 'false')
OPT_PARAM('_gby_hash_aggregation_enabled' 'false')
OPT_PARAM('optimizer_index_cost_adj' 10)
OPT_PARAM('optimizer_index_caching' 80)
FIRST_ROWS(10)

With OPTIMIZER_FEATURES_ENABLED set to 11.1.0.7:

OPTIMIZER_FEATURES_ENABLE('11.1.0.7')
DB_VERSION('11.2.0.2')
OPT_PARAM('_optim_peek_user_binds' 'false')
OPT_PARAM('optimizer_index_cost_adj' 10)
OPT_PARAM('optimizer_index_caching' 80)
FIRST_ROWS(10)

Well, that is a bit interesting.  The OPTIMIZER_INDEX_COST_ADJ parameter was set to 10 in both cases, and the OPTIMIZER_INDEX_CACHING parameter was set to 80 in both cases – could the change in the execution plan be caused by a cost rounding error that is somehow slightly different when the OPTIMIZER_FEATURES_ENABLED parameter is changed?

While we are looking at the outline sections, notice that in the test with the OPTIMIZER_FEATURES_ENABLED parameter set at 11.2.0.2 that the _OPTIM_PEEK_USER_BINDS, _PUSH_JOIN_UNION_VIEW, and _GBY_HASH_AGGREGATION_ENABLED hidden parameters are all set at the non-default value of FALSE, while the test with the OPTIMIZER_FEATURES_ENABLED parameter set at 11.1.0.7 only shows that the _OPTIM_PEEK_USER_BINDS parameter is set to the non-default value of FALSE – could that difference cause of the execution plan change, and might those two hidden parameters magically change back to their default values when adjusting the OPTIMIZER_FEATURES_ENABLED parameter?

The OP did not supply a test case script, so I will create a simple test case script just to see if those two hidden parameters might magically change back to their default values when the  OPTIMIZER_FEATURES_ENABLED parameter is adjusted.

DROP TABLE T1 PURGE;

CREATE TABLE T1 AS
SELECT
  ROWNUM C1,
  DECODE(ROWNUM,1,1,0) C2,
  LPAD('A',255,'A') C3
FROM
  DUAL
CONNECT BY
  LEVEL<=10000;

CREATE UNIQUE INDEX IND_T1_C1 ON T1(C1);
CREATE INDEX IND_T1_C2 ON T1(C2);

ALTER TABLE T1 MODIFY (C1 NOT NULL, C2 NOT NULL);

EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T1',CASCADE=>TRUE,ESTIMATE_PERCENT=>100,METHOD_OPT=>'FOR ALL INDEXED COLUMNS SIZE 254') 

SET PAGESIZE 1000
SET LINESIZE 140

It appears that the OP had the following parameters set for the execution at 11.2.0.2:

ALTER SESSION SET "_push_join_union_view"=false;
ALTER SESSION SET "_optim_peek_user_binds"=false;
ALTER SESSION SET "_gby_hash_aggregation_enabled"=false;
ALTER SESSION SET optimizer_index_cost_adj=10;
ALTER SESSION SET optimizer_index_caching=80;
ALTER SESSION SET OPTIMIZER_MODE=FIRST_ROWS_10;

Let’s try a test script (note that you may wish to change the event 10132 to 10053, because the output of the 10132 trace event seems to be a bit scrambled in Oracle Database 11.2.0.2):

ALTER SESSION SET OPTIMIZER_FEATURES_ENABLE='11.2.0.2';
ALTER SESSION SET TRACEFILE_IDENTIFIER='11.2.0.2 HIDDEN TEST';
ALTER SESSION SET EVENTS '10132 TRACE NAME CONTEXT FOREVER, LEVEL 1';

SELECT
  *
FROM
  T1
WHERE
  ROWNUM<=1000
  AND C2=2
  AND C1>=1;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'+OUTLINE'));

ALTER SESSION SET OPTIMIZER_FEATURES_ENABLE='11.1.0.7';
ALTER SESSION SET TRACEFILE_IDENTIFIER='11.1.0.7 HIDDEN TEST';

SELECT
  *
FROM
  T1
WHERE
  ROWNUM<=1000
  AND C2=2
  AND C1>=1;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'+OUTLINE'));
ALTER SESSION SET EVENTS '10132 TRACE NAME CONTEXT OFF';

Here is the output that I received for 11.2.0.2:

SQL> SELECT
  2    *
  3  FROM
  4    T1
  5  WHERE
  6    ROWNUM<=1000
  7    AND C2=2
  8    AND C1>=1;

no rows selected

SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'+OUTLINE'));

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------
SQL_ID  fb2ysnjkbq366, child number 0
-------------------------------------
SELECT   * FROM   T1 WHERE   ROWNUM<=1000   AND C2=2   AND C1>=1

Plan hash value: 1016793414

------------------------------------------------------------------------------------------
| Id  | Operation                    | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |           |       |       |     1 (100)|          |
|*  1 |  COUNT STOPKEY               |           |       |       |            |          |
|*  2 |   TABLE ACCESS BY INDEX ROWID| T1        |     1 |   136 |     1   (0)| 00:00:01 |
|*  3 |    INDEX RANGE SCAN          | IND_T1_C2 |     1 |       |     1   (0)| 00:00:01 |
------------------------------------------------------------------------------------------

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('11.2.0.2')
      DB_VERSION('11.2.0.2')
      OPT_PARAM('_push_join_union_view' 'false')
      OPT_PARAM('_optim_peek_user_binds' 'false')
      OPT_PARAM('_gby_hash_aggregation_enabled' 'false')
      OPT_PARAM('optimizer_index_cost_adj' 10)
      OPT_PARAM('optimizer_index_caching' 80)
      FIRST_ROWS(10)
      OUTLINE_LEAF(@"SEL$1")
      INDEX_RS_ASC(@"SEL$1" "T1"@"SEL$1" ("T1"."C2"))
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter(ROWNUM<=1000)
   2 - filter("C1">=1)
   3 - access("C2"=2) 

The output for 11.1.0.7:

SQL_ID  fb2ysnjkbq366, child number 1
-------------------------------------
SELECT   * FROM   T1 WHERE   ROWNUM<=1000   AND C2=2   AND C1>=1

Plan hash value: 1016793414

------------------------------------------------------------------------------------------
| Id  | Operation                    | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |           |       |       |     1 (100)|          |
|*  1 |  COUNT STOPKEY               |           |       |       |            |          |
|*  2 |   TABLE ACCESS BY INDEX ROWID| T1        |     1 |   136 |     1   (0)| 00:00:01 |
|*  3 |    INDEX RANGE SCAN          | IND_T1_C2 |     1 |       |     1   (0)| 00:00:01 |
------------------------------------------------------------------------------------------

Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('11.1.0.7')
      DB_VERSION('11.2.0.2')
      OPT_PARAM('_push_join_union_view' 'false')
      OPT_PARAM('_optim_peek_user_binds' 'false')
      OPT_PARAM('_gby_hash_aggregation_enabled' 'false')
      OPT_PARAM('optimizer_index_cost_adj' 10)
      OPT_PARAM('optimizer_index_caching' 80)
      FIRST_ROWS(10)
      OUTLINE_LEAF(@"SEL$1")
      INDEX_RS_ASC(@"SEL$1" "T1"@"SEL$1" ("T1"."C2"))
      END_OUTLINE_DATA
  */

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter(ROWNUM<=1000)
   2 - filter("C1">=1)
   3 - access("C2"=2) 

—

How would you answer the OP?  Could it just be the case that most of the required database blocks were in the buffer cache on the second execution when the OPTIMIZER_FEATURES_ENABLED parameter was adjusted?  If nothing else, this OTN thread provides an excuse to think about possible causes and effects when insufficient information is provided.

TKPROF Elapsed Time Challenge – the Elapsed Time is Half of the Wait Event Time

May 20, 2012

An interesting quirk was recently brought to my attention by Mich Talebzadeh.  He generated a 10046 trace at level 8 for a session, executed some SQL statements, disabled the trace, and then processed the resulting trace file with TKPROF.  His TKPROF output included the following:

UPDATE TESTWRITES SET PADDING1 = RPAD('y',4000,'y')
WHERE
OBJECT_ID = :B1

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        4      0.00       0.00          0          0          0           0
Execute  17293     21.62    1245.41    1064667    2411398    6280588     1000000
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total    17297     21.62    1245.41    1064667    2411398    6280588     1000000

Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 93     (recursive depth: 1)

Rows     Row Source Operation
-------  ---------------------------------------------------
      0  UPDATE  TESTWRITES (cr=214 pr=427 pw=0 time=0 us)
    100   INDEX UNIQUE SCAN TESTWRITES_PK (cr=110 pr=3 pw=0 time=0 us cost=2 size=4007 card=1)(object id 92380)

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  Disk file operations I/O                        2        0.00          0.00
  db file sequential read                   2160263        1.57       2304.22
  db file scattered read                      27769        0.26         99.78 

Mich noticed that the reported elapsed time, at 1,245.41 seconds, is less than the sum of the wait event times, at 2,400.00 seconds.  Typically, the sum of the wait events plus the CPU time should be equal to or slightly less than the elapsed time (assuming that parallel query is not used, and the CPU reporting granularity did not cause odd side effects, and the session’s process did not spend much time in the CPU run queue).  In this case, the elapsed time is about 51.9% of the wait event time for this SQL statement. 

How is this possible?  Mich stated the following:

“A comment has been it is due to using different clock granularities.”

Think about that for a minute – what could explain the odd inconsistency in the output?

—

—

—

—

—

—

—

—

I recall reading in the book “Optimizing Oracle Performance” about double-counting of statistics due to recursive database calls.  Page 92 of that book includes the following quote:

“In Oracle releases through at least Oracle9i Release 2, a database call’s c, e, p, cr, and cu statistics contain an aggregation of the resources consumed by the database call itself and its entire recursive progeny.”

Page 91 of the same book makes the following statement:

“The rule for determining the recursive relationships among database calls is simple:
A database call with dep=n + 1 is the recursive child of the first subsequent dep=n database call listed in the SQL trace data stream.”

Could it be the case that this particular issue with recursive calls was addressed in later releases of TKPROF?  Might that fix explain why the elapsed time is 51.9% of the wait event time?

—

—

—

—

—

—

—

—

The OP provided a partial test case script with a description of how the tables were created.  I reduced the size of the original source table from 1,700,000 rows to 60,000 rows, and reduced the secondary source table from 1,000,000 rows to 50,000 rows.  My test script follows (note that the line that builds the IND_T2_OBJ index is commented out – we will change that later):

DROP TABLE T1 PURGE;
DROP TABLE T2 PURGE;

CREATE TABLE T1 AS
SELECT
  *
FROM
  ALL_OBJECTS
WHERE
  ROWNUM<=60000;

ALTER TABLE T1 ADD (
  PADDING1 VARCHAR2(4000),
  PADDING2 VARCHAR2(4000));

CREATE TABLE T2 AS
SELECT
  *
FROM
  T1
WHERE
  ROWNUM<=50000;

CREATE UNIQUE INDEX IND_T1_OBJ ON T1(OBJECT_ID);
--CREATE UNIQUE INDEX IND_T2_OBJ ON T2(OBJECT_ID);

ALTER SYSTEM FLUSH SHARED_POOL;

ALTER SYSTEM FLUSH BUFFER_CACHE;
ALTER SYSTEM FLUSH BUFFER_CACHE;

EXEC DBMS_MONITOR.SESSION_TRACE_ENABLE ( waits=>true, plan_stat=>'ALL_EXECUTIONS' );

DECLARE
        type ObjIdArray is table of T1.object_id%TYPE index by binary_integer;
        l_ids objIdArray;
        CURSOR c IS SELECT object_id FROM T1;
BEGIN
        OPEN c;
        LOOP
                BEGIN
                        FETCH c BULK COLLECT INTO l_ids LIMIT 100;

                        FORALL rs in 1 .. l_ids.COUNT
                                UPDATE T2
                                  SET PADDING1 =  RPAD('y',4000,'y')
                                WHERE object_id = l_ids(rs);
                        commit;
                        FORALL rs in 1 .. l_ids.COUNT
                                DELETE FROM T2
                                WHERE object_id = l_ids(rs);
                        commit;
                        FORALL rs in 1 .. l_ids.COUNT
                                INSERT INTO T2
                                SELECT * FROM T1 t WHERE t.object_id = l_ids(rs);
                        commit;
                        EXIT WHEN C%NOTFOUND;
                EXCEPTION
                  WHEN NO_DATA_FOUND THEN
                    NULL;
                  WHEN OTHERS THEN
                    DBMS_OUTPUT.PUT_LINE('Transaction failed');
                END;
        END LOOP;
        CLOSE c;
END;
/

EXEC DBMS_MONITOR.SESSION_TRACE_DISABLE;

That test case definitely contains calls in the trace file at a dep greater than zero.  As a matter of fact, the SQL statement of interest is at a dep of one, resulting in space management (and other calls) at a dep of two or greater:

PARSING IN CURSOR #397285536 len=66 dep=1 uid=64 oct=6 lid=64 tim=1879257875778 hv=3592437181 ad='3edea5858' sqlid='1ahnbczb20gdx'
UPDATE T2 SET PADDING1 = RPAD('y',4000,'y') WHERE OBJECT_ID = :B1 
END OF STMT
PARSE #397285536:c=0,e=169,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=1,plh=0,tim=1879257875778

I executed the test case script on 64 bit Windows with Oracle Database 11.2.0.2 with patch 5 installed.  I processed the resulting trace file with TKPROF from the 32 bit Windows Oracle 11.2.0.1 client with the ODBC update patch 10155837 installed.  My TKPROF output for the UPDATE statement showed the following:

SQL ID: 1ahnbczb20gdx
Plan Hash: 2722410703
UPDATE T2 SET PADDING1 = RPAD('y',4000,'y') 
WHERE
 OBJECT_ID = :B1

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute    600    197.38     197.60        452   50299237     426582       50000
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total      601    197.38     197.60        452   50299237     426582       50000

Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 64     (recursive depth: 1)
Number of plan statistics captured: 1

Rows (1st) Rows (avg) Rows (max)  Row Source Operation
---------- ---------- ----------  ---------------------------------------------------
         0          0          0  UPDATE  T2 (cr=80409 pr=436 pw=0 time=357525 us)
       100        100        100   TABLE ACCESS FULL T2 (cr=80182 pr=420 pw=0 time=337295 us cost=116 size=16120 card=8)
         0          0          0  UPDATE  T2 (cr=83795 pr=0 pw=0 time=344133 us)
       100        100        100   TABLE ACCESS FULL T2 (cr=83598 pr=0 pw=0 time=332180 us cost=116 size=16120 card=8)
         0          0          0  UPDATE  T2 (cr=83790 pr=2 pw=0 time=303134 us)
       100        100        100   TABLE ACCESS FULL T2 (cr=83596 pr=0 pw=0 time=288815 us cost=116 size=16120 card=8)
         0          0          0  UPDATE  T2 (cr=83789 pr=2 pw=0 time=283738 us)
       100        100        100   TABLE ACCESS FULL T2 (cr=83595 pr=0 pw=0 time=271190 us cost=116 size=16120 card=8)
...
         0          0          0  UPDATE  T2 (cr=87900 pr=0 pw=0 time=375297 us)
         0          0          0   TABLE ACCESS FULL T2 (cr=87900 pr=0 pw=0 time=375015 us cost=116 size=16120 card=8)

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file sequential read                        36        0.00          0.01
  db file scattered read                         31        0.00          0.01

I decided to compare the output of TKPROF with the output of my Hyper-Extended Oracle Performance Monitor program, which I know does not attempt to resolve parent and child recursive statistics:

Statement Depth 1 (Trigger Code)
Cursor 220   Ver 1   Parse at 0.254283 SQL_ID='1ahnbczb20gdx'  (TD Prev 0.000877)  Similar Cnt 1
|PARSEs       1|CPU S    0.000000|CLOCK S    0.000169|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|EXECs      600|CPU S  197.403668|CLOCK S  197.659011|ROWs    50000|PHY RD BLKs       774|CON RD BLKs (Mem)  50299674|CUR RD BLKs (Mem)    426582|SHARED POOL MISs      1|
|FETCHs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
  CPU S 25.08%  CLOCK S 25.01%
|               +++++||               +++++|
UPDATE T2 SET PADDING1 = RPAD('y',4000,'y') WHERE OBJECT_ID = :B1

       (Rows 0)   UPDATE  T2 (cr=80409 pr=436 pw=0 time=357525 us)
     (Rows 100)    TABLE ACCESS FULL T2 (cr=80182 pr=420 pw=0 time=337295 us cost=116 size=16120 card=8)

       (Rows 0)   UPDATE  T2 (cr=83795 pr=0 pw=0 time=344133 us)
     (Rows 100)    TABLE ACCESS FULL T2 (cr=83598 pr=0 pw=0 time=332180 us cost=116 size=16120 card=8)
       (Rows 0)   UPDATE  T2 (cr=83790 pr=2 pw=0 time=303134 us)
     (Rows 100)    TABLE ACCESS FULL T2 (cr=83596 pr=0 pw=0 time=288815 us cost=116 size=16120 card=8)
...

Notice that the TKPROF summary shows 0.059011 fewer elapsed seconds for this SQL statement, 0.023668 fewer CPU seconds, 322 fewer blocks read from disk, 437 fewer consistent gets, and the same number of current mode reads.  Why the differences?

My program wrote the wait event details for the UPDATE statement to an Excel spreadsheet.  Examination of the wait events indicated that there were 36 db file sequential read waits (thus 36 total blocks read from disk), and 31 db file scattered read waits (416 total blocks read from disk) – the wait event details support the statistics output by TKPROF.  So, maybe the double counting of recursive calls was fixed since the release of Oracle Database 9.2.  Nice!  But we are not done yet, we still do not have the index on table T2, as was described by the OP.

Locate in the script the following line:

--CREATE UNIQUE INDEX IND_T2_OBJ ON T2(OBJECT_ID);

Change that line by removing the — at the start of the line, so that the line appears like this:

CREATE UNIQUE INDEX IND_T2_OBJ ON T2(OBJECT_ID);

Disconnect from the database, then reconnect (so that you will be assigned a different default trace filename).  Now, re-execute the script, and process the resulting trace file with TKPROF.  My TKPROF output for the UPDATE SQL statement looks like this:

SQL ID: 1ahnbczb20gdx
Plan Hash: 1751280057
UPDATE T2 SET PADDING1 = RPAD('y',4000,'y') 
WHERE
 OBJECT_ID = :B1 

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute    190      1.35       1.42        348      52552     165435       19000
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total      191      1.35       1.42        348      52552     165435       19000

Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 64     (recursive depth: 1)
Number of plan statistics captured: 1

Rows (1st) Rows (avg) Rows (max)  Row Source Operation
---------- ---------- ----------  ---------------------------------------------------
         0          0          0  UPDATE  T2 (cr=332 pr=30 pw=0 time=19223 us)
       100        100        100   INDEX UNIQUE SCAN IND_T2_OBJ (cr=105 pr=8 pw=0 time=1372 us cost=1 size=2015 card=1)(object id 103855)
         0          0          0  UPDATE  T2 (cr=288 pr=2 pw=0 time=8629 us)
       100        100        100   INDEX UNIQUE SCAN IND_T2_OBJ (cr=105 pr=0
...
         0          0          0  UPDATE  T2 (cr=291 pr=0 pw=0 time=6051 us)
       100        100        100   INDEX UNIQUE SCAN IND_T2_OBJ (cr=105 pr=0 pw=0 time=430 us cost=1 size=2015 card=1)(object id 103855)

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file scattered read                         40        0.00          0.03
  db file sequential read                        32        0.00          0.01
  log file switch completion                      2        0.00          0.01

For the sake of comparison, I also processed the trace file using my Hyper-Extended Oracle Performance Monitor program.  This is my program’s output:

Statement Depth 1 (Trigger Code)
Cursor 910   Ver 1   Parse at 0.187068 SQL_ID='1ahnbczb20gdx'  (TD Prev 0.000996)  Similar Cnt 1
|PARSEs       1|CPU S    0.000000|CLOCK S    0.000162|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|EXECs      600|CPU S    3.603618|CLOCK S    3.683344|ROWs    50000|PHY RD BLKs       912|CON RD BLKs (Mem)    144826|CUR RD BLKs (Mem)    426496|SHARED POOL MISs      1|
|FETCHs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
  CPU S 16.91%  CLOCK S 15.45%
|                 +++||                 +++|
UPDATE T2 SET PADDING1 = RPAD('y',4000,'y') WHERE OBJECT_ID = :B1

       (Rows 0)   UPDATE  T2 (cr=332 pr=30 pw=0 time=19223 us)
     (Rows 100)    INDEX UNIQUE SCAN IND_T2_OBJ (cr=105 pr=8 pw=0 time=1372 us cost=1 size=2015 card=1)

       (Rows 0)   UPDATE  T2 (cr=288 pr=2 pw=0 time=8629 us)
     (Rows 100)    INDEX UNIQUE SCAN IND_T2_OBJ (cr=105 pr=0 pw=0 time=430 us cost=1 size=2015 card=1)
...
       (Rows 0)   UPDATE  T2 (cr=105 pr=0 pw=0 time=193 us)
       (Rows 0)    INDEX UNIQUE SCAN IND_T2_OBJ (cr=105 pr=0 pw=0 time=153 us cost=1 size=2015 card=1)

Notice that there is now a significant difference in the elapsed time at 2.263344 seconds difference, CPU seconds at a difference of 2.253618 seconds, and the number of blocks read from disk at a difference of 564 blocks.

Let’s take a look at the wait events that my program wrote to Excel.   Examination of the wait events indicated that there were 42 db file sequential read waits (thus 42 total blocks read from disk), and 107 db file scattered read waits (842 total blocks read from disk), for a grand total of 884 blocks read from disk that are associated with this UPDATE SQL statement.  This time, not only are the TKPROF statistics not supported by the raw wait events, but the wait event summary included in the TKPROF summary also is not supported by the raw wait events.  Where did those number from TKPROF originate?

—

—

—

—

—

Well, let’s just process the 10046 trace file using TKPROF from 11.2.0.2 to confirm the output:

SQL ID: 1ahnbczb20gdx Plan Hash: 1751280057

UPDATE T2 SET PADDING1 = RPAD('y',4000,'y') 
WHERE
 OBJECT_ID = :B1 

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute    600      3.57       3.64        884     144451     426496       50000
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total      601      3.57       3.64        884     144451     426496       50000

Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 64     (recursive depth: 1)
Number of plan statistics captured: 600

Rows (1st) Rows (avg) Rows (max)  Row Source Operation
---------- ---------- ----------  ---------------------------------------------------
         0          0          0  UPDATE  T2 (cr=241 pr=1 pw=0 time=5550 us)
       100         83        100   INDEX UNIQUE SCAN IND_T2_OBJ (cr=106 pr=0 pw=0 time=425 us cost=1 size=2015 card=1)(object id 103855)

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file scattered read                        107        0.00          0.06
  db file sequential read                        42        0.00          0.01
  log file switch completion                      2        0.00          0.01

–

Those TKPROF numbers appear quite different; translation – it you do not like the numbers produced by TKPROF, just use a different version of TKPROF.  🙂

It appears that TKPROF in 11.2.0.1 potentially lies when reporting on a trace file generated by Oracle Database 11.2.0.2.  I wonder if this problem is caused by the large cursor number in 11.2.0.2+, or if TKPROF in 11.2.0.1 also has the same problem with trace files generated by Oracle Database 11.2.0.1?

Is anyone able to reproduce the problem reported by the OP?

—

If you wish to experiment with my trace file and TKPROF summaries, you may download the following file:

or1122p_ora_1808.zip

When you download the file, rename the download from or1122p_ora_1808-zip.doc to or1122p_ora_1808.zip.  The enclosed or1122p_ora_1808.trc file is the raw 10446 trace file, or1122p_ora_1808.txt is the 11.2.0.1 TKPROF generated output file, and or1122p_ora_1808-2.txt is the 11.2.0.2 generated output file.

Which PLAN_HASH_VALUE Appears in V$SQLAREA?

March 28, 2012

A recent question on the OTN forums asked which PLAN_HASH_VALUE appears in V$SQLAREA when there are multiple child cursors for a single SQL_ID value, when some child cursors have a different execution plan.  Certainly, this bit of information must be in the Oracle Database documentation.  Let’s check the V$SQLAREA documentation for Oracle Database 11.2:

“Numeric representation of the SQL plan for this cursor. Comparing one PLAN_HASH_VALUE to another easily identifies whether or not two plans are the same (rather than comparing the two plans line by line).”

Well, that was not helpful, but it does remind me of something that I saw when I went out for a drive in the countryside this past weekend (all within about a 50 meter radius – click a picture to see a larger view of the picture):

 

OK, now that the initial frustration of not obtaining an answer from the documentation has subsided, let’s put together a quick test case to see if we are able to help the OP find an answer to his question.  We will borrow a slightly modified version of a test script that generates skewed data which was used in another article: 

CREATE TABLE T1 AS
SELECT
  ROWNUM C1,
  DECODE(ROWNUM,1,1,0) C2,
  LPAD('A',255,'A') C3
FROM
  DUAL
CONNECT BY
  LEVEL<=10000;

CREATE UNIQUE INDEX IND_T1_C1 ON T1(C1);
CREATE INDEX IND_T1_C2 ON T1(C2);

ALTER TABLE T1 MODIFY (C1 NOT NULL, C2 NOT NULL);

EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T1',CASCADE=>TRUE,ESTIMATE_PERCENT=>100,METHOD_OPT=>'FOR ALL INDEXED COLUMNS SIZE 254') 

For the initial test (in Oracle Database 11.2.0.2), I will use the BIND_AWARE hint to save Oracle from having to determine that the execution plan could (should) depend on the bind variable value, rather than having to rely on adaptive cursor sharing to eventually obtain the same effect:

SET LINESIZE 120
SET PAGESIZE 1000

VARIABLE V1 NUMBER
EXEC :V1:=1

SELECT /*+ BIND_AWARE */
  C1,
  C2,
  C3
FROM
  T1
WHERE
  C2=:V1;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

SQL_ID  7p4yxrzwwuybt, child number 0
-------------------------------------
SELECT /*+ BIND_AWARE */   C1,   C2,   C3 FROM   T1 WHERE   C2=:V1

Plan hash value: 236868917

-----------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |       |       |     2 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T1        |     1 |   136 |     2   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | IND_T1_C2 |     1 |       |     1   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V1)

SELECT
  PLAN_HASH_VALUE
FROM
  V$SQLAREA
WHERE
  SQL_ID='7p4yxrzwwuybt';

PLAN_HASH_VALUE
---------------
      236868917 

As shown above, the PLAN_HASH_VALUE has a value of 236868917 in V$SQLAREA, which is the same value (shown in the execution plan) of the most recently executed child number.

Let’s repeat the previous SQL statements, this time with a different bind variable value:

EXEC :V1:=0

SELECT /*+ BIND_AWARE */
  C1,
  C2,
  C3
FROM
  T1
WHERE
  C2=:V1;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

SQL_ID  7p4yxrzwwuybt, child number 1
-------------------------------------
SELECT /*+ BIND_AWARE */   C1,   C2,   C3 FROM   T1 WHERE   C2=:V1

Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |    33 (100)|          |
|*  1 |  TABLE ACCESS FULL| T1   |  9999 |  1327K|    33   (0)| 00:00:01 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C2"=:V1)

SELECT
  PLAN_HASH_VALUE
FROM
  V$SQLAREA
WHERE
  SQL_ID='7p4yxrzwwuybt';

PLAN_HASH_VALUE
---------------
     3617692013 

As shown above, the execution plan changed, thus the Plan hash value in the execution plan changed to 3617692013, and that change corresponded with the PLAN_HASH_VALUE for the SQL_ID in V$SQLAREA changing to the value 3617692013 – the same value shown in the execution plan for the most recently executed child number.

Let’s trying again without changing the bind variable value to see what happens:

SELECT /*+ BIND_AWARE */
  C1,
  C2,
  C3
FROM
  T1
WHERE
  C2=:V1;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

SQL_ID  7p4yxrzwwuybt, child number 1
-------------------------------------
SELECT /*+ BIND_AWARE */   C1,   C2,   C3 FROM   T1 WHERE   C2=:V1

Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |    33 (100)|          |
|*  1 |  TABLE ACCESS FULL| T1   |  9999 |  1327K|    33   (0)| 00:00:01 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C2"=:V1)

SELECT
  PLAN_HASH_VALUE
FROM
  V$SQLAREA
WHERE
  SQL_ID='7p4yxrzwwuybt';

PLAN_HASH_VALUE
---------------
     3617692013 

As shown above, the PLAN_HASH_VALUE in V$SQLAREA remained at the value 3617692013, which is the PLAN_HASH_VALUE of the most recently executed child number.

Let’s switch back to the original bind variable value to see what happens in V$SQLAREA:

EXEC :V1:=1

SELECT /*+ BIND_AWARE */
  C1,
  C2,
  C3
FROM
  T1
WHERE
  C2=:V1;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

SQL_ID  7p4yxrzwwuybt, child number 0
-------------------------------------
SELECT /*+ BIND_AWARE */   C1,   C2,   C3 FROM   T1 WHERE   C2=:V1

Plan hash value: 236868917

-----------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |       |       |     2 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T1        |     1 |   136 |     2   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | IND_T1_C2 |     1 |       |     1   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V1)

SELECT
  PLAN_HASH_VALUE
FROM
  V$SQLAREA
WHERE
  SQL_ID='7p4yxrzwwuybt';

PLAN_HASH_VALUE
---------------
      236868917 

As shown above, the PLAN_HASH_VALUE for the SQL_ID found in V$SQLAREA switched back to the original value – it is again showing the PLAN_HASH_VALUE of the most recently executed child cursor.

—

But wait, there’s more information.  Pete Finnigan recently reminded me of a problem that I had several years ago when I tried to untangle the string of synonyms and views to see the definition of various Oracle Database performance views.  He not only reminded me of the problem that I had years ago, but showed me the process of untangling the (evoke suitable picture from above) that I learned and forgot several times over the subsequent years.

Let’s get started by determining what the V$SQLAREA synonym points at:

SET LONG 9000
COLUMN TABLE_OWNER FORMAT A11

SELECT
  TABLE_OWNER,
  TABLE_NAME
FROM
  DBA_SYNONYMS
WHERE
  SYNONYM_NAME='V$SQLAREA';

TABLE_OWNER TABLE_NAME
----------- ----------
SYS         V_$SQLAREA 

A viewed named V_$SQLAREA – now what?  Let’s see the definition of that view:

SELECT
  TEXT
FROM
  DBA_VIEWS
WHERE
  VIEW_NAME='V_$SQLAREA';

select "SQL_TEXT","SQL_FULLTEXT","SQL_ID","SHARABLE_MEM","PERSISTENT_MEM","RUNTI
...
D_TOTAL","PINNED_TOTAL","IO_CELL_UNCOMPRESSED_BYTES","IO_CELL_OFFLOAD_RETURNED_B
YTES" from v$sqlarea 

So, the synonym V$SQLAREA points to the view V_$SQLAREA which selects from V$SQLAREA … that name seems oddly familiar.

SELECT
  VIEW_DEFINITION
FROM
  V$FIXED_VIEW_DEFINITION
WHERE
  VIEW_NAME='V$SQLAREA';

select    SQL_TEXT,           SQL_FULLTEXT,           SQL_ID,           SHARABLE
_MEM,           PERSISTENT_MEM,           RUNTIME_MEM,           SORTS,
...
SICAL_WRITE_BYTES,            OPTIMIZED_PHY_READ_REQUESTS,            LOCKED_TOT
AL,             PINNED_TOTAL,            IO_CELL_UNCOMPRESSED_BYTES,
IO_CELL_OFFLOAD_RETURNED_BYTES from GV$SQLAREA where inst_id = USERENV('Instance
') 

So, the view V_$SQLAREA selects from V$SQLAREA which selects from GV$SQLAREA.

SELECT
  VIEW_DEFINITION
FROM
  V$FIXED_VIEW_DEFINITION
WHERE
  VIEW_NAME='GV$SQLAREA';

select inst_id,kglnaobj,kglfnobj,kglobt03,kglobhs0+kglobhs1+kglobhs2+kglobhs3+kg
lobhs4+kglobhs5+kglobhs6,kglobt08+kglobt11,kglobt10,kglobt01,kglobccc,kglobclc,k
...
t58,kglobt23,kglobt24,kglobt59,kglobt53 - ((kglobt55+kglobt57) - kglobt52)from
x$kglcursor_child_sqlid where kglobt02 != 0

So, the view V_$SQLAREA selects from V$SQLAREA which selects from GV$SQLAREA which selects from X$KGLCURSOR_CHILD_SQLID where KGLOBT02 != 0 (COMMAND_TYPE column in V$SQLAREA). (Note, must be logged in as the SYS user for the following SQL statement.)

SELECT
  KQFDTNAM,
  KQFDTEQU
FROM
  X$KQFDT
ORDER BY
  KQFDTNAM;

KQFDTNAM                       KQFDTEQU
----------------------------- -------------
...
X$KGLBODY                      X$KGLOB
X$KGLCLUSTER                   X$KGLOB
X$KGLCURSOR                    X$KGLOB
X$KGLCURSOR_CHILD              X$KGLOB
X$KGLCURSOR_CHILD_SQLID        X$KGLOB
X$KGLCURSOR_CHILD_SQLIDPH      X$KGLOB
X$KGLINDEX                     X$KGLOB
X$KGLTABLE                     X$KGLOB
X$KGLTRIGGER                   X$KGLOB 
...

So, in summary the synonym V$SQLAREA points to the view V_$SQLAREA which selects from V$SQLAREA which selects from GV$SQLAREA which selects from X$KGLCURSOR_CHILD_SQLID where KGLOBT02 != 0, which has as a base table of X$KGLOB just like 8 other fixed tables.  If you repeat the above steps for V$SQL, you will find that it is based on X$KGLCURSOR_CHILD, which also has X$KGLOB as its base table.

Here is a piece of SQL that joins the underlying fixed table for V$SQLAREA with V$SQL to hopefully determine if the PLAN_HASH_VALUE for the most recently executed child cursor for each SQL_ID is always what appears in V$SQLAREA (I do not suggest running this SQL statement in a production environment – the test database instance in my case was bounced a couple of hours ago).  The word DIFFERENT is output if the LAST_ACTIVE_TIME from V$SQLAREA does NOT match the LAST_ACTIVE_TIME for a specific child in V$SQL (Note, must be logged in as the SYS user for the following SQL statement.):

SELECT
  X.KGLOBT03 SQL_ID,
  X.KGLOBT30 PLAN_HASH_VALUE,
  TO_CHAR(X.KGLOBCLA,'HH24:MI:SS') LAST_ACTIVE_TIME,
  S.CHILD_NUMBER,
  S.PLAN_HASH_VALUE S_PLAN_HASH_VALUE,
  TO_CHAR(S.LAST_ACTIVE_TIME,'HH24:MI:SS') S_LAST_ACTIVE_TIME,
  DECODE(X.KGLOBCLA,S.LAST_ACTIVE_TIME,NULL,'DIFFERENT') TIMESTAMP_DIFFERS
FROM
  X$KGLCURSOR_CHILD_SQLID X,
  V$SQL S
WHERE
  X.KGLOBT02 != 0
  AND X.KGLOBT03=S.SQL_ID
ORDER BY
  S.SQL_ID,
  S.CHILD_NUMBER;

SQL_ID        PLAN_HASH_VALUE LAST_ACT CHILD_NUMBER S_PLAN_HASH_VALUE S_LAST_A TIMESTAMP
------------- --------------- -------- ------------ ----------------- -------- ---------
00fx7adv5q5gm      2256887934 22:03:44            0        2256887934 22:03:44
00yp7w9j0yqma      3600061239 20:28:45            0        3600061239 20:28:45
...
07pcqtmt58zv9      1964643588 20:44:48            0        1964643588 20:28:45 DIFFERENT
07pcqtmt58zv9      1964643588 20:44:48            1        1964643588 20:44:48
...
0fr8zhn4ymu3v      1231101765 21:03:43            0        3815847153 20:28:45 DIFFERENT
0fr8zhn4ymu3v      1231101765 21:03:43            1        1231101765 21:03:43
...
0kugqg48477gf       643665366 21:03:43            0         828554155 20:28:45 DIFFERENT
0kugqg48477gf       643665366 21:03:43            1         643665366 21:03:43
...
0m78skf1mudnb      3506511888 20:28:47            0        3506511888 20:28:45 DIFFERENT
0m78skf1mudnb      3506511888 20:28:47            1        3506511888 20:28:47
...
15w1t8qpdgg5k      2628186673 21:33:45            0        2628186673 20:28:45 DIFFERENT
15w1t8qpdgg5k      2628186673 21:33:45            1        2628186673 21:33:45
...
1gu8t96d0bdmu      2035254952 21:03:43            0        2035254952 21:03:43
1gu8t96d0bdmu      2035254952 21:03:43            1        3526770254 20:28:45 DIFFERENT
1gu8t96d0bdmu      2035254952 21:03:43            2        2035254952 20:48:45 DIFFERENT
...
2jr8c42qx700h      2445831428 20:28:55            0        3034582372 20:28:45 DIFFERENT
2jr8c42qx700h      2445831428 20:28:55            1        2445831428 20:28:45 DIFFERENT
2jr8c42qx700h      2445831428 20:28:55            2        2445831428 20:28:55
...
2q93zsrvbdw48      2874733959 21:03:43            0        2874733959 21:03:43
2q93zsrvbdw48      2874733959 21:03:43            1        2874733959 20:28:45 DIFFERENT
2q93zsrvbdw48      2874733959 21:03:43            2        2874733959 20:48:45 DIFFERENT
2qqqjzkhmsbgv      1308273333 20:28:45            0        1308273333 20:28:45
2syvqzbxp4k9z      1423211129 20:28:47            0        1423211129 20:28:45 DIFFERENT
2syvqzbxp4k9z      1423211129 20:28:47            1        1423211129 20:28:47
2tkw12w5k68vd      1457651150 21:03:43            0        1457651150 20:28:45 DIFFERENT
2tkw12w5k68vd      1457651150 21:03:43            1        1457651150 21:03:43
2xgubd6ayhyb1      3418045132 21:03:43            0        3418045132 21:03:43
2xyb5d6xg9srh       785096182 20:28:47            0         785096182 20:28:45 DIFFERENT
2xyb5d6xg9srh       785096182 20:28:47            1         785096182 20:28:47
2ysccdanw72pv      3801013801 20:28:55            0        3801013801 20:28:55
32bhha21dkv0v      3765558045 21:03:43            0        3765558045 20:28:45 DIFFERENT
32bhha21dkv0v      3765558045 21:03:43            1        3765558045 21:03:43
...
38fffx897xs0v      1042772069 21:33:45            0        1042772069 20:28:45 DIFFERENT
38fffx897xs0v      1042772069 21:33:45            1        1042772069 21:33:45
...
39m4sx9k63ba2      2317816222 21:03:43            0        2317816222 20:28:45 DIFFERENT
39m4sx9k63ba2      2317816222 21:03:43            1        2317816222 21:03:43
...
3k0c6241uw582      1964643588 20:44:48            0        1964643588 20:28:45 DIFFERENT
3k0c6241uw582      1964643588 20:44:48            1        1964643588 20:44:48
3ktacv9r56b51      4184428695 21:03:43            0        4184428695 20:28:45 DIFFERENT
3ktacv9r56b51      4184428695 21:03:43            1        4184428695 20:28:45 DIFFERENT
3ktacv9r56b51      4184428695 21:03:43            2        4184428695 21:03:43
3nhfzxjzx2btx      1043815174 20:28:45            0        1043815174 20:28:45
3nkd3g3ju5ph1      2853959010 21:03:43            0        2853959010 21:03:43
3nkd3g3ju5ph1      2853959010 21:03:43            1        2853959010 20:28:45 DIFFERENT
3nkd3g3ju5ph1      2853959010 21:03:43            2        2853959010 20:48:45 DIFFERENT
3np8cptn6uzn6      1754305749 20:28:46            0        1754305749 20:28:46
3nzv2smdzzbsf      2731876963 21:33:45            0        2731876963 20:28:45 DIFFERENT
3nzv2smdzzbsf      2731876963 21:33:45            1        2731876963 21:33:45
...
3rw49yhahg984      3808094885 20:28:55            0        3808094885 20:28:45 DIFFERENT
3rw49yhahg984      3808094885 20:28:55            1        3808094885 20:28:55
...
3w4qs0tbpmxr6      1224215794 21:03:43            0        1224215794 21:03:43
3w4qs0tbpmxr6      1224215794 21:03:43            1        1224215794 20:28:45 DIFFERENT
3w4qs0tbpmxr6      1224215794 21:03:43            2        1224215794 20:48:45 DIFFERENT
...
459f3z9u4fb3u       415205717 20:44:48            0         415205717 20:28:45 DIFFERENT
459f3z9u4fb3u       415205717 20:44:48            1         415205717 20:44:48
...
4cvqvs489pd6k      2300756036 20:28:55            0        2300756036 20:28:45 DIFFERENT
4cvqvs489pd6k      2300756036 20:28:55            1        2300756036 20:28:55
...
4zzxr8rvht74z      3368685730 20:44:48            0        3368685730 20:28:45 DIFFERENT
4zzxr8rvht74z      3368685730 20:44:48            1        3368685730 20:44:48
...
53saa2zkr6wc3      3954488388 21:03:43            0        3954488388 21:03:43
53saa2zkr6wc3      3954488388 21:03:43            1        1514015273 20:28:45 DIFFERENT
53saa2zkr6wc3      3954488388 21:03:43            2        3954488388 20:33:45 DIFFERENT
...
5n1fs4m2n2y0r       299250003 21:03:43            0         299250003 20:28:45 DIFFERENT
5n1fs4m2n2y0r       299250003 21:03:43            1         299250003 21:03:43
5n1fs4m2n2y0r       299250003 21:03:43            2         299250003 20:48:45 DIFFERENT
...
5wxyshspv54v4      3009292138 20:44:48            0        3009292138 20:28:45 DIFFERENT
5wxyshspv54v4      3009292138 20:44:48            1        3009292138 20:44:48
...
6aq34nj2zb2n7      2874733959 21:03:43            0        2874733959 21:03:43
6aq34nj2zb2n7      2874733959 21:03:43            1        2874733959 20:28:45 DIFFERENT
6aq34nj2zb2n7      2874733959 21:03:43            2        2874733959 20:48:45 DIFFERENT
6b7hj70p170j1      2605232930 20:28:45            0        2605232930 20:28:45
6c9wx6z8w9qpu       785096182 20:28:47            0         785096182 20:28:45 DIFFERENT
6c9wx6z8w9qpu       785096182 20:28:47            1         785096182 20:28:47
...
6qz82dptj0qr7      2819763574 21:03:43            0        2819763574 20:28:45 DIFFERENT
6qz82dptj0qr7      2819763574 21:03:43            1        2819763574 21:03:43
...
79w2cqu2gmjm8      4145101951 20:28:55            0        2645993454 20:28:45 DIFFERENT
79w2cqu2gmjm8      4145101951 20:28:55            1        4145101951 20:28:45 DIFFERENT
79w2cqu2gmjm8      4145101951 20:28:55            2        4145101951 20:28:55
7akvnu9t168d3      1964643588 20:44:48            0        1964643588 20:28:45 DIFFERENT
7akvnu9t168d3      1964643588 20:44:48            1        1964643588 20:44:48
...
7jpt4cpfvcy1k       284504113 20:28:46            0         284504113 20:28:45 DIFFERENT
7jpt4cpfvcy1k       284504113 20:28:46            1         284504113 20:28:46
7mgr3uwydqq8j       293268181 22:13:45            0         293268181 22:13:45
7ng34ruy5awxq      3992920156 21:03:43            0        3992920156 21:03:43
7ng34ruy5awxq      3992920156 21:03:43            1         306576078 20:28:45 DIFFERENT
7ng34ruy5awxq      3992920156 21:03:43            2        3992920156 20:48:45 DIFFERENT
7nuw4xwrnuwxq      1720483994 21:03:43            0        1720483994 20:28:45 DIFFERENT
7nuw4xwrnuwxq      1720483994 21:03:43            1        1720483994 21:03:43
...
83taa7kaw59c1      3765558045 21:03:43            0        3765558045 21:03:43
83taa7kaw59c1      3765558045 21:03:43            1        3765558045 20:28:45 DIFFERENT
83taa7kaw59c1      3765558045 21:03:43            2        3765558045 20:48:45 DIFFERENT
85sxp7kypwbx6      1395584798 20:28:46            0        1395584798 20:28:46
87gaftwrm2h68      1218588913 21:03:43            0        1218588913 20:28:45 DIFFERENT
87gaftwrm2h68      1218588913 21:03:43            1        1218588913 21:03:43
87gaftwrm2h68      1218588913 21:03:43            2        1218588913 20:48:45 DIFFERENT
...
8swypbbr0m372       893970548 21:03:43            0         893970548 20:28:45 DIFFERENT
8swypbbr0m372       893970548 21:03:43            1         893970548 20:28:45 DIFFERENT
8swypbbr0m372       893970548 21:03:43            2         893970548 21:03:43
...
9g485acn2n30m      2544153582 21:03:43            0        2544153582 20:28:45 DIFFERENT
9g485acn2n30m      2544153582 21:03:43            1        2544153582 21:03:43
9gkq7rruycsjp      3362549386 20:28:46            0        3362549386 20:28:45 DIFFERENT
9gkq7rruycsjp      3362549386 20:28:46            1        3362549386 20:28:46
...
9rfqm06xmuwu0       832500465 21:03:43            0         832500465 20:28:45 DIFFERENT
9rfqm06xmuwu0       832500465 21:03:43            1         832500465 21:03:43
9rvqpun1x1xjd      2760275752 20:28:46            0        2760275752 20:28:46
9tgj4g8y4rwy8      3755742892 21:03:43            0        3755742892 20:28:45 DIFFERENT
9tgj4g8y4rwy8      3755742892 21:03:43            1        3755742892 21:03:43
...
axmdf8vq7k1rh      2203911306 20:44:48            0        2203911306 20:28:45 DIFFERENT
axmdf8vq7k1rh      2203911306 20:44:48            1        2203911306 20:44:48
...
b1wc53ddd6h3p      1637390370 21:03:43            0        1637390370 20:28:45 DIFFERENT
b1wc53ddd6h3p      1637390370 21:03:43            1        1637390370 21:03:43
...
bsa0wjtftg3uw      1512486435 22:23:19            0        2020579421 20:28:45 DIFFERENT
bsa0wjtftg3uw      1512486435 22:23:19            1        1512486435 22:23:19
bsa0wjtftg3uw      1512486435 22:23:19            2        1512486435 20:48:45 DIFFERENT
...
c4nhd1ntptxq7      3477319146 20:28:46            0        3477319146 20:28:46
c4nhd1ntptxq7      3477319146 20:28:46            1        3477319146 20:28:46
...
cb21bacyh3c7d      3452538079 21:03:43            0        3452538079 20:28:45 DIFFERENT
cb21bacyh3c7d      3452538079 21:03:43            1        3452538079 20:28:46 DIFFERENT
cb21bacyh3c7d      3452538079 21:03:43            2        3452538079 21:03:43
...
cjk1ffy5kmm5s      1964104430 20:28:46            0        1964104430 20:28:45 DIFFERENT
cjk1ffy5kmm5s      1964104430 20:28:46            1        1964104430 20:28:46
...
cvn54b7yz0s8u      3246118364 21:03:43            0        3246118364 20:28:45 DIFFERENT
cvn54b7yz0s8u      3246118364 21:03:43            1        3246118364 21:03:43
d00a21h5ybffr       959325123 20:44:48            0        2829621105 20:28:45 DIFFERENT
d00a21h5ybffr       959325123 20:44:48            1         959325123 20:44:48
...
f3g84j69n0tjh      2335623859 21:03:43            0         914792125 20:28:45 DIFFERENT
f3g84j69n0tjh      2335623859 21:03:43            1        2335623859 21:03:43
...
fzrshwabvtwc0      3637245398 22:23:45            0        3637245398 20:28:45 DIFFERENT
fzrshwabvtwc0      3637245398 22:23:45            1        3637245398 22:23:45
...
g3wrkmxkxzhf2       749386351 21:03:43            0         749386351 20:28:45 DIFFERENT
g3wrkmxkxzhf2       749386351 21:03:43            1         749386351 21:03:43
...
g7smmy8ybh3gv        43085360 20:28:55            0          43085360 20:28:46 DIFFERENT
g7smmy8ybh3gv        43085360 20:28:55            1          43085360 20:28:55
...
ga9j9xk5cy9s0      1697022209 21:03:43            0        1697022209 20:28:45 DIFFERENT
ga9j9xk5cy9s0      1697022209 21:03:43            1        1697022209 21:03:43
...
grwydz59pu6mc      3684871272 21:03:43            0        3684871272 20:28:45 DIFFERENT
grwydz59pu6mc      3684871272 21:03:43            1        3684871272 21:03:43
...
gx4mv66pvj3xz      1932954096 21:03:43            0        1932954096 21:03:43
gx4mv66pvj3xz      1932954096 21:03:43            1        2570921597 20:28:45 DIFFERENT
gx4mv66pvj3xz      1932954096 21:03:43            2        1932954096 20:48:45 DIFFERENT
... 

I feel that I need some more suitable pictures now.  🙂

Hyper-Extended Oracle Performance Monitor 6.0 Beta

March 15, 2012 (Modified March 16, 2012)

Several people expressed an interest in using the Beta version of my Hyper-Extended Oracle Performance Monitor 6.0 that has been under development for about a decade.  Rather than trying to find a way to deliver the Beta version of the program to those people who left comments in the earlier thread, it seemed to be much easier to just post the Beta version to this blog.

The  Hyper-Extended Oracle Performance Monitor tool runs from a Windows client PC (XP with ADO 2.8+ installed, Vista, Windows 7 32/64 bit, Server 2003, Server 2008) and for some tasks, such as report generation, requires Microsoft Excel (2000, XP, 2003, 2007, or 2010) to be present on the PC.  Everything that is logged is written to a C:\OracleLog folder on the client computer, and unfortunately that likely means that User Access Control (UAC) in Vista and Windows 7 will either need to be turned down or turned off completely (UAC will prevent programs from writing in folders that are located directly in the root of the C:\ drive).  It is important to make certain that the Windows interfaces (all except MTS) are installed with the Oracle Client software, which should add the OraOLEDB functionality that is used by the program for connectivity to the databases.

An unfortunate side effect of using OraOLEDB functionality rather than ODBC is that the SYS user is not able to log in AS SYSDBA for certain tasks such as accessing the X$ structures (specifically X$BH, and the X$ structures (X$KSPPI, X$KSPPSV) needed for viewing the hidden initialization parameters). Setting the O7_DICTIONARY_ACCESSIBILITY initialization parameter to TRUE will allow the program to connect as the SYS user (without AS SYSDBA), but doing so may represent a security risk for the database.

The program writes nothing to the Oracle database that is monitored, although it might try to automatically adjust the MAX_DUMP_FILE_SIZE parameter if the user attempts to enable a 10046 trace using the program, and the program determines that the MAX_DUMP_FILE_SIZE parameter is set far too small.  The user that logs into the program will need proper permissions to access the various V$ views (GV$ views are not accessed) and also access the various packages that enable 10046/10053 traces (enabling a 10046 (or other trace) within the program’s interface requires that the user logging into the program have EXECUTE permission on the DBMS_SYSTEM and DBMS_MONITOR packages).

The 10046 trace file parser is still a bit stuck in the land of Oracle Database 8.1 – it still expects to find p1, p2, and p3 on WAIT event lines if the Table and Object Lookup option is selected for trace file parsing (and for certain wait event analysis).  Later versions of Oracle Database emit obj parameters on the WAIT lines, and the program should use the obj value rather than trying to look up the OBJECT_ID value using the p1, p2, and p3 parameters. The 10046 trace file parser performs a trick to handle the extremely long cursor numbers found in Oracle Database 11.2.0.2 and later.  The Hyper-Extended Oracle Performance Monitor is intended to work fully on Oracle Database 10.2, and (hopefully) gracefully degrade when an older version of Oracle Database is encountered.

The program supports several command line parameters, most of which are used to configure performance logging capabilities:

-D      The Database instance SID to which the program should connect.
-U      The user name to be used for connecting to the database instance.
-P      The password to be used for connecting to the database instance.

-LC 20  Specifies Force a Log Capture when CPU Usage Exceeds value to 20%
-LI 30  Specifies Force a Log Capture if No Log Captured in Minutes value to 30 minutes
-LB     Specifies the Force a Log Capture when a Blocking Lock is Detected value to checked
-LW     Specifies the Force a Log Capture when a Wait Reason is Detected value to checked
-LR     Specifies the Capture SQL Execution Statistics for Wait Reasons value to checked
-LD     Specifies the Capture Segment Change Statistics value to checked
-LO     Specifies the Capture Operating System and Time Model Statistics value to checked
-LH     Specifies the Capture High Load SQL Statement Statistics value to checked
-LT     Specifies the Capture High Load SQL Statement Text value to checked
-LP     Specifies the Capture High Load SQL Statement Plan value to checked
-LHC 60 Species the minimum CPU time that is considered high load to 60 seconds accum.
-LHE 90 Species the minimum elapsed time that is considered high load to 90 seconds accum.
-LS     Specifies that Smart Logging should begin as soon as the login completes
-LE 240 Specifies that Smart Logging should end after 240 minutes
-LQ     Specifies that the program should quit (end) when logging ends

–

Important: Keep in mind that there is an overhead, primarily server CPU utilization, associated with performance monitoring.  This overhead will be greatest when the program’s performance logging feature is utilized.  This overhead, while typically minor, might negatively impact the performance of other database sessions.  Under no circumstances should this program run directly on a Windows-based Oracle Database server’s console – doing so with performance logging enabled will significantly impact the performance of other database sessions.

This program does not phone home, nor does it collect any information that is not found in the C:\OracleLog folder on the client computer.  The C:\OracleLog folder could prove to provide additional information that is not presented directly in the program interface.  For example, when real-time performance is monitored, every 30 minutes the program will write one or more text files into the C:\OracleLog folder that show a crosstab style report of statistics and wait events (open the file with Microsoft Excel to aid readability).  The performance logging feature creates a Microsoft Access compatible database (named to correspond to the logging date and time) in the C:\OracleLog folder – various information, such as in-effect initialization parameters, are written in that Access database, even though that information is not displayed in the program’s user interface.

———————————————————————————————–
———————————————————————————————–
———————————————————————————————–

DOWNLOAD:

The documentation for the program is at least four years out of date.  You may download the program instructions for the Hyper-Extended Oracle Performance Monitor 3.0 Beta here: Hyper-ExtendedOraclePerformanceMonitor3Docs

The Beta version of the program is time limited, however it should continue functioning for the next 12 months.  You may download the program by right clicking the file and saving it as “Hyper-ExtendedOraclePerformanceMonitor6.zip” (the .zip extension must be specified): Hyper-ExtendedOraclePerformanceMonitor6.zip

The program is compressed using WinZip – Windows XP and later are able to directly open .zip files.  To install the program, simply extract the two files into the same folder; to uninstall, delete that folder and the C:\OracleLog folder.

If you find the program useful, feel free to leave a comment here.  If you find that this program is the biggest waste of a decade’s worth of free time, I would be happy to hear that too.  The program has a couple of known bugs – I know that they exist, but I do not know where they are in the program, nor do I yet know what the bugs affect.

———————————————————————————————–
———————————————————————————————–
———————————————————————————————–

Added March 16, 2012:

I thought that I would show a couple of screen captures from my program that are not necessarily performance tuning specific.

The Advanced Initialization Parameters Viewer (currently only works if the O7_DICTIONARY_ACCESSIBILITY initialization parameter is set to TRUE , but I am considering a couple of work around methods – note that the program’s description of the CURSOR_SHARING parameter does not yet mention that the SIMILAR value for the CURSOR_SHARING parameter is deprecated):

Keyword Search Viewer:

Lock/Wait Monitor:

Configure Data Change Log (showing one of the logging tables that was created by the program’s script generator):

DBMS_XPLAN and Trace:

Thoughts on a Hyper-Extended Oracle Performance Monitor Beta

March 12, 2012

As long time readers of this blog might know, in my free time during roughly the last 10 years I have been working on a program named “Hyper-Extended Oracle Performance Monitor”.  Since 2005 or 2006 I have permitted a few people to try the beta versions of the program, thinking that I might obtain a bit of feedback about what works well, and what needs a lot of work.  I was recently informed of a couple of situations where one or two features in the program were extremely useful – I would much rather hear that kind of feedback, rather than “I forgot about that program.” 🙂

What started as a simple 10046 trace file parser, easy method to execute a handful of scripts, and a V$ performance view logger has certainly grown over the years.  I have not updated the documentation for the program in almost four years, and some suggestions offered by the program seem to be Oracle Database 8.1 specific… one of these days I might have some time to address those issues.

Over the last couple of days I found a couple of unplanned features (bugs) in the program – some of those features have been in the program for a couple of years, others were added just last week.  I am currently debating whether or not to open up the beta of the program to a wider audience.  Are any readers of this blog interested?

The main screen in the program probably looks unlike any program that you have seen in the past – menus, who needs menus:

–

If you drag and drop an Oracle 10046 trace file on the picture in the main screen, you will see a daunting list of options:

–

If you have followed along with the six part series on building an Oracle Database Time Model Viewer, you might recognize this screen in my program:

–

One of the original purposes of the program was to log the various statistics found in certain V$ performance views.  Over the years I added additional information that the program is able to optionally capture, and set up the logging capabilities so that certain events will force more frequent logging of statistics: 

 

–

With logging enabled, statistics are written to an dynamically created Microsoft Access compatible database, and as the statistics are captured, a summary of the statistics is written to the main program window:

–

Once you have logged something interesting, you can go back and review the information using a variety of interfaces in the program (or just stare blankly at the Microsoft Access database that was created).  Among other things, the below screen capture shows that one session spent roughly 24 seconds of the roughly 60 second time period in the wait event enq: TX – row lock contention.

–

We can easily take a look at the system level wait events and statistics for this time period:

–

Or drill-down to the session level waits and statistics from the table at the bottom of the Review Time Model Statistics window.  There is the session and its wait event, but what caused the wait event?

–

Maybe we should investigate… there’s a button for that.  Blocker and Blocked near the bottom left of the window – I wonder if that is a clue?

–

Let’s double-click one of those rows to see what happens:

–

Nice start, but let’s ask for more information by clicking Yes.

We now have the SQL statement the blocked session was attempting to execute, and possibly the SQL statement that the blocker executed which caused the  enq: TX – row lock contention wait event (the SQL statement is actually the most recent SQL statement executed by the blocker in the time period).

–

We are also able to take a quick tour of some of the SQL statements executed in the capture period and an extended version of the execution plans for those SQL statements:

–

There are another eight year’s worth of development in the program…

Will Enabling a 10046 Trace Make My Query Slower (or Faster)?

February 13, 2012 (Modified February 14, 2012)

It seems that I often wonder about the things that I read… is it really true?  Several years ago, one of the items that I wondered about is whether or not an execution plan for a SQL statement could change magically simply because the Oracle RDBMS knows that it is being watched (a bit short-sighted, but stay with me for a couple of minutes).  When you first learn to read raw 10046 extended SQL trace files, you might also start to wonder whether or not enabling a 10046 trace could make the Oracle RDBMS suddenly start to behave differently.

Let’s take a quick look at a small portion of a 11.2.0.2 trace file:

PARSING IN CURSOR #371254768 len=53 dep=0 uid=62 oct=3 lid=62 tim=184435767875 hv=1334288504 ad='7ffb8c1be40' sqlid='a75xc757sg83s'
SELECT /*+ INDEX(T3) */
  *
FROM
  T3
WHERE
  C1<=200
END OF STMT
PARSE #371254768:c=0,e=699,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=216260220,tim=184435767874
EXEC #371254768:c=0,e=25,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=216260220,tim=184435767947
WAIT #371254768: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=0 tim=184435767976
WAIT #371254768: nam='db file scattered read' ela= 320 file#=4 block#=4145624 blocks=8 obj#=73036 tim=184435768358
WAIT #371254768: nam='db file scattered read' ela= 276 file#=4 block#=4145616 blocks=8 obj#=73035 tim=184435768704
FETCH #371254768:c=0,e=764,p=16,cr=2,cu=0,mis=0,r=1,dep=0,og=1,plh=216260220,tim=184435768757 

In the above, notice the mis=1 entry on the PARSE #371254768 line – that entry indicates a hard parse, a re-optimization of the SQL statement by the query optimizer.  If this SQL statement used bind variables, a feature known as bind variable peeking (introduced in Oracle 9.0.1) would take place during re-optimization, which might provide the optimizer with more information (with the help of a histogram, for instance) and potentially result in a change to the SQL statement’s execution plan.

(A partially related note: I just recently placed an order for the book “Oracle Database 11gR2 Performance Tuning Cookbook“, and while taking a quick peek at two or three pages in the book’s preview on Amazon, I found an error related to the above mentioned feature.)

Let’s take a look at a quick example of the above situation.  We will start by creating a table, inserting 100,000 rows where 20% of the rows have the value 0 in column C2, creating an index on column C2, and gathering statistics with histograms:

CREATE TABLE T5(
  C1 NUMBER NOT NULL,
  C2 NUMBER NOT NULL,
  C3 VARCHAR2(300) NOT NULL);

INSERT INTO
  T5
SELECT
  ROWNUM C1,
  DECODE(MOD(ROWNUM,5),0,0,ROWNUM) C2,
  RPAD('A',300,'A') C3
FROM
  DUAL
CONNECT BY
  LEVEL<=100000;

COMMIT;

CREATE INDEX IND_T5_C2 ON T5(C2);

EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>NULL,TABNAME=>'T5',CASCADE=>TRUE,ESTIMATE_PERCENT=>100,METHOD_OPT=>'FOR ALL COLUMNS SIZE 254') 

Now for the first segment of the test script, where a single row will be selected from the table.  To save space, I will not show the output from the select of the table T5, but will show the DBMS_XPLAN output:

ALTER SESSION SET TRACEFILE_IDENTIFIER = '10046TraceFindMe';

SET LINESIZE 120
SET PAGESIZE 1000
VAR V2 NUMBER
EXEC :V2:=4

SELECT
  C1
FROM
  T5
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

SQL_ID  cmm43s6usjbk1, child number 0
-------------------------------------
SELECT   C1 FROM   T5 WHERE   C2=:V2

Plan hash value: 3443248787

-----------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |       |       |     2 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T5        |     1 |    10 |     2   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | IND_T5_C2 |     1 |       |     1   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2) 

An index range scan, that is reasonable.  Let’s change the value of the bind variable so that 20% of the rows from the table will be selected, and take a look at the execution plan:

EXEC :V2:=0

SELECT
  C1
FROM
  T5
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

SQL_ID  cmm43s6usjbk1, child number 0
-------------------------------------
SELECT   C1 FROM   T5 WHERE   C2=:V2

Plan hash value: 3443248787

-----------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |       |       |     2 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T5        |     1 |    10 |     2   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | IND_T5_C2 |     1 |       |     1   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2) 

Notice in the above that the child number is still 0 and that the execution plan remained the same, even though we requested the retrieval of 20% of the rows from table T5.  In Oracle Database 11.1 and above, adaptive cursor sharing might eventually step in to change the execution plan for this bind variable value to use a full table scan, but that change could not happen on the first execution.

Now, we will enable a 10046 trace, set the bind variable value to the second value listed above, and return the execution plan:

ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT FOREVER, LEVEL 8';

EXEC :V2:=0

SELECT
  C1
FROM
  T5
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

SQL_ID  cmm43s6usjbk1, child number 1
-------------------------------------
SELECT   C1 FROM   T5 WHERE   C2=:V2

Plan hash value: 2002323537

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |   358 (100)|          |
|*  1 |  TABLE ACCESS FULL| T5   | 19685 |   192K|   358   (1)| 00:00:01 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C2"=:V2) 

Notice in the above output that while the SQL_ID has remained constant, the child number is now 1, and the execution plan changed to a full table scan.  Bind variable peeking and the histogram on column C2 provided the optimizer with enough information to result in an execution plan change.

Let’s switch back to the original bind variable value, which will result in a single row being returned:

SELECT
  C1
FROM
  T5
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

SQL_ID  cmm43s6usjbk1, child number 1
-------------------------------------
SELECT   C1 FROM   T5 WHERE   C2=:V2

Plan hash value: 2002323537

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |   358 (100)|          |
|*  1 |  TABLE ACCESS FULL| T5   | 19685 |   192K|   358   (1)| 00:00:01 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C2"=:V2) 

Note that child cursor 1 is still used – a full table scan where an index range scan was used previous to enabling a 10046 trace.  So, while we have not examined execution times, we have now seen cases where a query could potentially execute faster or slower due to a change in the execution plan after a 10046 extended SQL trace is enabled.

—

Let’s try one more time with the 10046 trace still enabled:

SELECT
  C1
FROM
  T5
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL')); 

SQL_ID  cmm43s6usjbk1, child number 2
-------------------------------------
SELECT   C1 FROM   T5 WHERE   C2=:V2

Plan hash value: 3443248787

-----------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |       |       |     2 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T5        |     1 |    10 |     2   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | IND_T5_C2 |     1 |       |     1   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2)

Now the child cursor number is displayed as 2, and the execution plan returned to an index range scan.  Adaptive cursor sharing kicked in and caused a re-optimization of the SQL statement.

Let’s disable the 10046 extended SQL trace:

ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT OFF';

A quick look at a small (marked up) portion of the raw 10046 extended SQL trace file – the first execution with the 10046 trace enabled:

PARSING IN CURSOR #429376080 len=20 dep=0 uid=62 oct=47 lid=62 tim=11461702777 hv=1269726367 ad='7ffb76adba8' sqlid='5dxda7t5uwz4z'
BEGIN :V2:=0; END;
END OF STMT
PARSE #429376080:c=0,e=56,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=0,tim=11461702777
WAIT #429376080: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=532 tim=11461702853
EXEC #429376080:c=0,e=53,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=1,plh=0,tim=11461702867
WAIT #429376080: nam='SQL*Net message from client' ela= 1401 driver id=1413697536 #bytes=1 p3=0 obj#=532 tim=11461704296
CLOSE #429376080:c=0,e=12,dep=0,type=0,tim=11461704325
=====================
PARSING IN CURSOR #429376080 len=36 dep=0 uid=62 oct=3 lid=62 tim=11461704495 hv=3045633601 ad='7ffb77363b0' sqlid='cmm43s6usjbk1'
SELECT
  C1
FROM
  T5
WHERE
  C2=:V2
END OF STMT
PARSE #429376080:c=0,e=152,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=0,tim=11461704494   <--- Hard Parse
EXEC #429376080:c=0,e=784,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=2002323537,tim=11461705325
WAIT #429376080: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=532 tim=11461705357
FETCH #429376080:c=0,e=35,p=0,cr=3,cu=0,mis=0,r=1,dep=0,og=1,plh=2002323537,tim=11461705408
...
WAIT #429376080: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=532 tim=11463556616
FETCH #429376080:c=0,e=21,p=0,cr=1,cu=0,mis=0,r=4,dep=0,og=1,plh=2002323537,tim=11463556630
STAT #429376080 id=1 cnt=20000 pid=0 pos=1 obj=73243 op='TABLE ACCESS FULL T5 (cr=5813 pr=0 pw=0 time=15413 us cost=358 size=196850 card=19685)'
... 

In the above, notice the mis=1 entry on the PARSE #429376080 line – the optimizer hard parsed the SQL statement.  Moving down a bit in the trace file, the second execution with the changed bind variable value:

PARSING IN CURSOR #429376080 len=20 dep=0 uid=62 oct=47 lid=62 tim=11505336665 hv=1039981032 ad='7ffb873a408' sqlid='857u964yztqg8'
BEGIN :V2:=4; END;
END OF STMT
PARSE #429376080:c=0,e=93,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=0,tim=11505336664
WAIT #429376080: nam='SQL*Net message to client' ela= 2 driver id=1413697536 #bytes=1 p3=0 obj#=532 tim=11505336754
EXEC #429376080:c=0,e=63,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=1,plh=0,tim=11505336769
WAIT #429376080: nam='SQL*Net message from client' ela= 2382 driver id=1413697536 #bytes=1 p3=0 obj#=532 tim=11505339191
CLOSE #429376080:c=0,e=13,dep=0,type=0,tim=11505339229
=====================
PARSING IN CURSOR #429376080 len=36 dep=0 uid=62 oct=3 lid=62 tim=11505339288 hv=3045633601 ad='7ffb77363b0' sqlid='cmm43s6usjbk1'
SELECT
  C1
FROM
  T5
WHERE
  C2=:V2
END OF STMT
PARSE #429376080:c=0,e=39,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=2002323537,tim=11505339288   <--- NO Hard Parse
EXEC #429376080:c=0,e=20,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=2002323537,tim=11505339355
WAIT #429376080: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=532 tim=11505339381
FETCH #429376080:c=0,e=296,p=0,cr=4,cu=0,mis=0,r=1,dep=0,og=1,plh=2002323537,tim=11505339693
... 

In the above, notice the mis=0 entry on the PARSE #429376080 line – the optimizer did NOT re-optimize this SQL statement, so the execution plan remained a full table scan.  Now the final section that was influenced by adaptive cursor sharing:

PARSING IN CURSOR #429376080 len=36 dep=0 uid=62 oct=3 lid=62 tim=12566538414 hv=3045633601 ad='7ffb77363b0' sqlid='cmm43s6usjbk1'
SELECT
  C1
FROM
  T5
WHERE
  C2=:V2
END OF STMT
PARSE #429376080:c=0,e=55,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=2002323537,tim=12566538414   <--- NO Hard Parse
EXEC #429376080:c=0,e=1594,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=3443248787,tim=12566540065  <--- Hard Parse
WAIT #429376080: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=532 tim=12566540198
FETCH #429376080:c=0,e=29,p=0,cr=3,cu=0,mis=0,r=1,dep=0,og=1,plh=3443248787,tim=12566540245 

In the above, notice the mis=0 entry on the PARSE #429376080 line – the optimizer did NOT re-optimize this SQL statement during the parse phase.  However, remember that the cursor number changed from the previous value of 1 to 2; the above EXEC #429376080 contains mis=1, so a hard parse was performed during the execution phase.

—

You might be thinking that maybe the execution plans can only change if we use bind variables when histograms are present.  Let’s try another experiment.  First, we will create a table, create an index on that table, insert 100,000 rows with the same characteristics as those found in table T5, and NOT collect statistics:

CREATE TABLE T6 (
  C1 NUMBER NOT NULL,
  C2 NUMBER NOT NULL,
  C3 VARCHAR2(300) NOT NULL);

CREATE INDEX IND_T6_C2 ON T6(C2);

INSERT INTO
  T6
SELECT
  ROWNUM C1,
  DECODE(MOD(ROWNUM,5),0,0,ROWNUM) C2,
  RPAD('A',300,'A') C3
FROM
  DUAL
CONNECT BY
  LEVEL<=100000;

COMMIT; 

Let’s repeat the previous test case:

SET LINESIZE 120
SET PAGESIZE 1000
VAR V2 NUMBER
EXEC :V2:=4

SELECT
  C1
FROM
  T6
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

SQL_ID  9krhnbpjpvpwh, child number 0
-------------------------------------
SELECT   C1 FROM   T6 WHERE   C2=:V2

Plan hash value: 4242288932

-----------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |       |       |     1 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T6        |     1 |    26 |     1   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | IND_T6_C2 |     1 |       |     1   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2)

Note
-----
   - dynamic sampling used for this statement (level=2) 

The above shows that dynamic sampling was used during the hard parse, and as expected an index range scan was performed.  Part 2 of the previous test script:

EXEC :V2:=0

SELECT
  C1
FROM
  T6
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

SQL_ID  9krhnbpjpvpwh, child number 0
-------------------------------------
SELECT   C1 FROM   T6 WHERE   C2=:V2

Plan hash value: 4242288932

-----------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |       |       |     1 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T6        |     1 |    26 |     1   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | IND_T6_C2 |     1 |       |     1   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2)

Note
-----
   - dynamic sampling used for this statement (level=2) 

Again, nothing new – no hard parse when the bind variable was changed, and so the index was still used.  Part 3 of the previous test script:

ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT FOREVER, LEVEL 8';

EXEC :V2:=0

SELECT
  C1
FROM
  T6
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

SQL_ID  9krhnbpjpvpwh, child number 1
-------------------------------------
SELECT   C1 FROM   T6 WHERE   C2=:V2

Plan hash value: 4242288932

-----------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |       |       |   142 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T6        | 34140 |   866K|   142   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | IND_T6_C2 | 34140 |       |   142   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2)

Note
-----
   - dynamic sampling used for this statement (level=2) 

There was a hard parse, as is noted by the change in the child cursor number, yet the optimizer still decided that it was most efficient to perform an index range scan.  The optimizer estimated that 34,140 rows would be returned with an estimated size of 866KB.  You might recall that earlier the same test for table T5 showed the following execution plan:

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |   358 (100)|          |
|*  1 |  TABLE ACCESS FULL| T5   | 19685 |   192K|   358   (1)| 00:00:01 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C2"=:V2) 

With a histogram (and without dynamic sampling), the optimizer estimated that 19,685 rows would be returned with an estimated size of 192KB – and in that case a full table scan was performed.  But why was a full table scan not performed when experimenting with dynamic sampling?  Is it possible that if the query optimizer uses dynamic sampling, that the execution plan is NOT subject to change when a 10046 extended SQL trace is enabled, or is there a simplier cause?

Let’s try executing the query with a FULL hint:

SELECT /*+ FULL(T6) */
  C1
FROM
  T6
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

SQL_ID  bhm179jq4vvca, child number 0
-------------------------------------
SELECT /*+ FULL(T6) */   C1 FROM   T6 WHERE   C2=:V2

Plan hash value: 1930642322

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |       |       |   357 (100)|          |
|*  1 |  TABLE ACCESS FULL| T6   | 34140 |   866K|   357   (1)| 00:00:01 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C2"=:V2)

Note
-----
   - dynamic sampling used for this statement (level=2) 

Well, that explains part of the issue.  While the calculated cost of the full table scan access path is 357 (very close to the 358 calculated for the same SQL statement that accessed table T5), the calculated cost of the index access path was only 142 – the lowest calculated cost access path wins.

What if we force an index access path for table T5 with the same value of 0 for the bind variable.  Will the calculated cost also be roughly 142?

SELECT /*+ INDEX(T5) */
  C1
FROM
  T5
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'TYPICAL'));

SQL_ID  5y34jm13up8xh, child number 0
-------------------------------------
SELECT /*+ INDEX(T5) */   C1 FROM   T5 WHERE   C2=:V2

Plan hash value: 3443248787

-----------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |       |       |  1826 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T5        | 19685 |   192K|  1826   (1)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | IND_T5_C2 | 19685 |       |    43   (0)| 00:00:01 |
-----------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2)  

Obviously, no – otherwise the index access path would have been selected for table T5 also – as shown above, the cost is actually 1826.

—-

The challenge (drifting a bit from the topic of this article): If dynamic sampling is used, and the optimizer knows that a many (possibly 34%) of the rows will be returned when the query is executed, why is an index access path used?  In other words, what causes the index access path to be calculated so cheaply when compared to the calculated cost for the same access path with table T5?  Is the above simply a bug found in Oracle Database 11.2.0.2, or is there a better explanation?

–

Added February 14, 2012:

One person (Narendra) noticed that there appeared to be a bug in the optimizer’s calculations.  If you closely examine the execution plans above, you just might notice that the Cost displayed on the INDEX RANGE SCAN operation’s line of the execution plan (142) is identical to the Cost displayed on its the parent operation TABLE ACCESS BY INDEX ROWID (142).  As is probably well known, a parent operation’s Cost, Buffers, Reads, and various other columns display their child statistics summed with the statistics that are directly associated with the parent operation.  Take another look at a portion of one of the earlier execution plans:

-----------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |       |       |   142 (100)|          |
|   1 |  TABLE ACCESS BY INDEX ROWID| T6        | 34140 |   866K|   142   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN          | IND_T6_C2 | 34140 |       |   142   (0)| 00:00:01 |
----------------------------------------------------------------------------------------- 

The cost displayed for the TABLE ACCESS BY INDEX ROWID operation should be greater than the cost displayed for the index access – unless the optimizer forgot that the table must be accessed when dynamic sampling is performed.  Oracle Database 11.2.0.1 and 11.2.0.2 appear to be affected by this particular bug, while Oracle Database 10.2.0.5 and 11.1.0.7 seem to under-estimate the cost of the table access when dynamic sampling is performed.  As such, if statistics on some of the tables (for instance temp tables) are locked without any statistics set on the tables (to force dynamic sampling), there could be a minor issue that becomes a more significant issue when upgrading to 11.2.0.1 or 11.2.0.2.

Recreating the same test table as earlier:

DROP TABLE T6 PURGE;

CREATE TABLE T6 (
  C1 NUMBER NOT NULL,
  C2 NUMBER NOT NULL,
  C3 VARCHAR2(300) NOT NULL);

CREATE INDEX IND_T6_C2 ON T6(C2);

INSERT INTO
  T6
SELECT
  ROWNUM C1,
  DECODE(MOD(ROWNUM,5),0,0,ROWNUM) C2,
  RPAD('A',300,'A') C3
FROM
  DUAL
CONNECT BY
  LEVEL<=100000;

COMMIT; 

The first portion of the test:

ALTER SESSION SET TRACEFILE_IDENTIFIER = '10053NO10046TraceFindMe';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';

SET LINESIZE 120
SET PAGESIZE 1000
VAR V2 NUMBER
EXEC :V2:=4

SELECT /*+ GATHER_PLAN_STATISTICS */
  C1
FROM
  T6
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +PEEKED_BINDS +COST'));

SQL_ID  bxnurhc1nmq2g, child number 0
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS */   C1 FROM   T6 WHERE   C2=:V2

Plan hash value: 4242288932

----------------------------------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers |
----------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |      1 |        |     1 (100)|      1 |00:00:00.01 |       4 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T6        |      1 |      1 |     1   (0)|      1 |00:00:00.01 |       4 |
|*  2 |   INDEX RANGE SCAN          | IND_T6_C2 |      1 |      1 |     1   (0)|      1 |00:00:00.01 |       3 |
----------------------------------------------------------------------------------------------------------------

Peeked Binds (identified by position):
--------------------------------------
   1 - (NUMBER): 4

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2)

Note
-----
   - dynamic sampling used for this statement (level=2)

The second portion of the test:

EXEC :V2:=0

SELECT /*+ GATHER_PLAN_STATISTICS */
  C1
FROM
  T6
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +PEEKED_BINDS +COST'));

SQL_ID  bxnurhc1nmq2g, child number 0
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS */   C1 FROM   T6 WHERE   C2=:V2

Plan hash value: 4242288932

----------------------------------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers |
----------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |      1 |        |     1 (100)|  20000 |00:00:00.06 |    6981 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T6        |      1 |      1 |     1   (0)|  20000 |00:00:00.06 |    6981 |
|*  2 |   INDEX RANGE SCAN          | IND_T6_C2 |      1 |      1 |     1   (0)|  20000 |00:00:00.03 |    1410 |
----------------------------------------------------------------------------------------------------------------

Peeked Binds (identified by position):
--------------------------------------
   1 - (NUMBER): 4

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2)

Note
-----
   - dynamic sampling used for this statement (level=2) 

Notice in the above that the execution plan has not changed, and the peeked bind variable value is still 4 (this is expected).

The third portion of the test:

ALTER SESSION SET TRACEFILE_IDENTIFIER = '1005310046TraceFindMe';
ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT FOREVER, LEVEL 8';

EXEC :V2:=0

SELECT /*+ GATHER_PLAN_STATISTICS */
  C1
FROM
  T6
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +PEEKED_BINDS +COST'));

SQL_ID  bxnurhc1nmq2g, child number 1
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS */   C1 FROM   T6 WHERE   C2=:V2

Plan hash value: 4242288932

----------------------------------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers |
----------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |      1 |        |    49 (100)|  20000 |00:00:00.06 |    6944 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T6        |      1 |  18399 |    49   (0)|  20000 |00:00:00.06 |    6944 |
|*  2 |   INDEX RANGE SCAN          | IND_T6_C2 |      1 |  18399 |    49   (0)|  20000 |00:00:00.03 |    1373 |
----------------------------------------------------------------------------------------------------------------

Peeked Binds (identified by position):
--------------------------------------
   1 - (NUMBER): 0

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2)

Note
-----
   - dynamic sampling used for this statement (level=2) 

Notice in the above that the Cost displayed for the INDEX RANGE SCAN operation is identical to the Cost displayed for the TABLE ACCESS BY INDEX ROWID – also notice the low estimated Cost for the index access (this might be a reasonable value, as will be shown below).

The fourth portion of the test, checking the cost of the full table scan access path:

SELECT /*+ FULL(T6) GATHER_PLAN_STATISTICS */
  C1
FROM
  T6
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +PEEKED_BINDS +COST'));

Plan hash value: 1930642322

-------------------------------------------------------------------------------------------------
| Id  | Operation         | Name | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers |
-------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |      1 |        |  1579 (100)|  20000 |00:00:00.05 |    5815 |
|*  1 |  TABLE ACCESS FULL| T6   |      1 |  18399 |  1579   (5)|  20000 |00:00:00.05 |    5815 |
-------------------------------------------------------------------------------------------------

Peeked Binds (identified by position):
--------------------------------------
   1 - (NUMBER): 0

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C2"=:V2)

Note
-----
   - dynamic sampling used for this statement (level=2)

So, the calculated cost of the full table scan access path is quite high compared to the cost of the index access path.

Let’s try an undocumented OPT_ESTIMATE hint to see what happens:

SELECT /*+ OPT_ESTIMATE(TABLE, T6, ROWS=20000) GATHER_PLAN_STATISTICS */
  C1
FROM
  T6
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +PEEKED_BINDS +COST'));

Plan hash value: 4242288932

----------------------------------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers |
----------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |      1 |        |     1 (100)|  20000 |00:00:00.06 |    6944 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T6        |      1 |  20000 |     1   (0)|  20000 |00:00:00.06 |    6944 |
|*  2 |   INDEX RANGE SCAN          | IND_T6_C2 |      1 |      1 |     1   (0)|  20000 |00:00:00.02 |    1373 |
----------------------------------------------------------------------------------------------------------------

Peeked Binds (identified by position):
--------------------------------------
   1 - (NUMBER): 0

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2)

Now the calculated Cost of the index access path is just 1 – bad idea for a solution.

Let’s collect statistics on the table with a histogram on column C2 to see what happens:

EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>NULL,TABNAME=>'T6',CASCADE=>TRUE,ESTIMATE_PERCENT=>100,METHOD_OPT=>'FOR COLUMNS C2 SIZE 254',NO_INVALIDATE=>FALSE)

SELECT /*+ GATHER_PLAN_STATISTICS */
  C1
FROM
  T6
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +PEEKED_BINDS +COST'));

SQL_ID  bxnurhc1nmq2g, child number 0
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS */   C1 FROM   T6 WHERE   C2=:V2

Plan hash value: 1930642322

-------------------------------------------------------------------------------------------------
| Id  | Operation         | Name | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers |
-------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |      1 |        |  1582 (100)|  20000 |00:00:00.05 |    5814 |
|*  1 |  TABLE ACCESS FULL| T6   |      1 |  19685 |  1582   (5)|  20000 |00:00:00.05 |    5814 |
-------------------------------------------------------------------------------------------------

Peeked Binds (identified by position):
--------------------------------------
   1 - (NUMBER): 0

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C2"=:V2)

The optimizer now selected to use a full table scan, rather than an index access path.

Let’s check the cost of the index access path:

SELECT /*+ INDEX(T6) GATHER_PLAN_STATISTICS */
  C1
FROM
  T6
WHERE
  C2=:V2;

SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +PEEKED_BINDS +COST'));

SQL_ID  4nqwgus0ghxmc, child number 0
-------------------------------------
SELECT /*+ INDEX(T6) GATHER_PLAN_STATISTICS */   C1 FROM   T6 WHERE
C2=:V2

Plan hash value: 4242288932

----------------------------------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers |
----------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |      1 |        |  1847 (100)|  20000 |00:00:00.07 |    6944 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T6        |      1 |  19685 |  1847   (2)|  20000 |00:00:00.07 |    6944 |
|*  2 |   INDEX RANGE SCAN          | IND_T6_C2 |      1 |  19685 |    44  (12)|  20000 |00:00:00.02 |    1373 |
----------------------------------------------------------------------------------------------------------------

Peeked Binds (identified by position):
--------------------------------------
   1 - (NUMBER): 0

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2)

–

The results of the test from 10.2.0.5:

The first portion of the test:

ALTER SESSION SET TRACEFILE_IDENTIFIER = '10053NO10046TraceFindMe';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';
 
SET LINESIZE 120
SET PAGESIZE 1000
VAR V2 NUMBER
EXEC :V2:=4
 
SELECT /*+ GATHER_PLAN_STATISTICS */
  C1
FROM
  T6
WHERE
  C2=:V2;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +PEEKED_BINDS +COST'));
 
SQL_ID  bxnurhc1nmq2g, child number 0
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS */   C1 FROM   T6 WHERE   C2=:V2
 
Plan hash value: 4242288932
 
----------------------------------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers |
----------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |      1 |        |     2 (100)|      1 |00:00:00.01 |       4 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T6        |      1 |      1 |     2   (0)|      1 |00:00:00.01 |       4 |
|*  2 |   INDEX RANGE SCAN          | IND_T6_C2 |      1 |      1 |     1   (0)|      1 |00:00:00.01 |       3 |
----------------------------------------------------------------------------------------------------------------
 
Peeked Binds (identified by position):
--------------------------------------
   1 - (NUMBER): 4
 
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2)
 
Note
-----
   - dynamic sampling used for this statement

Notice in the above output that the Cost displayed for the TABLE ACCESS BY INDEX ROWID operation is 2, while on 11.2.0.2 it was 1.

The second portion of the test script on 10.2.0.5:

EXEC :V2:=0
  
SELECT /*+ GATHER_PLAN_STATISTICS */
  C1
FROM
  T6
WHERE
  C2=:V2;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +PEEKED_BINDS +COST'));
 
SQL_ID  bxnurhc1nmq2g, child number 0
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS */   C1 FROM   T6 WHERE   C2=:V2
 
Plan hash value: 4242288932
 
----------------------------------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers |
----------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |      1 |        |     2 (100)|  20000 |00:00:00.04 |    6969 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T6        |      1 |      1 |     2   (0)|  20000 |00:00:00.04 |    6969 |
|*  2 |   INDEX RANGE SCAN          | IND_T6_C2 |      1 |      1 |     1   (0)|  20000 |00:00:00.01 |    1407 |
----------------------------------------------------------------------------------------------------------------
 
Peeked Binds (identified by position):
--------------------------------------
   1 - (NUMBER): 4
 
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2)
 
Note
-----
   - dynamic sampling used for this statement

No change in the execution plan, as expected.

The third portion of the script on 10.2.0.5:

ALTER SESSION SET TRACEFILE_IDENTIFIER = '1005310046TraceFindMe';
ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT FOREVER, LEVEL 8';
  
EXEC :V2:=0
 
SELECT /*+ GATHER_PLAN_STATISTICS */
  C1
FROM
  T6
WHERE
  C2=:V2;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +PEEKED_BINDS +COST'));
 
SQL_ID  bxnurhc1nmq2g, child number 1
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS */   C1 FROM   T6 WHERE   C2=:V2
 
Plan hash value: 4242288932
 
----------------------------------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers |
----------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |      1 |        |   209 (100)|  20000 |00:00:00.02 |    6932 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T6        |      1 |  20556 |   209   (0)|  20000 |00:00:00.02 |    6932 |
|*  2 |   INDEX RANGE SCAN          | IND_T6_C2 |      1 |  20556 |    49   (0)|  20000 |00:00:00.01 |    1370 |
----------------------------------------------------------------------------------------------------------------
 
Peeked Binds (identified by position):
--------------------------------------
   1 - (NUMBER): 0
 
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2)
 
Note
-----
   - dynamic sampling used for this statement

Notice in the output above the that while the cost of the TABLE ACCESS BY INDEX ROWID operation is significantly higher than the INDEX RANGE SCAN child operation, the Cost was not high enough to result in a full table scan.

Checking the calculated cost of a full table scan:

SELECT /*+ FULL(T6) GATHER_PLAN_STATISTICS */
  C1
FROM
  T6
WHERE
  C2=:V2;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +PEEKED_BINDS +COST'));
 
SQL_ID  bk8rwkf0ab8h5, child number 0
-------------------------------------
SELECT /*+ FULL(T6) GATHER_PLAN_STATISTICS */   C1 FROM   T6 WHERE   C2=:V2
 
Plan hash value: 1930642322
 
-------------------------------------------------------------------------------------------------
| Id  | Operation         | Name | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers |
-------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |      1 |        |   358 (100)|  20000 |00:00:00.02 |    5815 |
|*  1 |  TABLE ACCESS FULL| T6   |      1 |  20556 |   358   (1)|  20000 |00:00:00.02 |    5815 |
-------------------------------------------------------------------------------------------------
 
Peeked Binds (identified by position):
--------------------------------------
   1 - (NUMBER): 0
 
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C2"=:V2)
 
Note
-----
   - dynamic sampling used for this statement 

358 is greater than 209, so no full table scan without a hint.

OPT_ESTIMATE hint test:

SELECT /*+ OPT_ESTIMATE(TABLE, T6, ROWS=20000) GATHER_PLAN_STATISTICS */
  C1
FROM
  T6
WHERE
  C2=:V2;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +PEEKED_BINDS +COST')); 
 
SQL_ID  bud5avdhnmssf, child number 0
-------------------------------------
SELECT /*+ OPT_ESTIMATE(TABLE, T6, ROWS=20000) GATHER_PLAN_STATISTICS */   C1 FROM   T6 WHERE   C2=:V2
 
Plan hash value: 4242288932
 
----------------------------------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers |
----------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |      1 |        |     1 (100)|  20000 |00:00:00.04 |    6932 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T6        |      1 |  20000 |     1   (0)|  20000 |00:00:00.04 |    6932 |
|*  2 |   INDEX RANGE SCAN          | IND_T6_C2 |      1 |      1 |     1   (0)|  20000 |00:00:00.01 |    1370 |
----------------------------------------------------------------------------------------------------------------
 
Peeked Binds (identified by position):
--------------------------------------
   1 - (NUMBER): 0
 
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2)

Gather statistics on 10.2.0.5 with a histogram on column C2, then repeat the test:

EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>NULL,TABNAME=>'T6',CASCADE=>TRUE,ESTIMATE_PERCENT=>100,METHOD_OPT=>'FOR COLUMNS C2 SIZE 254',NO_INVALIDATE=>FALSE) 
 
SELECT /*+ GATHER_PLAN_STATISTICS */
  C1
FROM
  T6
WHERE
  C2=:V2;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +PEEKED_BINDS +COST'));
 
SQL_ID  bxnurhc1nmq2g, child number 0
-------------------------------------
SELECT /*+ GATHER_PLAN_STATISTICS */   C1 FROM   T6 WHERE   C2=:V2
 
Plan hash value: 1930642322
 
-------------------------------------------------------------------------------------------------
| Id  | Operation         | Name | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers |
-------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |      1 |        |   358 (100)|  20000 |00:00:00.02 |    5815 |
|*  1 |  TABLE ACCESS FULL| T6   |      1 |  19685 |   358   (1)|  20000 |00:00:00.02 |    5815 |
-------------------------------------------------------------------------------------------------
 
Peeked Binds (identified by position):
--------------------------------------
   1 - (NUMBER): 0
 
Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("C2"=:V2)

The optimizer determined that a full table scan would be most efficient.

Let’s check the cost of the index access path on 10.2.0.5:

SELECT /*+ INDEX(T6) GATHER_PLAN_STATISTICS */
  C1
FROM
  T6
WHERE
  C2=:V2;
 
SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY_CURSOR(NULL,NULL,'ALLSTATS LAST +PEEKED_BINDS +COST'));
 
SQL_ID  4nqwgus0ghxmc, child number 0
-------------------------------------
SELECT /*+ INDEX(T6) GATHER_PLAN_STATISTICS */   C1 FROM   T6 WHERE   C2=:V2
 
Plan hash value: 4242288932
 
----------------------------------------------------------------------------------------------------------------
| Id  | Operation                   | Name      | Starts | E-Rows | Cost (%CPU)| A-Rows |   A-Time   | Buffers |
----------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |           |      1 |        |  1822 (100)|  20000 |00:00:00.02 |    6932 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T6        |      1 |  19685 |  1822   (1)|  20000 |00:00:00.02 |    6932 |
|*  2 |   INDEX RANGE SCAN          | IND_T6_C2 |      1 |  19685 |    39   (0)|  20000 |00:00:00.01 |    1370 |
----------------------------------------------------------------------------------------------------------------
 
Peeked Binds (identified by position):
--------------------------------------
   1 - (NUMBER): 0
 
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C2"=:V2)

The Cost of 1822 is greater than the Cost of 358 for the full table scan access path, so that explains why the full table scan access path was selected.

Parent-Child Relationships and the Questions Left Unanswered by TKPROF, Re-Learning Something Old

January 30, 2012

As we have seen in the past, TKPROF output sometimes lies, and in a recent OTN thread I was reminded of another case or two where TKPROF output may be misleading.  In the OTN thread, the original poster (OP) started the thread by asking a simple question about an execution plan that appeared in a TKPROF output.  Below is an execution plan that is similar to the one posted by the OP:

Rows (1st) Rows (avg) Rows (max)  Row Source Operation
---------- ---------- ----------  ---------------------------------------------------
         1          1          1  TABLE ACCESS BY INDEX ROWID T4 (cr=3 pr=16 pw=0 time=1274 us cost=2 size=97 card=1)
         1          1          1   INDEX RANGE SCAN IND_T4_OBJECT_ID (cr=2 pr=8 pw=0 time=620 us cost=1 size=0 card=1)(object id 102460) 

For the above execution plan, the trace file was created on Oracle Database 11.2.0.2 and processed with a 11.2.0.1 client with the patch that fixes the ODBC problem.  The original poster (OP) in the OTN thread posed a question about the cr= (consistent gets blocks) and pr= (physical read blocks) statistics.  In the above, the INDEX RANGE SCAN operation shows cr=2 pr=8, and the TABLE ACCESS BY INDEX ROWID operation shows cr=3 pr=16.  The OP questioned whether the TABLE ACCESS BY INDEX ROWID operation statistics included the INDEX RANGE SCAN operation statistics for a total of 3 consistent gets and 16 physical blocks read from disk, or if the TABLE ACCESS BY INDEX ROWID operation statistics are in addition to the INDEX RANGE SCAN operation statistics for a total of 5 consistent gets and 24 physical blocks read from disk.  As sometimes happens with Internet forums, people responding to questions sometimes provide conflicting answers; in this case, at least one person stated that each interpretation is correct.

The OP’s actual Row Source Operation execution plan showed the following:

Rows     Row Source Operation
-------  ---------------------------------------------------
      1  TABLE ACCESS BY INDEX ROWID T (cr=5 pr=11 pw=0 time=0 us cost=4 size=8092 card=1)
      1   INDEX UNIQUE SCAN T_PK (cr=3 pr=6 pw=0 time=0 us cost=2 size=0 card=1)(object id 83084) 

In an attempt to provide the OP with answers to all of his unanswered questions to that point in the thread, I provided the following response:

In the above, the “TABLE ACCESS BY INDEX ROWID” operation is the parent operation and the “INDEX UNIQUE SCAN” is the child operation. The child’s statistics are rolled into the parent’s statistics. In Oracle Database 11.1 and above, the default behavior is to output the STAT lines (that is the Row Source Operation execution plan) after the first execution of the SQL statement. For the first execution of the SQL statement, there were 11 physical block reads, 6 from the index and 5 from the table. There were 5 consistent gets, 3 from the index and 2 from the table. The estimated cost for the index access is 2, and the estimated cost for just the table access is 2.
 
I see that you also used the EXPLAIN option of TKPROF – that can lead to misleading execution plans being written to the TKPROF output, especially when bind variables are involved. The 1 parse call should be expected – it would be bad if there were 72,181 parses, because that would mean that there would be 1 parse per execution – a sign that the cursor was not held open either by the application or by the session cursor cache. Simply enabling a 10046 trace in a session changes the environment for the session, which will typically require a hard parse of previously parsed SQL statements when those SQL statements are re-executed. Note that “recursive depth: 1” appears in the output, which suggests that this SQL statement is found in a PL/SQL block.
 
db file scattered read waits indicate that more than 1 block is read from disk at a time. You can refer to the raw 10046 trace file to determine how many blocks are read on average in the db file scattered read waits.

OK, so if you believe the above quote, the child statistics are rolled into the parent’s statistics, the EXPLAIN option of TKPROF may write incorrect execution plans to the output file especially when bind variables are used, and the original trace file included SQL statements marked with dep=0 or dep=1.  In the thread there was disagreement on the first point, so let’s throw together a very short test case to determine if that point is correct.

First, we will create a simple table with a primary key index, insert  200 rows into the table, and collect statistics on that table:

CREATE TABLE T3 (
  C1 NUMBER,
  C2 VARCHAR2(300),
  PRIMARY KEY(C1));

INSERT INTO
  T3
SELECT
  ROWNUM C1,
  RPAD('A',300,'A')
FROM
  DUAL
CONNECT BY
  LEVEL<=200;

COMMIT;

EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T3',CASCADE=>TRUE,NO_INVALIDATE=>FALSE) 

Next, I want to make certain that the two SQL statements that will be used in the test script will already exist in the library cache (so that we can test whether or not enabling a 10046 trace will cause a hard parse when the SQL statements are re-executed).  I also will set the fetch array size to 1000 to minimize the number of round trips between the server and client (the default fetch array size is 15 for SQL*Plus, and that value would make the generated trace file a bit more difficult to read due to the extra FETCH and SQL*Net wait events).  The second SQL statement, intended to make certain that the STAT lines for the first SQL statement will be written to the 10046 trace file, is only necessary prior to Oracle Database 11.1.0.6:

SELECT /*+ INDEX(T3) */
  *
FROM
  T3
WHERE
  C1<=200;

SELECT
  SYSDATE
FROM
  DUAL;

SET ARRAYSIZE 1000

Now for the final section of the test case script: flush the buffer cache, enable a 10046 trace, and execute the two SQL statements:

ALTER SYSTEM FLUSH BUFFER_CACHE;
ALTER SYSTEM FLUSH BUFFER_CACHE;

ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT FOREVER, LEVEL 8';
ALTER SESSION SET TRACEFILE_IDENTIFIER = '10046TESTING';

SELECT /*+ INDEX(T3) */
  *
FROM
  T3
WHERE
  C1<=200;

SELECT
  SYSDATE
FROM
  DUAL;

ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT OFF';  

Let’s start by taking a look at the raw 10046 trace file, slightly marked up:

=====================
PARSING IN CURSOR #371254768 len=53 dep=0 uid=62 oct=3 lid=62 tim=184435767875 hv=1334288504 ad='7ffb8c1be40' sqlid='a75xc757sg83s'
SELECT /*+ INDEX(T3) */
  *
FROM
  T3
WHERE
  C1<=200
END OF STMT
PARSE #371254768:c=0,e=699,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=216260220,tim=184435767874                                <--- hard parse
EXEC #371254768:c=0,e=25,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=216260220,tim=184435767947
WAIT #371254768: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=0 tim=184435767976
WAIT #371254768: nam='db file scattered read' ela= 320 file#=4 block#=4145624 blocks=8 obj#=73036 tim=184435768358          <--- 8 for index
WAIT #371254768: nam='db file scattered read' ela= 276 file#=4 block#=4145616 blocks=8 obj#=73035 tim=184435768704          <--- 8 for table
FETCH #371254768:c=0,e=764,p=16,cr=2,cu=0,mis=0,r=1,dep=0,og=1,plh=216260220,tim=184435768757
WAIT #371254768: nam='SQL*Net message from client' ela= 110 driver id=1413697536 #bytes=1 p3=0 obj#=73035 tim=184435768890
WAIT #371254768: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=73035 tim=184435768916
WAIT #371254768: nam='db file scattered read' ela= 283 file#=4 block#=4145632 blocks=8 obj#=73035 tim=184435769273          <--- 8 for table
FETCH #371254768:c=0,e=443,p=8,cr=10,cu=0,mis=0,r=199,dep=0,og=1,plh=216260220,tim=184435769349
STAT #371254768 id=1 cnt=200 pid=0 pos=1 obj=73035 op='TABLE ACCESS BY INDEX ROWID T3 (cr=12 pr=24 pw=0 time=762 us cost=10 size=60800 card=200)'
STAT #371254768 id=2 cnt=200 pid=1 pos=1 obj=73036 op='INDEX RANGE SCAN SYS_C008672 (cr=2 pr=8 pw=0 time=1012 us cost=1 size=0 card=200)'

*** 2012-01-25 09:17:32.639
WAIT #371254768: nam='SQL*Net message from client' ela= 1142603 driver id=1413697536 #bytes=1 p3=0 obj#=73035 tim=184436912005
CLOSE #371254768:c=0,e=19,dep=0,type=0,tim=184436912120
===================== 

The above shows a hard parse, 8 blocks read by multi-block read for the index, and 16 blocks read by multi-block read for the table.  If we process the trace file with TKPROF, a portion of the TKPROF output should look like this:

********************************************************************************
SELECT /*+ INDEX(T3) */
  *
FROM
  T3
WHERE
  C1<=200

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute      2      0.00       0.00          0          0          0           0
Fetch        2      0.00       0.00         24         12          0         200
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        5      0.00       0.00         24         12          0         200

Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 62 
Number of plan statistics captured: 1

Rows (1st) Rows (avg) Rows (max)  Row Source Operation
---------- ---------- ----------  ---------------------------------------------------
       200        200        200  TABLE ACCESS BY INDEX ROWID T3 (cr=12 pr=24 pw=0 time=762 us cost=10 size=60800 card=200)
       200        200        200   INDEX RANGE SCAN SYS_C008672 (cr=2 pr=8 pw=0 time=1012 us cost=1 size=0 card=200)(object id 73036)

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  SQL*Net message to client                       3        0.00          0.00
  SQL*Net message from client                     3        1.14          1.14
  db file scattered read                          3        0.00          0.00
******************************************************************************** 

By examining the raw trace file, we are able to confirm that the pr=24 statistic in the first row of the Row Source Execution plan includes the 16 blocks for the table and the 8 blocks from the index, as identified in the raw 10046 trace file.

—

The OP later asked a handful of questions, including whether or not he should add the statistics found in the OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS section of the TKPROF output to the statistics found in the OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS section.  My first thought was to modify my original test case script shown above like this, so that there is a logging trigger on the T3 table that records changes made to the T3 table in the T4 table:

DROP TABLE T3 PURGE;
DROP TABLE T4 PURGE;

CREATE TABLE T3 (
  C1 NUMBER,
  C2 VARCHAR2(300),
  PRIMARY KEY(C1));

CREATE TABLE T4(
  NEW_C1 NUMBER,
  OLD_C2 VARCHAR2(300),
  NEW_C2 VARCHAR2(300),
  LOG_MODIFIED_USER_ID VARCHAR2(30) DEFAULT USER,
  LOG_MODIFIED_DATE DATE DEFAULT SYSDATE,
  LOG_TRANSACTION_TYPE VARCHAR2(1))
  PCTFREE 0
  PCTUSED 99;

CREATE OR REPLACE TRIGGER HPM_T3 AFTER
INSERT OR DELETE OR UPDATE OF C1, C2 ON T3
REFERENCING OLD AS OLDDATA NEW AS NEWDATA FOR EACH ROW
BEGIN
  IF UPDATING THEN
    IF (NVL(:OLDDATA.C1,0.0000019) <> NVL(:NEWDATA.C1,0.0000019)) OR (NVL(:OLDDATA.C2,'3!3D$FF') <> NVL(:NEWDATA.C2,'3!3D$FF'))  THEN
     INSERT INTO T4 (
      NEW_C1,
      OLD_C2,
      NEW_C2,
      LOG_TRANSACTION_TYPE)
     VALUES (
      :NEWDATA.C1,
      :OLDDATA.C2,
      :NEWDATA.C2,
      'U');
    END IF;
  END IF;

  IF DELETING THEN
    INSERT INTO T4 (
      NEW_C1,
      OLD_C2,
      LOG_TRANSACTION_TYPE)
    VALUES (
      :OLDDATA.C1,
      :OLDDATA.C2,
      'D');
  END IF;

  IF INSERTING THEN
    INSERT INTO T4 (
      NEW_C1,
      NEW_C2,
      LOG_TRANSACTION_TYPE)
    VALUES (
      :NEWDATA.C1,
      :NEWDATA.C2,
      'I');
  END IF;
END;
/

INSERT INTO
  T3
SELECT
  ROWNUM C1,
  RPAD('A',300,'A')
FROM
  DUAL
CONNECT BY
  LEVEL<=200;

COMMIT;

EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T3',CASCADE=>TRUE,NO_INVALIDATE=>FALSE)

SELECT /*+ INDEX(T3) */
  *
FROM
  T3
WHERE
  C1<=200;

SELECT
  SYSDATE
FROM
  DUAL;

SET ARRAYSIZE 1000 

Then I would add a new section to the second half of the original test script to update 50 rows of table T3:

ALTER SYSTEM FLUSH BUFFER_CACHE;
ALTER SYSTEM FLUSH BUFFER_CACHE;

ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT FOREVER, LEVEL 8';
ALTER SESSION SET TRACEFILE_IDENTIFIER = '10046TESTING2';

SELECT /*+ INDEX(T3) */
  *
FROM
  T3
WHERE
  C1<=200;

SELECT
  SYSDATE
FROM
  DUAL;

ALTER SESSION SET TRACEFILE_IDENTIFIER = '10046TESTING3';

UPDATE
  T3
SET
  C1=C1+1000,
  C2='UPDATED BY 1000'
WHERE
  C1<=50;

ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT OFF'; 

–

I decided to abandon that approach, and instead try to reproduce the test script that the OP was attempting to use.  My best guess is that the OP’s test script looks similar to the following:

CREATE TABLE T4 AS
SELECT
  *
FROM
  ALL_OBJECTS;

ALTER TABLE T4 ADD (
  PADDING1 VARCHAR2(4000),
  PADDING2 VARCHAR2(4000));

CREATE INDEX IND_T4_OBJECT_ID ON T4(OBJECT_ID);

EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>USER,TABNAME=>'T4',CASCADE=>TRUE,NO_INVALIDATE=>FALSE)

ALTER SYSTEM FLUSH BUFFER_CACHE;
ALTER SYSTEM FLUSH BUFFER_CACHE;

ALTER SESSION SET TRACEFILE_IDENTIFIER = '10046_FIND_ME';
EXEC DBMS_MONITOR.SESSION_TRACE_ENABLE ( waits=>true );

DECLARE
        type array is table of t4%ROWTYPE index by binary_integer;
        l_data array;
        l_rec t4%rowtype;
BEGIN
        SELECT
                a.*
                ,RPAD('*',4000,'*') AS PADDING1
                ,RPAD('*',4000,'*') AS PADDING2
        BULK COLLECT INTO
        l_data
        FROM ALL_OBJECTS a;

        FOR rs IN 1 .. l_data.count
        LOOP
                BEGIN
                        SELECT * INTO l_rec FROM t4 WHERE object_id = l_data(rs).object_id;
                EXCEPTION
                  WHEN NO_DATA_FOUND THEN NULL;
                END;
        END LOOP;
END;
/

EXEC DBMS_MONITOR.SESSION_TRACE_DISABLE; 

In the script I set the NO_INVALIDATE parameter of the DBMS_STATS.GATHER_TABLE_STATS procedure to FALSE so that the session cursor cache would be flushed of all references to the T4 table, which then leads to the SQL statements appearing in the trace file on repeated executions if you need to execute the script a couple of times from that line of the test script to the end.  I also moved the DBMS_MONITOR.SESSION_TRACE_ENABLE call outside of the anonymous PL/SQL block so that PL/SQL block would be included in the trace file.

After executing the test script, I found the following lines near the start of the generated 10046 trace file:

PARSING IN CURSOR #481810328 len=63 dep=0 uid=64 oct=47 lid=64 tim=2302523409025 hv=2690357649 ad='3edbc1610' sqlid='dsvxr0ah5r6cj'
BEGIN DBMS_MONITOR.SESSION_TRACE_ENABLE ( waits=>true ); END;
END OF STMT 

In the above, note the tim=2302523409025 value.  Near the end of the 10046 trace file I found the following lines:

PARSING IN CURSOR #481810424 len=48 dep=0 uid=64 oct=47 lid=64 tim=2302533226450 hv=3127860446 ad='3eb311428' sqlid='80u1a4kx6yr6y'
BEGIN DBMS_MONITOR.SESSION_TRACE_DISABLE; END;
END OF STMT

In the above, note the tim=2302533226450 value.

We are able to determine the elapsed time between these two calls with simple mathematics: (2302533226450 – 2302523409025) / 1000000 = 9.817425 seconds elapsed time in the trace file between the start of the 10046 trace and the call to end the 10046 trace.

Now, let’s see if we are able to find another case where TKPROF lies.  We are able to find a hint of the problems in the book “Optimizing Oracle Performance” (see page xv and page 90).  Here is the TKPROF summary for my execution of the test script (generated with this command: tkprof or1122p_ora_19180_10046_FIND_ME.trc or1122p_ora_19180_10046_FIND_ME.txt):

OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        2      0.00       0.00          0          0          0           0
Execute      3      5.36       5.34          0          0          0           3
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        5      5.36       5.35          0          0          0           3

Misses in library cache during parse: 2
Misses in library cache during execute: 1

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  SQL*Net message to client                       2        0.00          0.00
  SQL*Net message from client                     2        0.10          0.10

OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute  72720      0.99       1.08          0          0          0           0
Fetch    72720      1.01       1.02       1240     218319          0       72719
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total   145441      2.01       2.10       1240     218319          0       72719

Misses in library cache during parse: 1

Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file sequential read                      1047        0.00          0.25
  db file scattered read                        182        0.00          0.10
  asynch descriptor resize                      291        0.00          0.00

    4  user  SQL statements in session.
    0  internal SQL statements in session.
    4  SQL statements in session.
********************************************************************************
Trace file: or1122p_ora_19180_10046_FIND_ME.trc
Trace file compatibility: 11.1.0.7
Sort options: default

       1  session in tracefile.
       4  user  SQL statements in trace file.
       0  internal SQL statements in trace file.
       4  SQL statements in trace file.
       4  unique SQL statements in trace file.
  220086  lines in trace file.
       9  elapsed seconds in trace file. 

Does the above output indicate that my anonymous PL/SQL block completed in 5.35 seconds, and that the recursive call accounted for 2.1 seconds of that time?  Or, should we add the non-recursive and recursive elapsed times together to determine the total time?  Should we add the total blocks read from disk displayed in the non-recursive portion (0) to the total blocks read from disk displayed in the recursive portion (1240) to determine the total number of blocks read from disk during the execution?  The above statistics indicate that there were 1047 waits for the db file sequential read wait event and 182 waits for the db file scattered read wait event in the OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS section of the TKPROF output.  We know that each db file sequential read wait event represents reading one block from disk, and that each db file scattered read wait event represents reading at least 2 blocks from disk.  Let’s try some simple mathematics: 1047 + (2 * 182) = 1411 – the wait event statistics for the OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS section of the TKPROF output indicate that at least 1,411 blocks should have been read from disk, while the TKPROF output only shows 1,240 blocks.  Is TKPROF lying?

Let’s take another look at the raw 10046 trace file.  Very close to the end of the trace file I found the following line:

EXEC #481810328:c=9266459,e=9631079,p=3231,cr=267427,cu=0,mis=0,r=1,dep=0,og=1,plh=0,tim=2302533148184 

The above indicates that cursor number 481810328 consumed 9.266459 seconds of CPU time, 9.631079 seconds of elapsed time, read 3,231 blocks from disk, and performed 267,427 consistent gets when that cursor was executed – yet that is not in agreement with the above TKPROF output (the numbers are greater than even the non-recursive plus recursive statistics would suggest).  If we search upward in the trace file, we are able to find the code that is associated with cursor number 481810328:

PARSING IN CURSOR #481810328 len=610 dep=0 uid=64 oct=47 lid=64 tim=2302523516952 hv=468044383 ad='3edbc2e48' sqlid='8zytk5ndybkkz'
DECLARE
        type array is table of t4%ROWTYPE index by binary_integer;
        l_data array;
        l_rec t4%rowtype;
BEGIN
        SELECT
                a.*
                ,RPAD('*',4000,'*') AS PADDING1
                ,RPAD('*',4000,'*') AS PADDING2
        BULK COLLECT INTO
        l_data
        FROM ALL_OBJECTS a;
        FOR rs IN 1 .. l_data.count
        LOOP
                BEGIN
                        SELECT * INTO l_rec FROM t4 WHERE object_id = l_data(rs).object_id;
                EXCEPTION
                  WHEN NO_DATA_FOUND THEN NULL;
                END;
        END LOOP;
END;
END OF STMT 

That is our anonymous PL/SQL block, which would fall into the non-recursive totals that shows a total of 5.36 seconds of CPU time, 5.35 seconds of elapsed time, 0 blocks read from disk, and 0 consistent gets.

So, did TKPROF answer more questions than it generated?

I was curious, so I processed the trace file using my Hyper-Extended Oracle Performance Monitor program (AKA: Toy Project for Performance Monitoring).  This is the what was written to one of the output files:

Total for Trace File:
|PARSEs       3|CPU S    0.000000|CLOCK S    0.006473|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      3|
|EXECs    72723|CPU S   10.264864|CLOCK S   10.715043|ROWs        3|PHY RD BLKs      3231|CON RD BLKs (Mem)    267427|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|FETCHs   72720|CPU S    1.014008|CLOCK S    1.021754|ROWs    72719|PHY RD BLKs      1240|CON RD BLKs (Mem)    218319|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|

Wait Event Summary:
SQL*Net message to client           0.000010  On Client/Network   Min Wait:     0.000003  Avg Wait:     0.000005  Max Wait:     0.000007  Waits:      2
SQL*Net message from client         0.107469  On Client/Network   Min Wait:     0.006123  Avg Wait:     0.053735  Max Wait:     0.101346  Waits:      2
db file sequential read             0.254673  On DB Server        Min Wait:     0.000023  Avg Wait:     0.000243  Max Wait:     0.001280  Waits:   1047
db file scattered read              0.100087  On DB Server        Min Wait:     0.000088  Avg Wait:     0.000550  Max Wait:     0.004496  Waits:    182
asynch descriptor resize            0.000359  On DB Server        Min Wait:     0.000000  Avg Wait:     0.000001  Max Wait:     0.000004  Waits:    291

Total for Similar SQL Statements in Each Group:
----------------------------------------------------------------------------------
Similar SQL Statements in Group: 1 
First Reference: Cursor 621   Ver 1   Parse at 0.000000 SQL_ID='dsvxr0ah5r6cj'
|PARSEs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
|EXECs        1|CPU S    0.000000|CLOCK S    0.000941|ROWs        1|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|FETCHs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
  CPU S 0.00%  CLOCK S 0.01%
  *    0.101349 seconds of time related to client/network events

----------------------------------------------------------------------------------
Similar SQL Statements in Group: 1 
First Reference: Cursor 621   Ver 2   Parse at 0.107927 SQL_ID='8zytk5ndybkkz'
|PARSEs       1|CPU S    0.000000|CLOCK S    0.004720|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|EXECs        1|CPU S    9.266459|CLOCK S    9.631079|ROWs        1|PHY RD BLKs      3231|CON RD BLKs (Mem)    267427|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
|FETCHs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
  CPU S 82.16%  CLOCK S 82.05%
  *    0.006130 seconds of time related to client/network events
|    ++++++++++++++++||    ++++++++++++++++|

----------------------------------------------------------------------------------
Similar SQL Statements in Group: 1 
First Reference: Cursor 648   Ver 1   Parse at 2.291806 SQL_ID='5k52u3cqwkkmw'
|PARSEs       1|CPU S    0.000000|CLOCK S    0.000441|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|EXECs    72720|CPU S    0.998405|CLOCK S    1.081928|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
|FETCHs   72720|CPU S    1.014008|CLOCK S    1.021754|ROWs    72719|PHY RD BLKs      1240|CON RD BLKs (Mem)    218319|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
  CPU S 17.84%  CLOCK S 17.92%
  *    0.075437 seconds of time related data file I/O
|                ++++||                ++++|

----------------------------------------------------------------------------------
Similar SQL Statements in Group: 1 
First Reference: Cursor 717   Ver 1   Parse at 9.817425 SQL_ID='80u1a4kx6yr6y'
|PARSEs       1|CPU S    0.000000|CLOCK S    0.001312|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|EXECs        1|CPU S    0.000000|CLOCK S    0.001095|ROWs        1|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
|FETCHs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
  CPU S 0.00%  CLOCK S 0.02%

End of Summary, Detail Follows
--------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------
Similar SQL Statements in Group: 1
|PARSEs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
|EXECs        1|CPU S    0.000000|CLOCK S    0.000941|ROWs        1|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|FETCHs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|

Statement Depth 0 (Application Code)
Cursor 621   Ver 1   Parse at 0.000000 SQL_ID='dsvxr0ah5r6cj'  Similar Cnt 1
|PARSEs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
|EXECs        1|CPU S    0.000000|CLOCK S    0.000941|ROWs        1|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|FETCHs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
  CPU S 0.00%  CLOCK S 0.01%

BEGIN DBMS_MONITOR.SESSION_TRACE_ENABLE ( waits=>true ); END;
------------
----------------------------------------------------------------------------------
Similar SQL Statements in Group: 1
|PARSEs       1|CPU S    0.000000|CLOCK S    0.004720|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|EXECs        1|CPU S    9.266459|CLOCK S    9.631079|ROWs        1|PHY RD BLKs      3231|CON RD BLKs (Mem)    267427|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
|FETCHs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|

Statement Depth 0 (Application Code)
Cursor 621   Ver 2   Parse at 0.107927 SQL_ID='8zytk5ndybkkz'  (TD Prev 0.107927)  Similar Cnt 1
|PARSEs       1|CPU S    0.000000|CLOCK S    0.004720|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|EXECs        1|CPU S    9.266459|CLOCK S    9.631079|ROWs        1|PHY RD BLKs      3231|CON RD BLKs (Mem)    267427|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
|FETCHs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
  CPU S 82.16%  CLOCK S 82.05%
|    ++++++++++++++++||    ++++++++++++++++|
DECLARE
        type array is table of t4%ROWTYPE index by binary_integer;
        l_data array;
        l_rec t4%rowtype;
BEGIN
        SELECT
                a.*
                ,RPAD('*',4000,'*') AS PADDING1
                ,RPAD('*',4000,'*') AS PADDING2
        BULK COLLECT INTO
        l_data
        FROM ALL_OBJECTS a;
        FOR rs IN 1 .. l_data.count
        LOOP
                BEGIN
                        SELECT * INTO l_rec FROM t4 WHERE object_id = l_data(rs).object_id;
                EXCEPTION
                  WHEN NO_DATA_FOUND THEN NULL;
                END;
        END LOOP;
END;
------------
----------------------------------------------------------------------------------
Similar SQL Statements in Group: 1
|PARSEs       1|CPU S    0.000000|CLOCK S    0.000441|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|EXECs    72720|CPU S    0.998405|CLOCK S    1.081928|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
|FETCHs   72720|CPU S    1.014008|CLOCK S    1.021754|ROWs    72719|PHY RD BLKs      1240|CON RD BLKs (Mem)    218319|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|

Statement Depth 1 (Trigger Code)
Cursor 648   Ver 1   Parse at 2.291806 SQL_ID='5k52u3cqwkkmw'  (TD Prev 2.183879)  Similar Cnt 1
|PARSEs       1|CPU S    0.000000|CLOCK S    0.000441|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|EXECs    72720|CPU S    0.998405|CLOCK S    1.081928|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
|FETCHs   72720|CPU S    1.014008|CLOCK S    1.021754|ROWs    72719|PHY RD BLKs      1240|CON RD BLKs (Mem)    218319|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
  CPU S 17.84%  CLOCK S 17.92%
|                ++++||                ++++|
SELECT * FROM T4 WHERE OBJECT_ID = :B1

       (Rows 1)   TABLE ACCESS BY INDEX ROWID T4 (cr=3 pr=16 pw=0 time=1274 us cost=2 size=97 card=1)
       (Rows 1)    INDEX RANGE SCAN IND_T4_OBJECT_ID (cr=2 pr=8 pw=0 time=620 us cost=1 size=0 card=1)

------------
----------------------------------------------------------------------------------
Similar SQL Statements in Group: 1
|PARSEs       1|CPU S    0.000000|CLOCK S    0.001312|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|EXECs        1|CPU S    0.000000|CLOCK S    0.001095|ROWs        1|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
|FETCHs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|

Statement Depth 0 (Application Code)
Cursor 717   Ver 1   Parse at 9.817425 SQL_ID='80u1a4kx6yr6y'  (TD Prev 7.525619)  Similar Cnt 1
|PARSEs       1|CPU S    0.000000|CLOCK S    0.001312|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|EXECs        1|CPU S    0.000000|CLOCK S    0.001095|ROWs        1|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
|FETCHs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0|
  CPU S 0.00%  CLOCK S 0.02%

BEGIN DBMS_MONITOR.SESSION_TRACE_DISABLE; END;

So, what does the above indicate when compared with the TKPROF output?

Let’s look at the summary again from TKPROF:

OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS
call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        2      0.00       0.00          0          0          0           0
Execute      3      5.36       5.34          0          0          0           3
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        5      5.36       5.35          0          0          0           3

OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS
call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute  72720      0.99       1.08          0          0          0           0
Fetch    72720      1.01       1.02       1240     218319          0       72719
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total   145441      2.01       2.10       1240     218319          0       72719 

And the summary from my program:

Total for Trace File:
|PARSEs       3|CPU S    0.000000|CLOCK S    0.006473|PHY RD BLKs         0|CON RD BLKs (Mem)         0|
|EXECs    72723|CPU S   10.264864|CLOCK S   10.715043|PHY RD BLKs      3231|CON RD BLKs (Mem)    267427|
|FETCHs   72720|CPU S    1.014008|CLOCK S    1.021754|PHY RD BLKs      1240|CON RD BLKs (Mem)    218319| 

My program double counted the CPU time and elapsed time (and ? … see below) from the recursive calls.  TKPROF only reported the consistent gets (218,319) and physical blocks read from disk (1,240) from the recursive FETCH lines in the trace file, but discarded those statistics from the non-recursive EXEC lines in the trace file.  That possibly explans why the number of db file single block read waits plus two times the number of db file scattered read waits exceeds the number reported in the disk column of the TKPROF output:

1047 + (2 * 182) = 1411 > 1200

The partially interesting item is that the db file scattered read waits, on average, read 12.00 blocks at a time, and the many of the db file scattered read waits are attributed to a mysterious cursor #482246872.  Another output of my program is an Excel spreadsheet that lists all wait events found in the trace file.  I added the following formula in cell F2 and copied that formula down to the last wait event line:

=IF(C2="db file sequential read",VALUE(MID(D2,FIND("blocks=",D2)+7,FIND(" ",D2,FIND("blocks",D2)+1) - (FIND("blocks=",D2)+7))),"")

I added the following formula in cell G2 and copied that formula down to the last wait event line:

=IF(C2="db file scattered read",VALUE(MID(D2,FIND("blocks=",D2)+7,FIND(" ",D2,FIND("blocks",D2)+1) - (FIND("blocks=",D2)+7))),"")

With those two formulas in place, I was able to determine the actual number of physical block reads, and whether or not the blocks were read one at a time, or more than one block at a time.  The resulting Excel spreadsheet looks like this:

…

…

 

So, the above screen captures show that there were 9.8159 seconds in the trace file between the first and last tim= values.  The last of the above screen captures shows there were in total 1,047 single block reads (db file sequential read waits) and 182 multi-block reads (db file scattered read waits), with a total of 2,184 blocks read by multi-block read for an average of 12 blocks per multi-block read.

Let’s take another look at the other output file from my program:

Total for Trace File:
|PARSEs       3|CPU S    0.000000|CLOCK S    0.006473|PHY RD BLKs         0|CON RD BLKs (Mem)         0|
|EXECs    72723|CPU S   10.264864|CLOCK S   10.715043|PHY RD BLKs      3231|CON RD BLKs (Mem)    267427|
|FETCHs   72720|CPU S    1.014008|CLOCK S    1.021754|PHY RD BLKs      1240|CON RD BLKs (Mem)    218319|  

Well, it seems that the recursive SQL has caused a bit of a problem for my program.  The 1,240 physical block reads found on the FETCHs line are actually included in the 3,231 (1,047 + 2,184) physical block reads found on the EXECs line.

—

As a final self-answering question, which is worse: that TKPROF reports only 1,240 physical block reads, or that my program double-counts these 1,240 physical block reads and suggest that there were 4,471 physical block reads?  Maybe we should just ignore the recursive SQL statistics by checking an option box in my program:

Total for Trace File:
|PARSEs       2|CPU S    0.000000|CLOCK S    0.006032|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      2|
|EXECs        3|CPU S    9.266459|CLOCK S    9.633115|ROWs        3|PHY RD BLKs      3231|CON RD BLKs (Mem)    267427|CUR RD BLKs (Mem)         0|SHARED POOL MISs      1|
|FETCHs       0|CPU S    0.000000|CLOCK S    0.000000|ROWs        0|PHY RD BLKs         0|CON RD BLKs (Mem)         0|CUR RD BLKs (Mem)         0|SHARED POOL MISs      0| 

That option in my program fixed the CPU seconds, elapsed time, total physical blocks read, and total consistent gets, but now the number of EXECs, FETCHs, and ROWs do not match what is found in the raw 10046 trace file.

As the above Excel output and my program’s fixed output demonstrates, sometimes there is no replacement for examining the raw 10046 trace files.

Dumping Trace Events – What You See is Not Necessarily What You Get

January 9, 2012

A recent thread in the comp.databases.oracle.server Usenet group brought back memories of when Randolf Geist and I worked on the two chapters for the “Expert Oracle Practices” book.  In the chapters, among other things, we demonstrated how to enable 10046 trace files for sessions using various approaches (some of the approaches change the SQL_TRACE, SQL_TRACE_WAITS, and SQL_TRACE_BINDS columns in V$SESSION to indicate that a 10046 trace is active, and other methods do not), and how to use ORADEBUG to see which events are enabled in other sessions.

The OP in the thread mentioned that the ORADEBUG EVENTDUMP session command, when executed in SQL*Plus by a SYSDBA user, showed no events enabled for another session when the DBMS_MONITOR package was used to enable a 10046 trace at level 12 for the other session.  I have a vague memory of reading (somewhere) that the ORADEBUG EVENTDUMP session command will only show newly enabled events for another session after that other session issues another call to the database (parse, execute, fetch, etc.).  In the Usenet group thread I stated something that is slightly less complete: “I believe that the session with the trace enabled through DBMS_MONITOR must execute at least one SQL statement after tracing is enabled for the session, before ORADEBUG will report that the trace is enabled.”

In the Usenet thread I put together a quick test case script that demonstrated that after the second session (Session 1) issued a SELECT statement, the first session (Session 2) was able to issue a ORADEBUG EVENTDUMP session command and determine that a 10046 trace at level 12 was enabled for the other session.  I believe that the OP answered his own question after a bit of experimentation and searching.  I thought that I would provide an extended version of the test case script in case anyone would like to experiment a bit.  In this test case, Session 1 will be the target of the tracing – that session must be able to query V$SESSION and V$PROCESS during the script execution, while Session 2 will be connected as the SYS user.  In the test case script I have removed the SQL> prompts to make it easier for you to copy and paste the scripts into SQL*Plus, and when the returned output is important, that information also appears in the test case script (all commands executed begin with UPPERCASE keywords).

In Session 1, execute the following SQL statement to pick up the SID, SERIAL# and PID for Session 1, along with the columns in V$SESSION that indicate whether or not a 10046 trace at level 1, 4, 8, or 12 is enabled.  The SID and SERIAL# will be used in Session 2 to enable a trace in Session 1, and the PID will be used with ORADEBUG in Session 2:

SELECT
  S.SID,
  S.SERIAL#,
  P.PID,
  S.SQL_TRACE,
  S.SQL_TRACE_WAITS,
  S.SQL_TRACE_BINDS
FROM
  V$SESSION S,
  V$PROCESS P
WHERE
  S.SID=(SELECT SID FROM V$MYSTAT WHERE ROWNUM=1)
  AND S.PADDR=P.ADDR;

SID    SERIAL#        PID SQL_TRAC SQL_T SQL_T
--- ---------- ---------- -------- ----- -----
  4        143         24 DISABLED FALSE FALSE 

As can be determined by the above output, the SID is 4, PID is 24, and a 10046 trace is not enabled for the session (fair warning, the SQL_TRACE column can be misleading).

In Session 2, enable a 10046 trace using the DBMS_MONITOR package for Session 1 at level 12, and dump the events for Session 1 (note that this test case script is being executed on a Windows server with Oracle Database 11.2.0.2):

EXEC DBMS_MONITOR.SESSION_TRACE_ENABLE(SESSION_ID=>4,SERIAL_NUM=>143,WAITS=>TRUE,BINDS=>TRUE)

ORADEBUG SETORAPID 24

ORADEBUG EVENTDUMP session
Statement processed. 

The output of the ORADEBUG EVENTDUMP session command is significant – the Statement processed. note indicates that the statement executed successfully with no return output. 

In Session 1, let’s execute the same SQL statement as was executed earlier, to not only confirm that the DBMS_MONITOR.SESSION_TRACE_ENABLE call executed by Session 2 was successful, but also to issue another database call in the session (the significance will be seen later):

SELECT
  S.SID,
  S.SERIAL#,
  P.PID,
  S.SQL_TRACE,
  S.SQL_TRACE_WAITS,
  S.SQL_TRACE_BINDS
FROM
  V$SESSION S,
  V$PROCESS P
WHERE
  S.SID=(SELECT SID FROM V$MYSTAT WHERE ROWNUM=1)
  AND S.PADDR=P.ADDR;

SID    SERIAL#        PID SQL_TRAC SQL_T SQL_T
--- ---------- ---------- -------- ----- -----
  4        143         24 ENABLED  TRUE  TRUE

The above output shows that the DBMS_MONITOR.SESSION_TRACE_ENABLE call was successful, as indicated by the changed values in the SQL_TRACE, SQL_TRACE_WAITS, and SQL_TRACE_BINDS columns – a level 12 10046 trace is enabled.

In Session 2, let’s check again which events are enabled for Session 1:

ORADEBUG EVENTDUMP session
sql_trace level=12 

Notice in the above output that ORADEBUG now indicates that a 10046 trace at level 12 is enabled for Session 1, because another database call (initiated when the SQL statement was executed) was performed in Session 1 after the trace was enabled.

—

Let’s try a bit more experimentation.  In Session 2, let’s disable the trace in Session 1 using the DBMS_MONITOR package, and verify whether or not tracing is enabled in Session 1 by using ORADEBUG:

EXEC DBMS_MONITOR.SESSION_TRACE_DISABLE(SESSION_ID=>4,SERIAL_NUM=>143)
PL/SQL procedure successfully completed.

ORADEBUG EVENTDUMP session
sql_trace level=12 

The above output shows that ORADEBUG still indicates that a 10046 trace at level 12 is enabled for Session 1.  In Session 2, let’s check the SQL_TRACE, SQL_TRACE_WAITS, and SQL_TRACE_BINDS columns for Session 1:

SELECT
  S.SID,
  S.SERIAL#,
  P.PID,
  S.SQL_TRACE,
  S.SQL_TRACE_WAITS,
  S.SQL_TRACE_BINDS
FROM
  V$SESSION S,
  V$PROCESS P
WHERE
  S.SID=4
  AND S.PADDR=P.ADDR;

SID    SERIAL#        PID SQL_TRAC SQL_T SQL_T
--- ---------- ---------- -------- ----- -----
  4        143         24 DISABLED FALSE FALSE

ORADEBUG EVENTDUMP session
sql_trace level=12 

So, the query of V$SESSION indicates that a 10046 trace for Session 1 is disabled, yet ORADEBUG still insists that a 10046 trace at level 12 is enabled.  WYSIWYG (What You See Is What You Get) is taking a bit of a vacation.  Let’s issue another database call in Session 1 so that ORADEBUG provides correct output about whether or not a 10046 trace is enabled.  In Session 1:

SELECT
  S.SID,
  S.SERIAL#,
  P.PID,
  S.SQL_TRACE,
  S.SQL_TRACE_WAITS,
  S.SQL_TRACE_BINDS
FROM
  V$SESSION S,
  V$PROCESS P
WHERE
  S.SID=(SELECT SID FROM V$MYSTAT WHERE ROWNUM=1)
  AND S.PADDR=P.ADDR;

SID    SERIAL#        PID SQL_TRAC SQL_T SQL_T
--- ---------- ---------- -------- ----- -----
  4        143         24 DISABLED FALSE FALSE 

Let’s confirm in Session 2 that ORADEBUG now reports that a 10046 trace is not enabled in Session 1:

ORADEBUG EVENTDUMP session
Statement processed.  

OK, but what happens if we use ORADEBUG in Session 2 to enable a 10046 trace is Session 1?  In Session 2:

ORADEBUG session_event 10046 trace name context forever,level 12
Statement processed.

ORADEBUG EVENTDUMP session
sql_trace level=12 

The above shows that if ORADEBUG enables a 10046 trace in Session 1, the ORADEBUG EVENTDUMP session command will correctly indicate that a 10046 trace is enabled for Session 1.  Nice, but what about the SQL_TRACE, SQL_TRACE_WAITS, and SQL_TRACE_BINDS columns in V$SESSION for Session 1?  Let’s check using Session 2:

SELECT
  S.SID,
  S.SERIAL#,
  P.PID,
  S.SQL_TRACE,
  S.SQL_TRACE_WAITS,
  S.SQL_TRACE_BINDS
FROM
  V$SESSION S,
  V$PROCESS P
WHERE
  S.SID=4
  AND S.PADDR=P.ADDR;

SID    SERIAL#        PID SQL_TRAC SQL_T SQL_T
--- ---------- ---------- -------- ----- -----
  4        143         24 DISABLED FALSE FALSE 

The above output suggests that a 10046 trace at level 12 is not enabled for Session 1.  Let’s check again from within Session 1 and then determine today’s date (if that second SQL statement appears in the 10046 trace file, then we know that the 10046 trace was in fact enabled for Session 1):

SELECT
  S.SID,
  S.SERIAL#,
  P.PID,
  S.SQL_TRACE,
  S.SQL_TRACE_WAITS,
  S.SQL_TRACE_BINDS
FROM
  V$SESSION S,
  V$PROCESS P
WHERE
  S.SID=(SELECT SID FROM V$MYSTAT WHERE ROWNUM=1)
  AND S.PADDR=P.ADDR;

SID    SERIAL#        PID SQL_TRAC SQL_T SQL_T
--- ---------- ---------- -------- ----- -----
  4        143         24 DISABLED FALSE FALSE

SELECT SYSDATE FROM DUAL;

SYSDATE
---------
08-JAN-12  

The above still suggests that a 10046 trace is not enabled for Session 1 and that today’s date is January 8, 2012.  Let’s jump back to Session 2 and disable the 10046 trace using ORADEBUG:

ORADEBUG session_event 10046 trace name context off
Statement processed. 

Out of curiosity, let’s repeat a portion of the initial test, just to confirm that the initial test results were not a fluke:

EXEC DBMS_MONITOR.SESSION_TRACE_ENABLE(SESSION_ID=>4,SERIAL_NUM=>143,WAITS=>TRUE,BINDS=>TRUE)

ORADEBUG EVENTDUMP session
Statement processed.

SELECT
  S.SID,
  S.SERIAL#,
  P.PID,
  S.SQL_TRACE,
  S.SQL_TRACE_WAITS,
  S.SQL_TRACE_BINDS
FROM
  V$SESSION S,
  V$PROCESS P
WHERE
  S.SID=4
  AND S.PADDR=P.ADDR;

SID    SERIAL#        PID SQL_TRAC SQL_T SQL_T
--- ---------- ---------- -------- ----- -----
  4        143         24 ENABLED  TRUE  TRUE

ORADEBUG EVENTDUMP session
Statement processed. 

Once again, the above shows that ORADEBUG does not initially report that a 10046 trace is enabled in Session 1 when the officially supported method for enabling a 10046 trace in another session is used.  Let’s disable the 10046 trace using the DBMS_MONITOR package and then confirm that ORADEBUG still indicates that a 10046 trace is not enabled in Session 1:

EXEC DBMS_MONITOR.SESSION_TRACE_DISABLE(SESSION_ID=>4,SERIAL_NUM=>143)

ORADEBUG EVENTDUMP session
Statement processed. 

So, what about the unsupported method of enabling a 10046 trace in another session, using SYS.DBMS_SYSTEM.SET_EV?  In Session 2:

EXEC SYS.DBMS_SYSTEM.SET_EV(4, 143, 10046, 12, '')

SELECT
  S.SID,
  S.SERIAL#,
  P.PID,
  S.SQL_TRACE,
  S.SQL_TRACE_WAITS,
  S.SQL_TRACE_BINDS
FROM
  V$SESSION S,
  V$PROCESS P
WHERE
  S.SID=4
  AND S.PADDR=P.ADDR;

SID    SERIAL#        PID SQL_TRAC SQL_T SQL_T
--- ---------- ---------- -------- ----- -----
  4        143         24 ENABLED  TRUE  TRUE

ORADEBUG EVENTDUMP session
Statement processed. 

The SYS.DBMS_SYSTEM.SET_EV call changed the value of the columns in V$SESSION for Session 1, but still did not alter the output of ORADEBUG EVENTDUMP session.

One final test in Session 2:

ORADEBUG session_event 10046 trace name context forever,level 12
Statement processed.

ORADEBUG EVENTDUMP session
sql_trace level=12 

Once again, if we enable the 10046 trace using ORADEBUG, then ORADEBUG is able to show that the trace is enabled for Session 1.

———————————————

At this point you might be curious why I mentioned that the above test case script was performed on a Windows server running Oracle Database 11.2.0.2.  Does the Oracle Database version make a difference in the results?  Does the operating system (Windows, Linux, or Unix) make a difference in the results?  It appears that a part 2 is required for this blog article, where I will compare the above results with the results obtained with 10.2.0.5 on a Windows server and 11.1.0.6 on a Linux server.

———————————————

You are probably curious to see the contents of the trace file for Session 1.  Here is the trace file (if you need help in reading the trace file, see this three part article series):

*** 2012-01-08 09:47:51.686
*** SESSION ID:(4.143) 2012-01-08 09:47:51.686
*** CLIENT ID:() 2012-01-08 09:47:51.686
*** SERVICE NAME:(OR1122) 2012-01-08 09:47:51.686
*** MODULE NAME:(SQL*Plus) 2012-01-08 09:47:51.686
*** ACTION NAME:() 2012-01-08 09:47:51.686

Received ORADEBUG command (#1) 'EVENTDUMP session' from process 'Windows thread id: 12292, image: <none>'
Dumping Event (group=SESSION)

*** 2012-01-08 09:47:51.686
Finished processing ORADEBUG command (#1) 'EVENTDUMP session'

*** 2012-01-08 09:48:18.426
Received ORADEBUG command (#2) 'EVENTDUMP session' from process 'Windows thread id: 12292, image: <none>'
Dumping Event (group=SESSION)

*** 2012-01-08 09:48:18.426
Finished processing ORADEBUG command (#2) 'EVENTDUMP session'

*** 2012-01-08 09:49:16.415
CLOSE #3:c=0,e=20,dep=0,type=0,tim=488596106412
=====================
PARSING IN CURSOR #2 len=37 dep=1 uid=0 oct=3 lid=0 tim=488596107091 hv=1398610540 ad='7ffb8e4b250' sqlid='grwydz59pu6mc'
select text from view$ where rowid=:1
END OF STMT
PARSE #2:c=0,e=264,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=488596107088
BINDS #2:
 Bind#0
  oacdty=11 mxl=16(16) mxlc=00 mal=00 scl=00 pre=00
  oacflg=18 fl2=0001 frm=00 csi=00 siz=16 off=0
  kxsbbbfp=1dbe86d8  bln=16  avl=16  flg=05
  value=00000273.0010.0001
EXEC #2:c=0,e=550,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=3684871272,tim=488596107744
FETCH #2:c=0,e=25,p=0,cr=2,cu=0,mis=0,r=1,dep=1,og=4,plh=3684871272,tim=488596107790
STAT #2 id=1 cnt=1 pid=0 pos=1 obj=69 op='TABLE ACCESS BY USER ROWID VIEW$ (cr=1 pr=0 pw=0 time=0 us cost=1 size=15 card=1)'
CLOSE #2:c=0,e=501,dep=1,type=0,tim=488596108309
=====================
PARSING IN CURSOR #5 len=37 dep=1 uid=0 oct=3 lid=0 tim=488596108602 hv=1398610540 ad='7ffb8e4b250' sqlid='grwydz59pu6mc'
select text from view$ where rowid=:1
END OF STMT
PARSE #5:c=0,e=23,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=3684871272,tim=488596108602
BINDS #5:
 Bind#0
  oacdty=11 mxl=16(16) mxlc=00 mal=00 scl=00 pre=00
  oacflg=18 fl2=0001 frm=00 csi=00 siz=16 off=0
  kxsbbbfp=1dbe86d8  bln=16  avl=16  flg=05
  value=00000273.0012.0001
EXEC #5:c=0,e=47,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=3684871272,tim=488596108701
FETCH #5:c=0,e=11,p=0,cr=2,cu=0,mis=0,r=1,dep=1,og=4,plh=3684871272,tim=488596108727
STAT #5 id=1 cnt=1 pid=0 pos=1 obj=69 op='TABLE ACCESS BY USER ROWID VIEW$ (cr=1 pr=0 pw=0 time=0 us cost=1 size=15 card=1)'
CLOSE #5:c=0,e=23,dep=1,type=0,tim=488596108766
=====================
PARSING IN CURSOR #3 len=37 dep=1 uid=0 oct=3 lid=0 tim=488596109217 hv=1398610540 ad='7ffb8e4b250' sqlid='grwydz59pu6mc'
select text from view$ where rowid=:1
END OF STMT
PARSE #3:c=0,e=14,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=3684871272,tim=488596109217
BINDS #3:
 Bind#0
  oacdty=11 mxl=16(16) mxlc=00 mal=00 scl=00 pre=00
  oacflg=18 fl2=0001 frm=00 csi=00 siz=16 off=0
  kxsbbbfp=1dbe86d8  bln=16  avl=16  flg=05
  value=00000274.0010.0001
EXEC #3:c=0,e=50,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=3684871272,tim=488596109318
FETCH #3:c=0,e=11,p=0,cr=2,cu=0,mis=0,r=1,dep=1,og=4,plh=3684871272,tim=488596109344
STAT #3 id=1 cnt=1 pid=0 pos=1 obj=69 op='TABLE ACCESS BY USER ROWID VIEW$ (cr=1 pr=0 pw=0 time=0 us cost=1 size=15 card=1)'
CLOSE #3:c=0,e=21,dep=1,type=0,tim=488596109385
=====================
PARSING IN CURSOR #4 len=204 dep=0 uid=91 oct=3 lid=91 tim=488596111910 hv=1585533233 ad='7ffb7578ec0' sqlid='dk42yjjg82n9j'
SELECT
  S.SID,
  S.SERIAL#,
  P.PID,
  S.SQL_TRACE,
  S.SQL_TRACE_WAITS,
  S.SQL_TRACE_BINDS
FROM
  V$SESSION S,
  V$PROCESS P
WHERE
  S.SID=(SELECT SID FROM V$MYSTAT WHERE ROWNUM=1)
  AND S.PADDR=P.ADDR
END OF STMT
PARSE #4:c=0,e=5410,p=0,cr=6,cu=0,mis=1,r=0,dep=0,og=1,plh=1985757361,tim=488596111909
EXEC #4:c=0,e=23,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1985757361,tim=488596112094
WAIT #4: nam='SQL*Net message to client' ela= 4 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=488596112181
FETCH #4:c=0,e=1190,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=1,plh=1985757361,tim=488596113404
WAIT #4: nam='SQL*Net message from client' ela= 232 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=488596113665
FETCH #4:c=0,e=397,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1985757361,tim=488596114102
STAT #4 id=1 cnt=1 pid=0 pos=1 obj=0 op='NESTED LOOPS  (cr=0 pr=0 pw=0 time=0 us cost=0 size=53 card=1)'
STAT #4 id=2 cnt=1 pid=1 pos=1 obj=0 op='NESTED LOOPS  (cr=0 pr=0 pw=0 time=0 us cost=0 size=49 card=1)'
STAT #4 id=3 cnt=810 pid=2 pos=1 obj=0 op='MERGE JOIN CARTESIAN (cr=0 pr=0 pw=0 time=346 us cost=0 size=16146 card=598)'
STAT #4 id=4 cnt=30 pid=3 pos=1 obj=0 op='FIXED TABLE FULL X$KSUPR (cr=0 pr=0 pw=0 time=0 us cost=0 size=494 card=26)'
STAT #4 id=5 cnt=810 pid=3 pos=2 obj=0 op='BUFFER SORT (cr=0 pr=0 pw=0 time=130 us cost=0 size=184 card=23)'
STAT #4 id=6 cnt=27 pid=5 pos=1 obj=0 op='FIXED TABLE FULL X$KSLWT (cr=0 pr=0 pw=0 time=26 us cost=0 size=184 card=23)'
STAT #4 id=7 cnt=1 pid=2 pos=2 obj=0 op='FIXED TABLE FIXED INDEX X$KSUSE (ind:1) (cr=0 pr=0 pw=0 time=0 us cost=0 size=22 card=1)'
STAT #4 id=8 cnt=1 pid=7 pos=1 obj=0 op='COUNT STOPKEY (cr=0 pr=0 pw=0 time=0 us)'
STAT #4 id=9 cnt=1 pid=8 pos=1 obj=0 op='FIXED TABLE FULL X$KSUMYSTA (cr=0 pr=0 pw=0 time=0 us cost=0 size=40 card=2)'
STAT #4 id=10 cnt=1 pid=9 pos=1 obj=0 op='FIXED TABLE FULL X$KSUSGIF (cr=0 pr=0 pw=0 time=0 us cost=0 size=4 card=1)'
STAT #4 id=11 cnt=1 pid=1 pos=2 obj=0 op='FIXED TABLE FIXED INDEX X$KSLED (ind:2) (cr=0 pr=0 pw=0 time=0 us cost=0 size=4 card=1)'
WAIT #4: nam='SQL*Net message to client' ela= 2 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=488596114237

*** 2012-01-08 09:50:22.069
Received ORADEBUG command (#3) 'EVENTDUMP session' from process 'Windows thread id: 12292, image: <none>'
Dumping Event (group=SESSION)

*** 2012-01-08 09:50:22.069
Finished processing ORADEBUG command (#3) 'EVENTDUMP session'

*** 2012-01-08 09:51:24.578
Received ORADEBUG command (#4) 'EVENTDUMP session' from process 'Windows thread id: 12292, image: <none>'
Dumping Event (group=SESSION)

*** 2012-01-08 09:51:24.578
Finished processing ORADEBUG command (#4) 'EVENTDUMP session'

*** 2012-01-08 09:54:05.768
Received ORADEBUG command (#5) 'EVENTDUMP session' from process 'Windows thread id: 12292, image: <none>'
Dumping Event (group=SESSION)

*** 2012-01-08 09:54:05.768
Finished processing ORADEBUG command (#5) 'EVENTDUMP session'

*** 2012-01-08 09:55:01.068
WAIT #4: nam='SQL*Net message from client' ela= 344636055 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=488940750307

*** 2012-01-08 09:55:48.744
Received ORADEBUG command (#6) 'EVENTDUMP session' from process 'Windows thread id: 12292, image: <none>'
Dumping Event (group=SESSION)

*** 2012-01-08 09:55:48.744
Finished processing ORADEBUG command (#6) 'EVENTDUMP session'

*** 2012-01-08 09:56:50.964
Received ORADEBUG command (#7) 'session_event 10046 trace name context forever,level 12' from process 'Windows thread id: 12292, image: <none>'

*** 2012-01-08 09:56:50.964
Finished processing ORADEBUG command (#7) 'session_event 10046 trace name context forever,level 12'

*** 2012-01-08 09:57:18.634
Received ORADEBUG command (#8) 'EVENTDUMP session' from process 'Windows thread id: 12292, image: <none>'
Dumping Event (group=SESSION)

*** 2012-01-08 09:57:18.634
Finished processing ORADEBUG command (#8) 'EVENTDUMP session'

*** 2012-01-08 10:00:01.094
WAIT #2: nam='SQL*Net message from client' ela= 300017366 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=489240770346
CLOSE #2:c=0,e=19,dep=0,type=0,tim=489240770476
=====================
PARSING IN CURSOR #5 len=204 dep=0 uid=91 oct=3 lid=91 tim=489240770687 hv=1585533233 ad='7ffb7578ec0' sqlid='dk42yjjg82n9j'
SELECT
  S.SID,
  S.SERIAL#,
  P.PID,
  S.SQL_TRACE,
  S.SQL_TRACE_WAITS,
  S.SQL_TRACE_BINDS
FROM
  V$SESSION S,
  V$PROCESS P
WHERE
  S.SID=(SELECT SID FROM V$MYSTAT WHERE ROWNUM=1)
  AND S.PADDR=P.ADDR
END OF STMT
PARSE #5:c=0,e=130,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1985757361,tim=489240770686
EXEC #5:c=0,e=31,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1985757361,tim=489240770814
WAIT #5: nam='SQL*Net message to client' ela= 3 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=489240770862
FETCH #5:c=0,e=7007,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=1,plh=1985757361,tim=489240777893
WAIT #5: nam='SQL*Net message from client' ela= 192 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=489240778675
FETCH #5:c=0,e=336,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1985757361,tim=489240779049
STAT #5 id=1 cnt=1 pid=0 pos=1 obj=0 op='NESTED LOOPS  (cr=0 pr=0 pw=0 time=0 us cost=0 size=53 card=1)'
STAT #5 id=2 cnt=1 pid=1 pos=1 obj=0 op='NESTED LOOPS  (cr=0 pr=0 pw=0 time=0 us cost=0 size=49 card=1)'
STAT #5 id=3 cnt=700 pid=2 pos=1 obj=0 op='MERGE JOIN CARTESIAN (cr=0 pr=0 pw=0 time=349 us cost=0 size=16146 card=598)'
STAT #5 id=4 cnt=28 pid=3 pos=1 obj=0 op='FIXED TABLE FULL X$KSUPR (cr=0 pr=0 pw=0 time=0 us cost=0 size=494 card=26)'
STAT #5 id=5 cnt=700 pid=3 pos=2 obj=0 op='BUFFER SORT (cr=0 pr=0 pw=0 time=81 us cost=0 size=184 card=23)'
STAT #5 id=6 cnt=25 pid=5 pos=1 obj=0 op='FIXED TABLE FULL X$KSLWT (cr=0 pr=0 pw=0 time=0 us cost=0 size=184 card=23)'
STAT #5 id=7 cnt=1 pid=2 pos=2 obj=0 op='FIXED TABLE FIXED INDEX X$KSUSE (ind:1) (cr=0 pr=0 pw=0 time=0 us cost=0 size=22 card=1)'
STAT #5 id=8 cnt=1 pid=7 pos=1 obj=0 op='COUNT STOPKEY (cr=0 pr=0 pw=0 time=0 us)'
STAT #5 id=9 cnt=1 pid=8 pos=1 obj=0 op='FIXED TABLE FULL X$KSUMYSTA (cr=0 pr=0 pw=0 time=0 us cost=0 size=40 card=2)'
STAT #5 id=10 cnt=1 pid=9 pos=1 obj=0 op='FIXED TABLE FULL X$KSUSGIF (cr=0 pr=0 pw=0 time=0 us cost=0 size=4 card=1)'
STAT #5 id=11 cnt=1 pid=1 pos=2 obj=0 op='FIXED TABLE FIXED INDEX X$KSLED (ind:2) (cr=0 pr=0 pw=0 time=0 us cost=0 size=4 card=1)'
WAIT #5: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=489240779190

*** 2012-01-08 10:00:49.694
WAIT #5: nam='SQL*Net message from client' ela= 48590415 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=489289369620
CLOSE #5:c=0,e=16,dep=0,type=0,tim=489289369744
=====================
PARSING IN CURSOR #3 len=24 dep=0 uid=91 oct=3 lid=91 tim=489289370485 hv=124468195 ad='7ffb78cd410' sqlid='c749bc43qqfz3'
SELECT SYSDATE FROM DUAL
END OF STMT
PARSE #3:c=0,e=704,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=1388734953,tim=489289370484
EXEC #3:c=0,e=24,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1388734953,tim=489289370563
WAIT #3: nam='SQL*Net message to client' ela= 3 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=489289370620
FETCH #3:c=0,e=13,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=1,plh=1388734953,tim=489289370661
STAT #3 id=1 cnt=1 pid=0 pos=1 obj=0 op='FAST DUAL  (cr=0 pr=0 pw=0 time=0 us cost=2 size=0 card=1)'
WAIT #3: nam='SQL*Net message from client' ela= 189 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=489289370905
FETCH #3:c=0,e=1,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=0,plh=1388734953,tim=489289370957
WAIT #3: nam='SQL*Net message to client' ela= 2 driver id=1413697536 #bytes=1 p3=0 obj#=-1 tim=489289370997

*** 2012-01-08 10:09:13.997
Received ORADEBUG command (#9) 'session_event 10046 trace name context off' from process 'Windows thread id: 12292, image: <none>'

*** 2012-01-08 10:09:13.997
Finished processing ORADEBUG command (#9) 'session_event 10046 trace name context off'

*** 2012-01-08 10:11:16.960
Received ORADEBUG command (#10) 'EVENTDUMP session' from process 'Windows thread id: 12292, image: <none>'
Dumping Event (group=SESSION)

*** 2012-01-08 10:11:16.960
Finished processing ORADEBUG command (#10) 'EVENTDUMP session'

*** 2012-01-08 10:12:42.528
Received ORADEBUG command (#11) 'EVENTDUMP session' from process 'Windows thread id: 12292, image: <none>'
Dumping Event (group=SESSION)

*** 2012-01-08 10:12:42.528
Finished processing ORADEBUG command (#11) 'EVENTDUMP session'

*** 2012-01-08 10:15:29.621
Received ORADEBUG command (#12) 'EVENTDUMP session' from process 'Windows thread id: 12292, image: <none>'
Dumping Event (group=SESSION)

*** 2012-01-08 10:15:29.621
Finished processing ORADEBUG command (#12) 'EVENTDUMP session'

*** 2012-01-08 10:19:54.941
Received ORADEBUG command (#13) 'EVENTDUMP session' from process 'Windows thread id: 12292, image: <none>'
Dumping Event (group=SESSION)

*** 2012-01-08 10:19:54.941
Finished processing ORADEBUG command (#13) 'EVENTDUMP session'

*** 2012-01-08 10:21:25.459
Received ORADEBUG command (#14) 'session_event 10046 trace name context forever,level 12' from process 'Windows thread id: 12292, image: <none>'

*** 2012-01-08 10:21:25.459
Finished processing ORADEBUG command (#14) 'session_event 10046 trace name context forever,level 12'

*** 2012-01-08 10:21:40.549
Received ORADEBUG command (#15) 'EVENTDUMP session' from process 'Windows thread id: 12292, image: <none>'
Dumping Event (group=SESSION)

*** 2012-01-08 10:21:40.549
Finished processing ORADEBUG command (#15) 'EVENTDUMP session' 

Idle Thoughts – SSD, Redo Logs, and Sector Size

December 18, 2011

It seems that I have been quite busy lately with computer related tasks that are not associated with Oracle Database, making it difficult to slow down and focus on items that have a foundation in logic.  Today I have had some spare time to dig further into the recently released “Oracle Core” book.  A Note on page 123 (obviously a logical page number) gave me a moment to pause.  Quoting a small portion of the Note:

“Recently disk manufacturers have started to produce disks with a 4KB sector size… From 11.2 Oracle lets the DBA choose which is the lesser of two evils by allowing you to specify the block size for the log file as 512 bytes, 1KB, or 4KB.” 

Do you see it yet, the reason to pause?

–

–

–

I am typing this blog article on a laptop that is a bit over a year old with two 256GB Crucial SSD drives in a RAID 0 arrangement.  Even though the laptop supports 3Gb/s transfer speeds from the internal drives rather than the more recent 6Gb/s transfer speeds, the drive arrangement is very quick (quick and potentially subject to a very bad data loss if one drive in the RAID 0 array fails), hitting a speed of 539 MB per second in a parallel full table scan.

I believe that the Crucial SSD drives have a 4 KB sector size, much like many other SSD drives.  It seems that some of Intel’s higher-end single layer drives use a 16 KB sector size (4000h = 16,384, see page 17 for the X25 series and page 22 for the 510 series).  I recall from reading the Oracle Database documentation the following statement:

“Unlike the database block size, which can be between 2K and 32K, redo log files always default to a block size that is equal to the physical sector size of the disk. Historically, this has typically been 512 bytes (512B).

Some newer high-capacity disk drives offer 4K byte (4K) sector sizes for both increased ECC capability and improved format efficiency. Most Oracle Database platforms are able to detect this larger sector size. The database then automatically creates redo log files with a 4K block size on those disks.” 

Do you see it yet, the reason to pause?

–

–

–

I recently read an interesting article titled De-Confusing SSD (for Oracle Databases) that inspired several interesting comments.  I think that the following two statements in the article contributed to the comment count:

SSD’s base memory unit is a cell, which holds 1 bit in SLC and 2 bits in MLC. Cells are organized in pages (usually 4k) and pages are organized in blocks (512K). Data can be read and written in pages, but is always deleted in blocks. This will become really important in a moment.

* Placing redo logs on SSD is not recommended. Exadata now has a “Smart Flash Logging” feature that uses redo logs on SSD. Note that it uses redo logs *also* on SSD. This feature allows Oracle to write redo in parallel to a file on SSD and a file on the magnetic disk, and finish the operation when one of the calls is successful.

Do you see it yet, the reason to pause?

–

–

–

So, the “Oracle Core” book states that as of Oracle Database 11.2 the DBA is able to select a 512 byte, 1024 byte, or 4096 byte (4 KB) block size for the redo log files.  Idle thinking… I have Oracle Database 11.2.0.2 installed on this laptop for educational purposes, let’s try executing a simple SQL statement that retrieves the block size for the redo log files:

SELECT
  GROUP#,
  BLOCKSIZE
FROM
  V$LOG
ORDER BY
  GROUP#;

GROUP#  BLOCKSIZE
------ ----------
     1        512
     2        512
     3        512 

So, if the sector size of my Crucial SSD drive is 4 KB, I have a (default) block size for the redo log files that potentially conflicts with the documentation, and if this is not an isolated issue, it might explain (in part) why the De-Confusing SSD (for Oracle Databases) article states that placing redo logs on SSD is not recommended.

Do you see it yet, the reason to pause?

–

–

–

I wonder if in testing redo log performance on SSD drives, if those drives were provided a “fair” test environment, where the BLOCKSIZE of the redo log files was sized appropriately for the write characteristics of SSD drives?  Any thoughts?

Why Isn’t My Index Used… When USER2 Executes this Query?

November 23, 2011

I previously wrote a couple of articles that mention reasons why an index might not be use for a particular query, including an article that was formatted as a True or False quiz with several reference articles.  A few days ago I saw an OTN thread that caught my curiosity, where the original poster (OP) claimed that the optimizer simply will not use an index to access a table when any user other than the schema owner or the SYS user executes a particular query.

Why is the OP attempting to execute the SQL statement as the SYS user?  The SYS user is special.  As mentioned in my review of the book “Practical Oracle 8i“, as I read the book I wrote the following paraphrase into my notes, the SYS user is special:

Oracle 8i introduces row level security, which uses a PL/SQL function to apply an additional WHERE clause predicate to a table – row level security does not apply to the SYS user. It is important to use CONSISTENT=Y when exporting partitioned tables. When CONSISTENT=N is specified, the export of each partition in a table is treated as a separate transaction, and may be exported at a different SCN number (incremented when any session commits). When tables are exported which contain nested tables, the two physical segments are exported in separate transactions, potentially resulting in inconsistent data during the import if the export was performed with the default CONSISTENT=N.

Is the above paraphrase from this 10 year old book a clue?  Maybe it is a problem related to secure view merging because the SQL statement uses the index when the schema owner executes the SQL statement (there is a very good example of this type of problem found in the book “Troubleshooting Oracle Performance“).  Maybe it is a problem where the public synonym for the table actually points to a view or an entirely different table – the execution plan for the non-schema owner did show a VIEW operation, while the execution plan for the schema owner did not show the VIEW operation.  Maybe it is a problem where the optimizer parameters are adjusted differently for different users – in such a case we might need to dig into the V$SYS_OPTIMIZER_ENV, V$SES_OPTIMIZER_ENV, and V$SQL_OPTIMIZER_ENV views.

Maybe taking a look at the DBMS_XPLAN output would help.  Why does the Predicate Information section of the execution plan show the following only for the non-schema owner?

7 - filter(("SEAL_FLAG" IS NULL OR "SEAL_FLAG"<>'Y'))
9 - filter(("SEAL_FLAG" IS NULL OR "SEAL_FLAG"<>'Y'))
11 - filter(("SEAL_FLAG"<>'Y' OR "SEAL_FLAG" IS NULL))
13 - filter(("SEAL_FLAG"<>'Y' OR "SEAL_FLAG" IS NULL))
19 - filter(("SEAL_FLAG"<>'Y' OR "SEAL_FLAG" IS NULL)) 

A significant clue?  If those predicates were also found in the DBMS_XPLAN generated output for the schema owner (and the SYS user), I would probably conclude that the optimizer generated those additional predicates from defined column constraints, and that a review of a 10053 trace file might help determine what caused those predicates to be automatically created.  However, those predicates did not appear in the execution plan that was generated for the schema owner.  It might be time to start checking the V$VPD_POLICY view for this particular SQL_ID, for example (a completely unrelated test case output):

SELECT
  *
FROM
  V$VPD_POLICY
WHERE
  SQL_ID='6hqw5p9d8g8wf';

ADDRESS          PARADDR            SQL_HASH SQL_ID        CHILD_NUMBER OBJECT_OWNER OBJECT_NAME                    POLICY_GROUP                   POLICY                 POLICY_FUNCTION_OWNER          PREDICATE
---------------- ---------------- ---------- ------------- ------------ ------------ ------------------------------ ------------------------------ ---------------------- ------------------------------ ------------------------------------------------------------------------------------
000007FFB7701608 000007FFB7743350 1518838670 6hqw5p9d8g8wf            0 TESTUSER     T12                            SYS_DEFAULT                    T_SEC                  TESTUSER                       ID < 10 

Maybe we should also check some of the other virtual private database (VPD) related views including ALL_POLICIES (once again from a completely unrelated test case):

SELECT
  *
FROM
  ALL_POLICIES;

OBJECT_OWNER                   OBJECT_NAME                    POLICY_GROUP                  POLICY_NAME                    PF_OWNER                       PACKAGE                       FUNCTION                       SEL INS UPD DEL IDX CHK ENA STA POLICY_TYPE              LON
------------------------------ ------------------------------ ----------------------------- ------------------------------ ------------------------------ ----------------------------- ------------------------------ --- --- --- --- --- --- --- --- ------------------------ ---
TESTUSER                       T12                            SYS_DEFAULT                   T_SEC                          TESTUSER                       S                                                            YES YES YES YES NO  NO  YES NO  DYNAMIC                  NO 

There are known performance problems related to the use of VPD, some of which are Oracle Database version dependent, and some of which have been corrected in recent versions.  Maybe a quick check of one of the following articles would help, if the OP finds that VPD is in fact in use (the second article provides step by step directions for investigation):

• Metalink (MOS) Doc ID 728292.1 “Known Performance Issues When Using TDE and Indexes on the Encrypted Columns”
• Metalink (MOS) Doc ID 967042.1 “How to Investigate Query Performance Regressions Caused by VPD (FGAC) Predicates?”

—

Take a look at the OTN thread.  Any other suggestions for the OP?

10200 Trace Shows Consistent Reads, but Not All Consistent Reads

October 4, 2011

An interesting, but poorly worded, problem appeared in an OTN thread recently where the OP claimed that a 10200 trace was not showing a consistent get, even though a concurrent 10046 trace’s EXEC and STAT lines indicated one consistent get.  The provided test case and sample 10200/10046 trace left a couple of details unspecified, such as the table and index definitions, the tablespace definition, the Oracle Database release version, and exactly what the OP was trying to understand: does a 10200 trace never work, or is it just not producing output in this specific case.

Let’s try a couple of experiments to try to reproduce the output provided by the OP.  Frst let’s make certain that we know what the OP wanted to see.  A 10200 trace is supposed to write out additional lines in a trace file every time a consistent block read is performed – we have previously experimented a bit with 10200 trace file entries.  Let’s connect to the database using two SQL*Plus sessions, session 1 will connect as a standard database user, and session 2 will connect as the SYS user.  We will create a locally managed, non-ASSM, tablespace with uniform 1MB extent sizes to help improve the consistency of the results.  A sample CREATE TABLESPACE command follows (created in session 2):

CREATE SMALLFILE TABLESPACE "LOCAL_UNIFORM1M" DATAFILE 'C:\Oracle\OraData\OR1122D\locun1MOR1122.dbf' SIZE 100M AUTOEXTEND ON NEXT 10M
  MAXSIZE UNLIMITED LOGGING EXTENT MANAGEMENT LOCAL UNIFORM SIZE 1M SEGMENT SPACE MANAGEMENT MANUAL; 

In session 1, we will create a table in the recently created tablespace, populate the table with 100,001 rows, and then execute a couple of throw-away SQL statements that will be used later (this is done to eliminate the hard parse during the later re-execution of the SQL statements):

DROP TABLE T4 PURGE;

CREATE TABLE T4(
  C1 NUMBER,
  C2 VARCHAR2(10),
  C3 VARCHAR2(10))
  TABLESPACE LOCAL_UNIFORM1M;

INSERT INTO
  T4
SELECT
  ROWNUM+10,
  TO_CHAR(ROWNUM),
  TO_CHAR(ROWNUM+10)
FROM
  DUAL
CONNECT BY
  LEVEL<=100000;

COMMIT;

EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>NULL,TABNAME=>'T4',CASCADE=>TRUE)

INSERT INTO T4 VALUES(1,NULL,NULL);
COMMIT;

SET AUTOTRACE TRACEONLY STATISTICS

ALTER SESSION SET TRACEFILE_IDENTIFIER = 'T4_IGNORE_ME';
ALTER SESSION SET EVENTS '10200 TRACE NAME CONTEXT FOREVER, LEVEL 1';
ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT FOREVER, LEVEL 8';

SELECT
  *
FROM
  T4;

SELECT
  SYSDATE
FROM
  DUAL;

INSERT INTO T4 VALUES(2,NULL,NULL);
ROLLBACK;

ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT OFF';
ALTER SESSION SET EVENTS '10200 TRACE NAME CONTEXT OFF';

In session 2 (the SYS session), we will execute a SQL statement that examines X$BH where the STATE column is not equal to 0, after flushing the buffer cache.  This query will indicate the blocks that remain in the buffer cache as well as the STATE of the block in the buffer cache.  From http://www.ixora.com.au/q+a/0103/22151356.htm, the meaning of the STATE column (we could have just queried V$BH instead to see the meaning of that column):

0, FREE, no valid block image
1, XCUR, a current mode block, exclusive to this instance
2, SCUR, a current mode block, shared with other instances
3, CR,   a consistent read (stale) block image
4, READ, buffer is reserved for a block being read from disk
5, MREC, a block in media recovery mode
6, IREC, a block in instance (crash) recovery mode

We will need to re-execute the SQL statement a couple of times (waiting a few seconds between executions) until the query results stabilize:

SELECT
  TS#,
  FILE#,
  DBARFIL,
  DBABLK,
  STATE
FROM
  X$BH
WHERE
  STATE<>0
ORDER BY
  TS#,
  FILE#,
  DBABLK;

/

/ 

Once stabilized, the query output appeared as follows for my test case:

 TS#  FILE#    DBARFIL     DBABLK  STATE
---- ------ ---------- ---------- ------
   0      1          1       2016      1
   0      1          1       2017      1 

The above shows two blocks from the SYSTEM tablespace, with a STATE of 1 (current version of the block).

Quickly switching back to session 1, we execute a SQL statement for which we are interested in examining, followed by a second SQL statement that is simply intended to make certain that the STAT lines were written to the trace file (will will need to quickly switch back to session 2 after executing this script):

ALTER SESSION SET TRACEFILE_IDENTIFIER = 'T4_SELECT';
ALTER SESSION SET EVENTS '10200 TRACE NAME CONTEXT FOREVER, LEVEL 1';
ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT FOREVER, LEVEL 8';

SELECT
  *
FROM
  T4;

SELECT
  SYSDATE
FROM
  DUAL;

ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT OFF';
ALTER SESSION SET EVENTS '10200 TRACE NAME CONTEXT OFF'; 

In session 2 (the SYS session), execute the following:

/ 

Session 2 produced the following output (slightly modified due to length, with each added block found in the buffer cache numbered):

 TS#  FILE#    DBARFIL     DBABLK  STATE
---- ------ ---------- ---------- ------
   0      1          1       2016      1
   0      1          1       2017      1
   5      5          5        128      1  *   1
   5      5          5        129      1  *   2
   5      5          5        130      1  *   3
   5      5          5        131      1  *   4
   5      5          5        132      1  *   5
   5      5          5        133      1  *   6
   5      5          5        134      1  *   7
   5      5          5        135      1  *   8
   5      5          5        136      1  *   9
   5      5          5        137      1  *  10
...
   5      5          5        423      1  * 296
   5      5          5        424      1  * 297
   5      5          5        425      1  * 298
   5      5          5        426      1  * 299
   5      5          5        427      1  * 300
   5      5          5        428      1  * 301

303 rows selected. 

The query executed in session 1 added 301 blocks to the buffer cache (XCUR – the current versions of the block).  Let’s take a look at the statistics displayed in session 1 for the execution of the select from T4:

SQL> SELECT
  2    *
  3  FROM
  4    T4;

100001 rows selected.

Statistics
---------------------------------------------------
          0  recursive calls
          0  db block gets
       6967  consistent gets
        301  physical reads
          0  redo size
    2353883  bytes sent via SQL*Net to client
      73685  bytes received via SQL*Net from client
       6668  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
     100001  rows processed 

The above shows 301 blocks read from disk (physical reads), with 6,967 blocks read by consistent read (consistent gets), and 0 current mode reads (db block gets).

Taking a look at a portion of the generated 10200/10046 trace file (some of the rows are numbered to match the comments added to the query of X$BH):

PARSE #391723720:c=0,e=17,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=2560505625,tim=90982737037
EXEC #391723720:c=0,e=15,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=2560505625,tim=90982737090
WAIT #391723720: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=13759 tim=90982737111
WAIT #391723720: nam='Disk file operations I/O' ela= 133 FileOperation=2 fileno=5 filetype=2 obj#=13759 tim=90982737271
WAIT #391723720: nam='db file sequential read' ela= 163 file#=5 block#=128 blocks=1 obj#=71123 tim=90982737471               * 1
ktrget2(): started for block  <0x0005 : 0x01400081> objd: 0x000115d3 env [0x000000001759AF24]: (scn: 0x0000.0048429e  xid: 0x0000.000.00000000  uba: 0x00000000.0000.00  statement num=0  parent xid: 0x0000.000.00000000  st-scn: 0x0000.00000000  hi-scn: 0x0000.00000000  ma-scn: 0x0000.00484274  flg: 0x00000661) 
WAIT #391723720: nam='db file scattered read' ela= 2876 file#=5 block#=129 blocks=127 obj#=71123 tim=90982740602             * 2 through 128
ktrgcm(): completed for block  <0x0005 : 0x01400081> objd: 0x000115d3 
ktrget2(): completed for  block <0x0005 : 0x01400081> objd: 0x000115d3 
FETCH #391723720:c=0,e=3865,p=128,cr=3,cu=0,mis=0,r=1,dep=0,og=1,plh=2560505625,tim=90982740990
WAIT #391723720: nam='SQL*Net message from client' ela= 93 driver id=1413697536 #bytes=1 p3=0 obj#=71123 tim=90982741103
ktrget2(): started for block  <0x0005 : 0x01400081> objd: 0x000115d3 env [0x000000001759AF24]: (scn: 0x0000.0048429e  xid: 0x0000.000.00000000  uba: 0x00000000.0000.00  statement num=0  parent xid: 0x0000.000.00000000  st-scn: 0x0000.00000000  hi-scn: 0x0000.00000000  ma-scn: 0x0000.00484274  flg: 0x00000660) 
ktrexf(): returning 9 on:  0000000012658C48  cr-scn: 0xffff.ffffffff  xid: 0x0000.000.00000000  uba: 0x00000000.0000.00  cl-scn: 0xffff.ffffffff  sfl: 0 
ktrgcm(): completed for block  <0x0005 : 0x01400081> objd: 0x000115d3 ktrget2(): completed for  block <0x0005 : 0x01400081> objd: 0x000115d3
WAIT #391723720: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=71123 tim=90982741172
FETCH #391723720:c=0,e=55,p=0,cr=1,cu=0,mis=0,r=15,dep=0,og=1,plh=2560505625,tim=90982741183
...
WAIT #391723720: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=71123 tim=90983507550
ktrget2(): started for block  <0x0005 : 0x014001ac> objd: 0x000115d3 env [0x000000001759AF24]: (scn: 0x0000.0048429e  xid: 0x0000.000.00000000  uba: 0x00000000.0000.00  statement num=0  parent xid: 0x0000.000.00000000  st-scn: 0x0000.00000000  hi-scn: 0x0000.00000000  ma-scn: 0x0000.00484274  flg: 0x00000660) 
ktrexf(): returning 9 on:  0000000012658C48  cr-scn: 0xffff.ffffffff  xid: 0x0000.000.00000000  uba: 0x00000000.0000.00  cl-scn: 0xffff.ffffffff  sfl: 0 
ktrgcm(): completed for block  <0x0005 : 0x014001ac> objd: 0x000115d3 ktrget2(): completed for  block <0x0005 : 0x014001ac> objd: 0x000115d3
FETCH #391723720:c=0,e=88,p=0,cr=2,cu=0,mis=0,r=10,dep=0,og=1,plh=2560505625,tim=90983507599
STAT #391723720 id=1 cnt=100001 pid=0 pos=1 obj=71123 op='TABLE ACCESS FULL T4 (cr=6967 pr=301 pw=0 time=303617 us cost=84 size=1700000 card=100000)'
WAIT #391723720: nam='SQL*Net message from client' ela= 530 driver id=1413697536 #bytes=1 p3=0 obj#=71123 tim=90983508174

In the above, we see various entries that begin with ktrget2(), ktrgcm(), ktrexf(),  and ktrgcm().  Those entries do not normally appear in a 10046 trace file, so we know that Oracle Database will write 10200 trace information for consistent gets to a trace file, when that event is enabled.  The information conveyed by the additional lines found in the trace file is partially explained here.

Let’s try a second test that reproduces what the OP was attempting to investigate.  In session 2 (the SYS session), let’s flush the buffer cache, and query X$BH until the query results stabilize:

ALTER SYSTEM FLUSH BUFFER_CACHE;
ALTER SYSTEM FLUSH BUFFER_CACHE;

SELECT
  TS#,
  FILE#,
  DBARFIL,
  DBABLK,
  STATE
FROM
  X$BH
WHERE
  STATE<>0
ORDER BY
  TS#,
  FILE#,
  DBABLK;

/

/ 

Once stabilized, the query output for my test appeared as follows (the same blocks as were seen at the start of the previous test above):

 TS#  FILE#    DBARFIL     DBABLK  STATE
---- ------ ---------- ---------- ------
   0      1          1       2016      1
   0      1          1       2017      1 

Quickly switching to session 1, execute the following (then quickly jump to session 2 again):

ALTER SESSION SET TRACEFILE_IDENTIFIER = 'T4_INSERT';
ALTER SESSION SET EVENTS '10200 TRACE NAME CONTEXT FOREVER, LEVEL 1';
ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT FOREVER, LEVEL 8';

INSERT INTO T4 VALUES(2,NULL,NULL);

SELECT
  SYSDATE
FROM
  DUAL;

ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT OFF';
ALTER SESSION SET EVENTS '10200 TRACE NAME CONTEXT OFF'; 

We will later switch back to session 1 to issue a ROLLBACK.  Switch to session 2 (the SYS session) and determine which blocks are in the buffer cache, and the state of those blocks:

/ 

The output that I received in session 2 follows (with a comment that counts the blocks that were added to the buffer cache):

 TS#  FILE#    DBARFIL     DBABLK  STATE
---- ------ ---------- ---------- ------
   0      1          1       2016      1
   0      1          1       2017      1
   2      3          3        224      1   * 1
   2      3          3        230      1   * 2
   5      5          5        128      1   * 3
   5      5          5        427      1   * 4
   5      5          5        428      1   * 5 

7 rows selected.

In the above, notice that 5 blocks were added to the buffer cache, all with a STATE of XCUR (the current version of the blocks).

The statistics that were output in session 1 follow:

SQL> INSERT INTO T4 VALUES(2,NULL,NULL);

1 row created.

Statistics
---------------------------------------------------
          0  recursive calls
          3  db block gets
          1  consistent gets
         35  physical reads
          0  redo size
        557  bytes sent via SQL*Net to client
        501  bytes received via SQL*Net from client
          3  SQL*Net roundtrips to/from client
          1  sorts (memory)
          0  sorts (disk)
          1  rows processed 

Notice in the above that the statistics show 35 blocks read from disk (physical reads), with 1 block read by consistent read (consistent gets), and 3 current mode reads (db block gets).  The OP mentioned finding similar output.  Let’s take a look at a portion of the 10200/10046 trace file (with comments that tie back to the output from X$BH):

PARSING IN CURSOR #392100208 len=34 dep=0 uid=62 oct=2 lid=62 tim=91041503275 hv=544833741 ad='7ffb77546b8' sqlid='96mny1hh7m06d'
INSERT INTO T4 VALUES(2,NULL,NULL)
END OF STMT
PARSE #392100208:c=0,e=23,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=0,tim=91041503275
WAIT #392100208: nam='db file sequential read' ela= 175 file#=5 block#=128 blocks=1 obj#=71123 tim=91041503526     * 3
WAIT #392100208: nam='db file scattered read' ela= 971 file#=5 block#=427 blocks=32 obj#=71123 tim=91041504601     * 4 and 5
WAIT #392100208: nam='db file sequential read' ela= 144 file#=3 block#=224 blocks=1 obj#=0 tim=91041504795         * 1
WAIT #392100208: nam='db file sequential read' ela= 142 file#=3 block#=230 blocks=1 obj#=0 tim=91041504992         * 2
EXEC #392100208:c=0,e=1749,p=35,cr=1,cu=3,mis=0,r=1,dep=0,og=1,plh=0,tim=91041505052                               * One Consistent Read, 3 Current Mode Reads
STAT #392100208 id=1 cnt=0 pid=0 pos=1 obj=0 op='LOAD TABLE CONVENTIONAL  (cr=1 pr=35 pw=0 time=1729 us)'          * One Consistent Read
...
CLOSE #392100208:c=0,e=3,dep=0,type=1,tim=91041508828 

Note in the above that there is no 10200 trace information that indicates which block was read by consistent read – the problem posed by the OP.  What happened, which block was read by consistent read?

Before we forget, roll back the INSERT in session 1:

ROLLBACK; 

A search found this Oracle-L post that lead to another blog article that suggested making the following change and bouncing the database (note: do not change hidden Oracle parameters in a production environment without the consent of Oracle Support):

ALTER SYSTEM SET "_TRACE_PIN_TIME"=1 SCOPE=SPFILE;

Go through the same process as outlined above of connecting the sessions, creating the table, and making certain that hard parses do not confuse the output.

In session 2 (the SYS session), flush the buffer cache, determine which blocks are in the buffer cache (execute a couple of times until the output stabilizes):

ALTER SYSTEM FLUSH BUFFER_CACHE;
ALTER SYSTEM FLUSH BUFFER_CACHE;

SELECT
  TS#,
  FILE#,
  DBARFIL,
  DBABLK,
  STATE
FROM
  X$BH
WHERE
  STATE<>0
ORDER BY
  TS#,
  FILE#,
  DBABLK;

/

/

Just as shown earlier in this article, I again received the following output:

 TS#  FILE#    DBARFIL     DBABLK  STATE
---- ------ ---------- ---------- ------
   0      1          1       2016      1
   0      1          1       2017      1 

Quickly switch to session 1 and execute the following (prepare to switch back to session 2):

ALTER SESSION SET TRACEFILE_IDENTIFIER = 'T4_NO_INDEX2';
ALTER SESSION SET EVENTS '10200 TRACE NAME CONTEXT FOREVER, LEVEL 1';
ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT FOREVER, LEVEL 8';

INSERT INTO T4 VALUES(2,NULL,NULL);

In session 2 (the SYS session) check the contents of the buffer cache:

/ 

The output that I received follows (note that the numbers to the right of the lines correspond to the order in which the physical block reads were unpinned, as it appeared in the trace file that follows):

 TS#  FILE#    DBARFIL     DBABLK  STATE
---- ------ ---------- ---------- ------
   0      1          1       2016      1
   0      1          1       2017      1
   2      3          3        192      1  *  3
   2      3          3      16029      1  *  1
   5      5          5        128      1  *  X
   5      5          5        427      1  *  2
   5      5          5        428      1  *  2 

A portion of the 10200/10046 trace file follows (with the unpinning order numbered):

PARSING IN CURSOR #367514528 len=34 dep=0 uid=62 oct=2 lid=62 tim=21584856863 hv=544833741 ad='7ffb77d8140' sqlid='96mny1hh7m06d'
INSERT INTO T4 VALUES(2,NULL,NULL)
END OF STMT
PARSE #367514528:c=0,e=489,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=0,tim=21584856863
WAIT #367514528: nam='db file sequential read' ela= 156 file#=5 block#=128 blocks=1 obj#=71114 tim=21584857108  X ****
pin ktswh28: ktsgsp dba 0x1400080:4 time 110020667                                                              X Pin
WAIT #367514528: nam='db file scattered read' ela= 935 file#=5 block#=427 blocks=32 obj#=71114 tim=21584858155  2 ****
pin ktswh72: ktsbget dba 0x14001ab:1 time 110021704                                                             2 Pin
WAIT #367514528: nam='db file sequential read' ela= 132 file#=3 block#=192 blocks=1 obj#=0 tim=21584858344      3 ****
pin ktuwh59: ktugus:ktubnd dba 0xc000c0:25 time 110021892                                                       3 Pin
WAIT #367514528: nam='db file sequential read' ela= 130 file#=3 block#=16029 blocks=1 obj#=0 tim=21584858538    1 ****
pin ktuwh09: ktugfb dba 0xc03e9d:26 time 110022085                                                              1 Pin
pin release        18 ktuwh09: ktugfb dba 0xc03e9d:26                                                           1 Unpin
pin release       428 ktswh72: ktsbget dba 0x14001ab:1                                                          2 Unpin
pin release       246 ktuwh59: ktugus:ktubnd dba 0xc000c0:25                                                    3 Unpin
EXEC #367514528:c=0,e=1731,p=35,cr=1,cu=3,mis=0,r=1,dep=0,og=1,plh=0,tim=21584858629
STAT #367514528 id=1 cnt=0 pid=0 pos=1 obj=0 op='LOAD TABLE CONVENTIONAL  (cr=1 pr=35 pw=0 time=1703 us)'
WAIT #367514528: nam='SQL*Net message to client' ela= 0 driver id=1413697536 #bytes=1 p3=0 obj#=0 tim=21584858681 

In the above, it appears that block 128 was read from file #5, was pinned, and then was still pinned when the EXEC and STAT lines were written to the trace file.  I do not claim to fully understand what appears above – maybe someone can confirm that in fact block 128 was still pinned.  Let’s determine the header block for table T4:

SELECT
  HEADER_FILE,
  HEADER_BLOCK
FROM
  DBA_SEGMENTS
WHERE
  SEGMENT_NAME='T4';

HEADER_FILE HEADER_BLOCK
----------- ------------
          5          128 

Block 128 in file #5 is the segment header block for the T4 test table. I suspect that it is the segment header block that is accounting for the one consistent read in the EXEC and STAT lines. My best guess is that the session is re-reading that block while it is still pinned, and is thus considering that read as a consistent get, even though the session already has the block pinned. That re-read after being pinned could also explain why a 10200 trace would not show that consistent read.

Opinions – or a better explanation?

Invalid Hints are Silently Ignored? An Invalid USE_HASH Hint Transforms a Sort-Merge Join into a Nested Loops Join

September 30, 2011

How many times have you heard or read that the Oracle Database query optimizer may freely ignore hints if it so chooses?  How many times have you heard or read that the query optimizer silently ignores hints that are invalid?  I was recently reminded of both of these concepts while reading the book “Oracle Database 11g Performance Tuning Recipes“.

Page 494 of the book states:

“Tip: Hints with incorrect or improper syntax are ignored by the optimizer.”

Page 496 of the book states:

“Note: Hints influence the optimizer, but the optimizer may still choose to ignore any hints specified in the query.”

Hints with incorrect syntax may cause the optimizer to consider the hints as invalid, but the changed outcome is potentially much worse (or better) than the hint being simply ignored, as demonstrated in this blog article.  As described in another blog article, hints are directives and must be obeyed unless…

I thought that I would craft an interesting, yet simple test case to see if invalid hints are silently ignored.  Here is the test case script:

SET AUTOTRACE OFF

DROP TABLE T1 PURGE;
DROP TABLE T2 PURGE;

CREATE TABLE T1 AS
SELECT
  ROWNUM RN
FROM
  DUAL
CONNECT BY
  LEVEL<=100;

CREATE TABLE T2 AS
SELECT
  ROWNUM RN
FROM
  DUAL
CONNECT BY
  LEVEL<=100;

EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>NULL,TABNAME=>'T1')
EXEC DBMS_STATS.GATHER_TABLE_STATS(OWNNAME=>NULL,TABNAME=>'T2')

ALTER SESSION SET TRACEFILE_IDENTIFIER = 'USE_HASH_10053';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';

SET AUTOTRACE TRACEONLY EXPLAIN

SELECT
  *
FROM
  T1,
  T2
WHERE
  T1.RN>T2.RN;

SELECT /*+ USE_HASH(T1 T2) */
  *
FROM
  T1,
  T2
WHERE
  T1.RN>T2.RN;

SET AUTOTRACE OFF

ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT OFF';

The following shows the output that I received with Oracle Database 11.2.0.2:

SQL> SELECT
  2    *
  3  FROM
  4    T1,
  5    T2
  6  WHERE
  7    T1.RN>T2.RN;

Execution Plan
----------------------------------------------------------
Plan hash value: 412793182

----------------------------------------------------------------------------
| Id  | Operation           | Name | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------
|   0 | SELECT STATEMENT    |      |  4950 | 29700 |     9  (34)| 00:00:01 |
|   1 |  MERGE JOIN         |      |  4950 | 29700 |     9  (34)| 00:00:01 |
|   2 |   SORT JOIN         |      |   100 |   300 |     4  (25)| 00:00:01 |
|   3 |    TABLE ACCESS FULL| T1   |   100 |   300 |     3   (0)| 00:00:01 |
|*  4 |   SORT JOIN         |      |   100 |   300 |     4  (25)| 00:00:01 |
|   5 |    TABLE ACCESS FULL| T2   |   100 |   300 |     3   (0)| 00:00:01 |
----------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   4 - access(INTERNAL_FUNCTION("T1"."RN")>INTERNAL_FUNCTION("T2"."RN"))
       filter(INTERNAL_FUNCTION("T1"."RN")>INTERNAL_FUNCTION("T2"."RN"))

SQL> SELECT /*+ USE_HASH(T1 T2) */
  2    *
  3  FROM
  4    T1,
  5    T2
  6  WHERE
  7    T1.RN>T2.RN;

Execution Plan
----------------------------------------------------------
Plan hash value: 1967407726

---------------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |  4950 | 29700 |   144   (5)| 00:00:01 |
|   1 |  NESTED LOOPS      |      |  4950 | 29700 |   144   (5)| 00:00:01 |
|   2 |   TABLE ACCESS FULL| T1   |   100 |   300 |     3   (0)| 00:00:01 |
|*  3 |   TABLE ACCESS FULL| T2   |    50 |   150 |     1   (0)| 00:00:01 |
---------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   3 - filter("T1"."RN">"T2"."RN")

So, the USE_HASH hint which is invalid because a hash join cannot be used when the join condition is an inequality, causes the optimizer to switch from a sort-merge join to a nested loops join.  Does this mean that invalid hints are potentially NOT silently ignored?

Oracle Database Time Model Viewer in Excel 6

August 18, 2011

(Back to the Previous Post in the Series)

It has been roughly five months since the last installment in this blog article series.  Hopefully, several people have found this series helpful and have adapted the solution to better fit their specific needs.  By the end of the last article the Excel based application not only displayed time model data at the system-wide and session-level, but also operating system statistics (from V$OSSTAT), system-wide and session level wait events, various other statistics (from V$SYSSTAT), execution plans, and also allowed enabling/disabling 10046 extended SQL traces.  A lot of features, but what else may be added to the project?

As of part four of the series, the Excel project should appear similar to the following screen capture (10046 tracing and DBMS XPLAN were added in part five):

You can find the current project, through part five of this series, linked at the bottom of part five’s blog article.  Open that project (or your customized version of the project), right-click the Sheet2 worksheet name and select Rename.  Change the name to Statistics.  Rename the Sheet3 worksheet to Charts by using the same process.  Finally, right-click the Wait Events worksheet name and select View Code.  Expand the Forms group, right-click the frmTimeModel name, and select View Source.  Just before selecting View Source (or View Code), your editor window should appear similar to the following screen capture:

Locate the UserForm_Initialize subroutine (this is the subroutine that connects to the database and prepares to start retrieving statistics from the database).  Locate the following code in that subroutine:

'More code will be copied here
'
'
'

Just above that section of code, add the following commands which will add column titles to the Statistics worksheet when the UserForm is displayed:

'   
    Sheets("Wait Events").Range("B2").Select
    ActiveWindow.FreezePanes = True

    Sheets("Statistics").Cells(1, 1).Value = "Time"
    Sheets("Statistics").Cells(1, 2).Value = "Administrative"
    Sheets("Statistics").Cells(1, 3).Value = "Application"
    Sheets("Statistics").Cells(1, 4).Value = "Commit"
    Sheets("Statistics").Cells(1, 5).Value = "Concurrency"
    Sheets("Statistics").Cells(1, 6).Value = "Configuration"
    Sheets("Statistics").Cells(1, 7).Value = "Network"
    Sheets("Statistics").Cells(1, 8).Value = "Other"
    Sheets("Statistics").Cells(1, 9).Value = "System I/O"
    Sheets("Statistics").Cells(1, 10).Value = "User I/O"

    Sheets("Statistics").Columns("B:AC").EntireColumn.AutoFit

    Sheets("Statistics").Rows("2:10000").Delete Shift:=xlUp

Next, we will programmatically create four charts on the Charts worksheet.  Directly below were the above code was copied (above the ‘More code will be copied here line), add the following code:

    'Remove existing charts, Add the charts
    Sheets("Charts").ChartObjects.Delete

    'Note that these chart styles are likely only compatible with Excel 2007 and later
    With Sheets("Charts").ChartObjects.Add(10, 10, 400, 200)
        .Chart.SetSourceData Source:=Sheets("Statistics").Range("$B$1:$J$21")
        .Chart.SeriesCollection(1).XValues = "='Statistics'!$A$2:$A$21"
        .Chart.ChartType = xlColumnStacked 'xlAreaStacked
        .Chart.HasTitle = True
        .Chart.ChartStyle = 42
        .Chart.ChartTitle.Text = "Wait Event Classes"

        'Rotate the X axis titles
        .Chart.Axes(xlCategory).TickLabels.Orientation = xlUpward

        'Add the vertical Y axis title
        .Chart.Axes(xlValue, xlPrimary).HasTitle = True
        .Chart.Axes(xlValue, xlPrimary).AxisTitle.Characters.Text = "Seconds"
        'Add a gradient to the background of the chart
        .Chart.PlotArea.Fill.OneColorGradient Style:=msoGradientHorizontal, Variant:=2, Degree:=0.756847486076142
        .Chart.PlotArea.Fill.ForeColor.SchemeColor = 23
        .Chart.PlotArea.Fill.Visible = True
    End With
    With Sheets("Charts").ChartObjects.Add(410, 10, 400, 200)
        .Chart.SetSourceData Source:=Sheets("Statistics").Range("$W$1:$AC$21")
        .Chart.SeriesCollection(1).XValues = "='Statistics'!$A$2:$A$21"
        .Chart.ChartType = xlColumn 'xlArea
        .Chart.HasTitle = True
        .Chart.ChartStyle = 42
        .Chart.ChartTitle.Text = "DB Time and CPU"

        'Rotate the X axis titles
        .Chart.Axes(xlCategory).TickLabels.Orientation = xlUpward

        'Add the vertical Y axis title
        .Chart.Axes(xlValue, xlPrimary).HasTitle = True
        .Chart.Axes(xlValue, xlPrimary).AxisTitle.Characters.Text = "Seconds"

        'Add a gradient to the background of the chart
        .Chart.PlotArea.Fill.OneColorGradient Style:=msoGradientHorizontal, Variant:=2, Degree:=0.756847486076142
        .Chart.PlotArea.Fill.ForeColor.SchemeColor = 23
        .Chart.PlotArea.Fill.Visible = True
    End With

    With Sheets("Charts").ChartObjects.Add(10, 215, 400, 200)
        .Chart.SetSourceData Source:=Sheets("Statistics").Range("$S$1:$T$21")
        .Chart.SeriesCollection(1).XValues = "='Statistics'!$A$2:$A$21"
        .Chart.ChartType = xlColumnStacked 'xlAreaStacked
        .Chart.HasTitle = True
        .Chart.ChartStyle = 42
        .Chart.ChartTitle.Text = "Server-Wide CPU Usage"

        'Rotate the X axis titles
        .Chart.Axes(xlCategory).TickLabels.Orientation = xlUpward

        'Add the vertical Y axis title
        .Chart.Axes(xlValue, xlPrimary).HasTitle = True
        .Chart.Axes(xlValue, xlPrimary).AxisTitle.Characters.Text = "Seconds"

        'Add a gradient to the background of the chart
        .Chart.PlotArea.Fill.OneColorGradient Style:=msoGradientHorizontal, Variant:=2, Degree:=0.756847486076142
        .Chart.PlotArea.Fill.ForeColor.SchemeColor = 23
        .Chart.PlotArea.Fill.Visible = True
    End With
    With Sheets("Charts").ChartObjects.Add(410, 215, 400, 200)
        .Chart.SetSourceData Source:=Sheets("Statistics").Range("$O$1:$P$21")
        .Chart.SeriesCollection(1).XValues = "='Statistics'!$A$2:$A$21"
        .Chart.ChartType = xlAreaStacked100 'xlColumnStacked100 'xlAreaStacked100
        .Chart.HasTitle = True
        .Chart.ChartStyle = 42
        .Chart.ChartTitle.Text = "Server-Wide CPU Utilization"

        'Rotate the X axis titles
        .Chart.Axes(xlCategory).TickLabels.Orientation = xlUpward

        'Add the vertical Y axis title
        .Chart.Axes(xlValue, xlPrimary).HasTitle = True
        .Chart.Axes(xlValue, xlPrimary).AxisTitle.Characters.Text = "Percent"

        'Add a gradient to the background of the chart
        .Chart.PlotArea.Fill.OneColorGradient Style:=msoGradientHorizontal, Variant:=2, Degree:=0.756847486076142
        .Chart.PlotArea.Fill.ForeColor.SchemeColor = 23
        .Chart.PlotArea.Fill.Visible = True
    End With

Let’s also change a couple of default settings found on the UserForm and correct the (intentional) spelling error found in the titlebar of the UserForm by using a couple of lines of code directly below the above code:

    chkPauseRefresh.Value = True
    chkDisplaySessionDetail.Value = True
    chkExcludeIdleWaits.Value = True
    Me.Caption = "Charles Hooper's Time Model Viewer in Microsoft Excel"

Now that the code to generate the blank charts has been added to the UserForm, we need code to add the chart data to the Statistics worksheet.  If I so chose, rather than adding the data to the Statistics worksheet, I could simply build an array of numbers and use that array as the charts’ source data, however it might at times be helpful to see the raw data that is presented in the chart.  Locate the UpdateDisplay subroutine in the UserForm’s code (Public Sub UpdateDisplay).  In that subroutine, locate the following line:

tvTimeModel.Nodes.Clear

Just above that line, add the following code:

'Added in Article 6
    If dblDBTimeLast > 0 Then
        'Note that the first two methods cause the data source range for the charts to shift down 1 row, so copty-paste is used
        'Sheets("Statistics").Rows("2:2").Insert Shift:=xlDown, CopyOrigin:=xlFormatFromLeftOrAbove
        'Range("Statistics!A2:AC1000").Cut Destination:=Range("Statistics!A3:AC1001")
        Sheets("Statistics").Range("A2:AC1000").Copy
        Sheets("Statistics").Range("A3:AC1001").PasteSpecial xlPasteValuesAndNumberFormats
        Sheets("Statistics").Range("A2").Select
        Sheets("Statistics").Cells(2, 1) = Format(Now, "hh:nn am/pm")
        Sheets("Statistics").Cells(2, 2) = 0
        Sheets("Statistics").Cells(2, 3) = 0
        Sheets("Statistics").Cells(2, 4) = 0
        Sheets("Statistics").Cells(2, 5) = 0
        Sheets("Statistics").Cells(2, 6) = 0
        Sheets("Statistics").Cells(2, 7) = 0
        Sheets("Statistics").Cells(2, 8 ) = 0
        Sheets("Statistics").Cells(2, 9) = 0
        Sheets("Statistics").Cells(2, 10) = 0

        Sheets("Statistics").Cells(2, 15) = Val(lblBusyTime)
        Sheets("Statistics").Cells(2, 16) = Val(lblIdleTime)
        Sheets("Statistics").Cells(2, 17) = Val(lblBusyPercent)

        Sheets("Statistics").Cells(2, 19) = Val(lblUserMode)
        Sheets("Statistics").Cells(2, 20) = Val(lblKernelMode)
        Sheets("Statistics").Cells(2, 21) = Val(lblUserModePercent)

        Sheets("Statistics").Cells(2, 23) = Format((dblDBTime - dblDBTimeLast) / 1000000, "0.00")
        Sheets("Statistics").Cells(2, 24) = Format((dblDBCPU - dblDBCPULast) / 1000000, "0.00")
        Sheets("Statistics").Cells(2, 25) = Val(lblCPUUsedBySession)
        Sheets("Statistics").Cells(2, 26) = Val(lblParseTimeCPU)
        Sheets("Statistics").Cells(2, 27) = Val(lblRecursiveCPUUsage)
        Sheets("Statistics").Cells(2, 28) = Val(lblOtherCPU)
        Sheets("Statistics").Cells(2, 29) = Format((dblBackgroundCPU - dblBackgroundCPULast) / 1000000, "0.00")

        Sheets("Statistics").Range("B2:AC2").NumberFormat = "0.00"
    End If

Scroll down toward the end of the UpdateDisplay subroutine and locate the following line:

Sheets("Wait Events").Cells(intLastWaitClassRow, 2).Value = Format(sglWaitClassTime / 100, "0.00")

Immediately after the above line, add the following lines of code:

                    'Added in Article 6
                    'Add wait events to statistics worksheet
                    If dblDBTimeLast > 0 Then
                        Select Case UCase(strLastWaitClass)
                            Case "ADMINISTRATIVE"
                                Sheets("Statistics").Cells(2, 2) = Format(sglWaitClassTime / 100, "0.00")
                            Case "APPLICATION"
                                Sheets("Statistics").Cells(2, 3) = Format(sglWaitClassTime / 100, "0.00")
                            Case "COMMIT"
                                Sheets("Statistics").Cells(2, 4) = Format(sglWaitClassTime / 100, "0.00")
                            Case "CONCURRENCY"
                                Sheets("Statistics").Cells(2, 5) = Format(sglWaitClassTime / 100, "0.00")
                            Case "CONFIGURATION"
                                Sheets("Statistics").Cells(2, 6) = Format(sglWaitClassTime / 100, "0.00")
                            Case "NETWORK"
                                Sheets("Statistics").Cells(2, 7) = Format(sglWaitClassTime / 100, "0.00")
                            Case "OTHER"
                                Sheets("Statistics").Cells(2, 8 ) = Format(sglWaitClassTime / 100, "0.00")
                            Case "SYSTEM I/O"
                                Sheets("Statistics").Cells(2, 9) = Format(sglWaitClassTime / 100, "0.00")
                            Case "USER I/O"
                                Sheets("Statistics").Cells(2, 10) = Format(sglWaitClassTime / 100, "0.00")
                        End Select
                    End If

Run the frmTimeModel UserForm by pressing the F5 key on the keyboard.  If the code is working correctly, the Wait Events worksheet should update just as it had in the past:

You should also find that the Statistics worksheet now shows running delta values of the various statistics, with the most recent delta values on the second row of the worksheet:

One of the advantages of using Excel for the charts is that the charts automatically update as new data is added to the Statistics worksheet.  Unfortunately, the data series range for the chart is also auto-modified every time a new row is inserted into the Statistics worksheet, such that the charts never actually show any information.  To avoid this situation, the above code does not perform a row insert, rather it copies the existing data and pastes that data one row down in the worksheet.

The generated Charts worksheet should contain four charts, as shown below:

The chart formatting shown above is quite fancy – so fancy that it requires Microsoft Excel 2007 or later.  The chart creation code may be altered to create the typical flat single color chart elements found in Excel 2003 and earlier.

Are we done yet?  Your Excel worksheet contents are probably flickering quite a bit as additional data is added to the various worksheets.  To correct that problem, switch back to the window that allows seeing the source code for the UserForm and again locate the UpdateDisplay subroutine.  Locate the following line of code:

On Error Resume Next

Just above that line of code, add the following, which will tell Excel not to try updating the worksheet contents as displayed on screen until ScreenUpdating is re-enabled:

Application.ScreenUpdating = False

Scroll down to the last line of the UpdateDisplay subroutine.  Immediately after the last line (intActivated = True), add the following line:

Application.ScreenUpdating = True

——————–

Are we done yet?  Part 7 of this blog article series is still a very rough sketch.  Any ideas for improvement?

Added August 19, 2015:

The Excel project code to this point, save with a .XLS extension (currently has a .DOC extension, please change):

TimeModelViewerExcelArticle6.xls

Give Me a Hint – How were These Autotrace Execution Statistics Achieved?

June 27, 2011

I recently received an email asking why different performance is achieved when a FIRST_ROWS hint, FIRST_ROWS(100) hint, and an unhinted version of the query are executed.  This seems to be a simple problem, yet it might also be an interesting problem.  I thought that it might be helpful to transform my response into a blog article (allowing the input of readers of this blog), rather than provide an email response (sorry, I typically do not respond to email requests).  What if (flushing the buffer cache between executions with a single execution of ALTER SYSTEM FLUSH BUFFER_CACHE;):

• The unhinted version of the query completes 2 seconds faster than the FIRST_ROWS(100) hinted version of the query?
• The FIRST_ROWS hinted version of the query completes 22 seconds faster than the unhinted version of the query (thus the FIRST_ROWS hinted version completes 24 seconds faster than then FIRST_ROWS(100) hinted version of the query).
• According to the AUTOTRACE output, the execution plans for the FIRST_ROWS hinted version and the FIRST_ROWS(100) hinted version of the query both perform a descending index range scan on the primary key column for the one table (the only data source) in the query.
• According to the AUTOTRACE output, the unhinted version of the query shows that a full table scan was performed.
• While the FIRST_ROWS(100) hinted execution required longer to complete than the FIRST_ROWS hinted execution, the FIRST_ROWS(100) hinted execution performed 69% as many physical block reads as the FIRST_ROWS hinted execution, while exactly the same number of consistent gets were performed in both cases.
• The queries retrieved 1,042,684 rows with an array fetch size of 15.
• For the FIRST_ROWS(100) and FIRST_ROWS hinted versions of the query, roughly 263.97MB was returned to the client, while only about 163.62MB (62% of the hinted version) was returned in the unhinted version of the query.

The question is how would one explain the results?  Here is a slightly modified version of the query, with just the table and column names replaced:

SELECT
  C1,
  C2,
  C3,
  C4,
  C5,
  C6,
  C7
FROM
  T1      
WHERE
  C1 < 1042685
ORDER BY
  C1 DESC; 

In the above, note that C1 must be the primary key column… unless of course there are multiple columns in the primary key (or the index displayed in the execution plans really is not the primary key index).

The order of the query executions was not specified, but let’s assume that the OP listed the executions in the same order as executed.

The unhinted execution:

Elapsed: 00:00:53.00

----------------------------------------------------------------------------------------
| Id  | Operation          | Name      | Rows  | Bytes |TempSpc| Cost (%CPU)| Time
----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |           |  1042K|   255M|       |   120K  (1)| 00:24:03 |
|   1 |  SORT ORDER BY     |           |  1042K|   255M|   313M|   120K  (1)| 00:24:03 |
|*  2 |   TABLE ACCESS FULL| T1        |  1042K|   255M|       | 62490   (1)| 00:12:30 |
---------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - filter("C1"<1042685)

Statistics
----------------------------------------------------------
          1  recursive calls
          1  db block gets
     244502  consistent gets
     244458  physical reads
       2836  redo size
  171565915  bytes sent via SQL*Net to client
     765155  bytes received via SQL*Net from client
      69514  SQL*Net roundtrips to/from client
          1  sorts (memory)
          0  sorts (disk)
    1042684  rows processed 

The FIRST_ROWS hinted execution:

Elapsed: 00:00:31.11

---------------------------------------------------------------------------------------------
| Id  | Operation                    | Name         | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |              |  1042K|   255M|   189K  (1)| 00:37:51 |
|   1 |  TABLE ACCESS BY INDEX ROWID | T1           |  1042K|   255M|   189K  (1)| 00:37:51 |
|*  2 |   INDEX RANGE SCAN DESCENDING| PK_T1        |  1042K|       |  2136   (1)| 00:00:26 |
---------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C1"<1042685)

Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
     345027  consistent gets
      38750  physical reads
          0  redo size
  276788545  bytes sent via SQL*Net to client
     765156  bytes received via SQL*Net from client
      69514  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
    1042684  rows processed

The FIRST_ROWS(100) hinted execution:

Elapsed: 00:00:55.69

Execution Plan
----------------------------------------------------------
Plan hash value: 2367722674
---------------------------------------------------------------------------------------------
| Id  | Operation                    | Name         | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |              |   101 | 25957 |    22   (0)| 00:00:01 |
|   1 |  TABLE ACCESS BY INDEX ROWID | T1           |  1042K|   255M|    22   (0)| 00:00:01 |
|*  2 |   INDEX RANGE SCAN DESCENDING| PK_T1        |   101 |       |     3   (0)| 00:00:01 |
---------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("C1"<1042685)

Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
     345027  consistent gets
      26788  physical reads
          0  redo size
  276788545  bytes sent via SQL*Net to client
     765156  bytes received via SQL*Net from client
      69514  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
    1042684  rows processed 

There are a number of items that we do not know about the OP’s configuration.  Just a couple of random thoughts:

• How might we explain why only 62% as many bytes were transferred over the network in the unhinted version of the query?  Could it be a case where SQL*Net compression has helped to strip the repeating column values from adjacent rows in the result set prior to transmission over the network.  Could it be the case that the rows were ordered differently if column C1 were the only column in the primary key?  What if there were multiple columns in the primary key – could that make a difference (note that the C1 < 1042685 predicate in the WHERE clause combined with the 1,042,684 rows returned suggests that column C1 might be the only column in the primary key)?  Does SQL*Net compression not work when rows are retrieved by an index access path to avoid a sort operation?  This portion of the result is a bit interesting.
• How might we explain the number 2836 next to redo size in the unhinted execution?  Might this be a case of Delayed Block Cleanout?  Could that number also partially explain the execution time for that version of the query?
• How might we explain the decrease in the number of physical block reads in the FIRST_ROWS(100) hinted execution compared to the FIRST_ROWS hinted execution?  Assuming that the executions were performed in the above order, what if the ALTER SYSTEM FLUSH BUFFER_CACHE; command did not actually flush all of the blocks from the buffer cache – is that possible?  Could undo blocks for a consistent read have an impact?
• The operations displayed in the execution plans for the two hinted queries are identical, yet the execution times are significantly different.  How might we explain the time difference?
• — Do we know for certain that the runtime execution engine used the execution plan that was displayed?
• — Is it possible that one of the queries was executed with SET AUTOTRACE ON specified in SQL*Plus while the fast execution was performed with SET AUTOTRACE ON STATISTICS specified in SQL*Plus (thus entirely avoiding the client-side formatting of the output)?
• — Is it possible that SET AUTOTRACE ON was specified in both cases, yet the SQL*Plus window was minimized during the FIRST_ROWS hinted execution (thus avoiding the client-side screen refreshes)?
• — Is it possible that there was less network congestion during the FIRST_ROWS hinted execution?
• — Is it possible that the client computer’s CPU (or the server’s CPU) was less busy with other tasks during the FIRST_ROWS hinted execution?
• — Is it possible that operating system caching is having an effect – if the entire execution time was spent in single block read wait events, the average single block read time for the FIRST_ROWS hinted execution was roughly 0.00080 seconds, while the average single block read time for the FIRST_ROWS(100) hinted execution was roughly 0.00208 seconds.
• — Note that the AUTOTRACE output indicates that one recursive call was performed per execution – is it possible that the time difference and the difference in the number of physical block reads could be attributed to the recursive call?

What other items might explain the time difference in the three executions?  What advice would you provide to the OP?  Should we answer the original question using 10 more questions?