Having been tasked to make our test, acceptance and production instances behave the same way when running the same SQL, I find myself fighting this same battle over and over again, when really it should be a lot easier. Constant source of headaches, as is.

What’s worse: every new release/patch level introduces the potential for different behaviour. Exhaustively testing all that and preventing it turns out to be expensive, if one considers the perspective of the software developer as pointed out by Charles. Any wonder why sometimes those developers take weird decisions in terms of db design and SQL usage? I can hear their pain…

Charles pointed out some of the very important CBO behaviour differences with seemingly harmless changes in unrelated parameters. Hidden features like the “behind the scenes parallelism” we both mentioned add to this problem: there is no way to be sure we’re not seeing a side effect of one of those… I’m abstracting the side effects of any bu^H^Hfeatures that might creep in, of course!

I suppose it makes the job interesting, but really I’d prefer to have it simplified!

And no: OEM and grid’s performance pack won’t resolve this problem, it doesn’t even begin to understand it! (just pre-empting the strike before someone else jumps in with that furphy!)

SQL> SELECT /* FIND_ME */
  2    T1.C4
  3  FROM
  4    T1
  5  WHERE
  6    T1.C2='MONDAY'
  7    AND T1.C3 BETWEEN 200 AND 255;

10952 rows selected.


Statistics
---------------------------------------------------
          1  recursive calls
          0  db block gets
       6884  consistent gets
       6876  physical reads
          0  redo size
     103802  bytes sent via SQL*Net to client
        470  bytes received via SQL*Net from client
         12  SQL*Net roundtrips to/from client
          1  sorts (memory)
          0  sorts (disk)
      10952  rows processed

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.01       0.01          0          0          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch       12      0.24       0.55       6876       6884          0       10952
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total       14      0.26       0.56       6876       6884          0       10952
 
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 286  
Number of plan statistics captured: 1
 
Rows (1st) Rows (avg) Rows (max)  Row Source Operation
---------- ---------- ----------  ---------------------------------------------------
     10952      10952      10952  TABLE ACCESS BY INDEX ROWID T1 (cr=6884 pr=6876 pw=0 time=20871 us cost=2973 size=234234 card=11154)
     10952      10952      10952   BITMAP CONVERSION TO ROWIDS (cr=346 pr=345 pw=0 time=2963 us)
         2          2          2    BITMAP AND  (cr=346 pr=345 pw=0 time=24470 us)
        11         11         11     BITMAP CONVERSION FROM ROWIDS (cr=183 pr=182 pw=0 time=57010 us)
     71428      71428      71428      INDEX RANGE SCAN IND_T1_C2 (cr=183 pr=182 pw=0 time=70788 us cost=194 size=0 card=0)(object id 82006)
         3          3          3     BITMAP CONVERSION FROM ROWIDS (cr=163 pr=163 pw=0 time=10584 us)
     76664      76664      76664      SORT ORDER BY (cr=163 pr=163 pw=0 time=12414 us)
     76664      76664      76664       INDEX RANGE SCAN IND_T1_C3 (cr=163 pr=163 pw=0 time=19709 us cost=164 size=0 card=0)(object id 82007)
 
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  SQL*Net message to client                      12        0.00          0.00
  db file sequential read                       493        0.00          0.08
  SQL*Net message from client                    12        0.00          0.00
  db file parallel read                         274        0.00          0.30
  SQL*Net more data to client                    11        0.00          0.00

SQL> SELECT /* FIND_ME */
  2    T1.C4,
  3    T2.C4
  4  FROM
  5    T1,
  6    T2
  7  WHERE
  8    T1.C2='MONDAY'
  9    AND T1.C3 BETWEEN 200 AND 255
 10    AND T1.C1=T2.C1
 11    AND T1.C3=T2.C3;

9484744 rows selected.


Statistics
---------------------------------------------------
          1  recursive calls
          0  db block gets
      54125  consistent gets
      45339  physical reads
          0  redo size
  124731018  bytes sent via SQL*Net to client
     104684  bytes received via SQL*Net from client
       9486  SQL*Net roundtrips to/from client
          1  sorts (memory)
          0  sorts (disk)
    9484744  rows processed

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.03       0.01          0          0          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch     9486      4.16       4.92      45339      54125          0     9484744
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total     9488      4.19       4.94      45339      54125          0     9484744
 
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 286  
Number of plan statistics captured: 1
 
Rows (1st) Rows (avg) Rows (max)  Row Source Operation
---------- ---------- ----------  ---------------------------------------------------
   9484744    9484744    9484744  HASH JOIN  (cr=54125 pr=45339 pw=0 time=4969125 us cost=5999 size=2994849 card=76791)
     10952      10952      10952   TABLE ACCESS BY INDEX ROWID T1 (cr=6876 pr=6876 pw=0 time=1043019 us cost=2973 size=267696 card=11154)
     10952      10952      10952    BITMAP CONVERSION TO ROWIDS (cr=345 pr=345 pw=0 time=4456 us)
         2          2          2     BITMAP AND  (cr=345 pr=345 pw=0 time=24642 us)
        11         11         11      BITMAP CONVERSION FROM ROWIDS (cr=182 pr=182 pw=0 time=53970 us)
     71428      71428      71428       INDEX RANGE SCAN IND_T1_C2 (cr=182 pr=182 pw=0 time=68743 us cost=194 size=0 card=0)(object id 82006)
         3          3          3      BITMAP CONVERSION FROM ROWIDS (cr=163 pr=163 pw=0 time=11190 us)
     76664      76664      76664       SORT ORDER BY (cr=163 pr=163 pw=0 time=10622 us)
     76664      76664      76664        INDEX RANGE SCAN IND_T1_C3 (cr=163 pr=163 pw=0 time=22013 us cost=164 size=0 card=0)(object id 82007)
     76664      76664      76664   TABLE ACCESS FULL T2 (cr=47249 pr=38463 pw=0 time=3398812 us cost=3025 size=1171125 card=78075)
 
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  SQL*Net message to client                    9486        0.00          0.00
  db file sequential read                       620        0.00          0.15
  db file parallel read                         274        0.00          0.31
  db file scattered read                        316        0.00          0.15
  SQL*Net message from client                  9486        0.06         10.45
  SQL*Net more data to client                  9485        0.00          0.34

Now I am starting to wonder if Oracle Corp. released a buggy version of 11.2.0.1 that can’t count physical block reads correctly, and a less buggy version that can count physical block reads and also adds the “Rows (1st) Rows (avg) Rows (max)” columns in the TKPROF output?

I said it and I agree! BTree/Bitmap conversion is kinda interesting to me, since you can still set the parameter in SE even in 11.2.0.1

SYS @ SE >@show_parameter _b_tree_bitmap_plans
old   4: AND kSPPINM='&1'
new   4: AND kSPPINM='_b_tree_bitmap_plans'

NAME                                                             VALUE
---------------------------------------------------------------- ------------------------------
_b_tree_bitmap_plans                                             TRUE

SYS @ SE >

but it doesn’t seem to be honoured (perhaps rightly, perhaps not).

I like your suggestion re: indicating which features have been considered. Perhaps externalizing the presumably existing STRUCT(s?) behind optimizer_features_enable would be useful. I’m pretty sure that even back to 8i there were the odd partitioned objects used by SYS even though constructing them was not available in the product.

I suspect that there is still a lot that I have not covered yet regarding the differences between Standard Edition and Enterprise Edition – probably plenty that I am not aware of yet. I look forward to reading your blog article series on the topic. I will have to take a closer look at the 11.2.0.1 Standard Edition results – did you use the same workload system statistics that I listed at the start of the article, or did you use other settings that are more accurate for your server environment? It took close to 5 hours to put together this blog article, even though it probably looks like 20 minutes of effort (writing the darn test script took too long, and I ended up deleting the last half of the test script because the results were too similar to the first half).

Just to spin things a bit, I repeated the test on my laptop, which has the Enterprise Edition of 11.2.0.1. (For some reason, TKPROF in 11.2.0.1 has now decided to start including the “Rows (1st) Rows (avg) Rows (max)” columns in the output – previously I could only see those columns when processing a trace file with TKPROF from 11.2.0.2.) I did not change the workload system (CPU) statistics, so the 10053 portion of the trace file showed those values as:

-----------------------------
SYSTEM STATISTICS INFORMATION
-----------------------------
  Using WORKLOAD Stats
  CPUSPEED: 2714 millions instructions/sec
  SREADTIM: 0.100000 milliseconds
  MREADTIM: 0.200000 millisecons
  MBRC: 78 blocks
  MAXTHR: 302080 bytes/sec
  SLAVETHR: -1 bytes/sec

Also, it appears that DB_FILE_MULTIBLOCK_READ_COUNT was forced to a value of 128, while it was autoset to 128 in the other databases. And rather than having the SGA_TARGET set to 8000M and the PGA_AGGREGATE_TARGET set 1800M, those parameters were set to 12000M and 2000M, respectively.

Now, what do you think happened to the statistics displayed in TKPROF and the execution plans?

The SQL*Plus output for the first SQL statement:

SQL> SELECT /* FIND_ME */
  2    T1.C4
  3  FROM
  4    T1
  5  WHERE
  6    T1.C2='MONDAY'
  7    AND T1.C3 BETWEEN 200 AND 255;
 
10952 rows selected.
 
Statistics
---------------------------------------------------
          1  recursive calls
          0  db block gets
      38485  consistent gets
      38463  physical reads
          0  redo size
     103802  bytes sent via SQL*Net to client
        469  bytes received via SQL*Net from client
         12  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
      10952  rows processed

The TKPROF output for the first SQL statement:

SELECT /* FIND_ME */
  T1.C4
FROM
  T1
WHERE
  T1.C2='MONDAY'
  AND T1.C3 BETWEEN 200 AND 255
 
call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.01       0.01          0          0          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch       12      0.21       0.55      38463      38485          0       10952
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total       14      0.23       0.57      38463      38485          0       10952
 
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 286  
Number of plan statistics captured: 1
 
Rows (1st) Rows (avg) Rows (max)  Row Source Operation
---------- ---------- ----------  ---------------------------------------------------
     10952      10952      10952  TABLE ACCESS FULL T1 (cr=38485 pr=38463 pw=0 time=3880132 us cost=2459 size=234234 card=11154)
 
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  SQL*Net message to client                      12        0.00          0.00
  db file sequential read                         1        0.00          0.00
  db file scattered read                        316        0.00          0.40
  SQL*Net message from client                    12        0.00          0.00
  SQL*Net more data to client                    11        0.00          0.00

The SQL*Plus output for the second SQL statement:

SQL> SELECT /* FIND_ME */
  2    T1.C4,
  3    T2.C4
  4  FROM
  5    T1,
  6    T2
  7  WHERE
  8    T1.C2='MONDAY'
  9    AND T1.C3 BETWEEN 200 AND 255
 10    AND T1.C1=T2.C1
 11    AND T1.C3=T2.C3;
 
9484744 rows selected.
 
Statistics
---------------------------------------------------
          1  recursive calls
          0  db block gets
      85724  consistent gets
      76926  physical reads
          0  redo size
  124731018  bytes sent via SQL*Net to client
     104683  bytes received via SQL*Net from client
       9486  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
    9484744  rows processed

The TKPROF output for the second SQL statement:

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.03       0.01          0          0          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch     9486      3.99       4.87      76926      85724          0     9484744
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total     9488      4.02       4.89      76926      85724          0     9484744
  
Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 286  
Number of plan statistics captured: 1
 
Rows (1st) Rows (avg) Rows (max)  Row Source Operation
---------- ---------- ----------  ---------------------------------------------------
   9484744    9484744    9484744  HASH JOIN  (cr=85724 pr=76926 pw=0 time=4592034 us cost=4982 size=2994849 card=76791)
     10952      10952      10952   TABLE ACCESS FULL T1 (cr=38475 pr=38463 pw=0 time=3799997 us cost=2459 size=267696 card=11154)
     76664      76664      76664   TABLE ACCESS FULL T2 (cr=47249 pr=38463 pw=0 time=2836531 us cost=2487 size=1171125 card=78075)
 
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  SQL*Net message to client                    9486        0.00          0.00
  db file sequential read                         2        0.00          0.00
  db file scattered read                        632        0.00          0.66
  SQL*Net message from client                  9486        0.06         10.27
  SQL*Net more data to client                  9485        0.00          0.35
  asynch descriptor resize                        1        0.00          0.00

Now, compare the above Row Source Operation execution plans with those of Standard Edition in the article – the Row Source Operation execution plans are quite similar to what we saw with Standard Edition, but the wait event might be a bit different. Note also the “asynch descriptor resize” wait event – I wonder what that means?

If I have learned anything from this exercise it is that there are plenty of opportunities to see different performance in development, test, and production databases when the database edition does not match, and when the workload system (CPU) statistics do not match. Scary? Yes, especially for application development companies that develop database applications using an entirely different database configuration than their customers.

SE does perform differently than EE for a variety of reasons, mostly related to features available in EE but not SE, these aren’t limited to parallelism and partitioning and the effects can be subtle. The interesting thing with Charles’ test is how suitable many of his queries are (regarded as being by the cbo) for b_tree_bitmap_plans. These are not available in SE (though I definitely seem to recall seeing them – and disabling them – in either 9.0 or 9.2 SE, don’t have either of those available to test anymore). In general you can expect these plans to occur when you get either an ‘and’ or an ‘or’ predicate on two columns which are each indexed and the indexes are somewhat selective.

Charles,
My you are pumping out these tests fast, and stealing some of my thunder from that email conversation , I was working through a similar line of enquiry, but the day job and next weeks presentation are getting in the way I am a little bit suprised that you didn’t get a CONCATENATION operation in the index join case in SE – that’s what I got in 11.2.0.1 from your first test. It would be interesting (to me) to see the 10053 traces and cost calculations.

Niall

But this demonstrates that it’s not just the creation of a bitmap index – which is what you might initially thing – it’s also internal operations like the old btree bitmap conversion.

This operation should also be familiar as a potential troublemaker (not always) and so it being available for consideration by the optimizer in EE is not always a good thing.