No doubt when buffer cache is busy, direct path reads (DPR) will be faster than conventional path reads (CPR). As suggested, I tried that as well. Exactly the same code on identical tables on different schemas. I started running the query in second schema and then I kicked off the query in the main schema. With DPR disabled, CPR results were slower compared to doing exactly the same concurrent run tests with DPR.

My current argument is that the optimiser costing will alway favour DPR to CPR when a table scan of > 10% of buffer cache is involved. In other words it does not matter how much free buffers are there (case a single session running nothing else), the run time engine will perform DPR and will always put the table in the LRU end of the chain to be discarded immediately.

So in summary with DPR, any table > 10% of buffer cache will be fetch and discard and will not be promoted to hot area.

I trust that this makes sense.

Cheers,

Mich

I think that this is the OTN thread where we discussed the topic of serial direct path reads (direct path read waits) versus conventional, buffered reads (db file scattered read waits) for table scans:
https://forums.oracle.com/forums/message.jspa?messageID=10077663

My opening paragraph in that thread stated:

… Keep in mind that the numbers that you are seeing are slightly distorted due to the fact that the buffer cache was empty when the test started. To understand how this may affect the test results, see the first comment on this page: http://dioncho.wordpress.com/2009/04/22/strong-doubt-on-small-table_small_table_threshold/

In short, you saw the significant 20% performance improvement with db file scattered reads because the buffer cache was empty at the start of the test, which is unlikely in a production environment. If you want to test this theory:

CREATE TABLE TDASH2 AS SELECT * FROM TDASH;

Bounce the database, execute a modified version of your script:

DECLARE
        type array is table of tdash%ROWTYPE index by binary_integer;
        l_data array;
        l_rec tdash%rowtype;
BEGIN
        SELECT
                a.*
                ,RPAD('*',4000,'*') AS PADDING1
                ,RPAD('*',4000,'*') AS PADDING2
        BULK COLLECT INTO
        l_data
        FROM ALL_OBJECTS a;

        FOR rs IN 1 .. 100
        LOOP
                BEGIN
                        SELECT * INTO l_rec FROM TDASH2 WHERE object_id = l_data(rs).object_id;
                EXCEPTION
                  WHEN NO_DATA_FOUND THEN NULL;
                END;
        END LOOP;
END;

Finally, execute your script with and without serial direct path reads enabled. You should now see that the buffer cache had less of an opportunity to cache frequently accessed blocks read during the full table scan of your tdash table, and you should find that the serial direct path read now not only uses less CPU, but also requires less elapsed time for its execution than the conventional buffered reads.

Jonathan, as usual, has provided excellent information in the comment section of this blog article, and on the OTN thread, and has mentioned items that I had not considered. For example, I did not consider that your cacher table example would result in chained rows (out of curiosity, I might test this just to see if I am able to identify the chained rows – I am assuming that Jonathan noticed that counting 67,500 rows required 136,006 consistent gets during the COUNT(*) FROM CACHER;).

If you recall a while back we discussed direct path reads vs conventional reads in OTN. The query was pretty simple:

DECLARE
        type array is table of tdash%ROWTYPE index by binary_integer;
        l_data array;
        l_rec tdash%rowtype;
BEGIN
        SELECT
                a.*
                ,RPAD('*',4000,'*') AS PADDING1
                ,RPAD('*',4000,'*') AS PADDING2
        BULK COLLECT INTO
        l_data
        FROM ALL_OBJECTS a;
 
        DBMS_MONITOR.SESSION_TRACE_ENABLE ( waits=>true );
        FOR rs IN 1 .. 100
        LOOP
                BEGIN
                        SELECT * INTO l_rec FROM tdash WHERE object_id = l_data(rs).object_id;
                EXCEPTION
                  WHEN NO_DATA_FOUND THEN NULL;
                END;
        END LOOP;
END;

The base table tdash had 1.7 million rows and I tried using direct path read (default) and db scattered file read. with conventional path it was 20% faster.
At that time the argument was that the run time engine preferred direct path read to save CPU usage in multi-user environment. Fair enough. My test was every time after reboot, one run only. However, we are looping over table 100 times fatching rows matching object_id. tkprof output showed:

for direct path read:

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          7         47          0           0
Execute    100      0.00       0.00          2         51          0           0
Fetch      100     10.52    6358.83  194142802  194831012          0         100
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total      201     10.53    6358.84  194142811  194831110          0         100

Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 96  (SSDTESTER)   (recursive depth: 1)

Rows     Row Source Operation
-------  ---------------------------------------------------
      1  TABLE ACCESS FULL TDASH (cr=1948310 pr=1941430 pw=0 time=0 us cost=526908 size=8091 card=1)

There was full table scan for every read in the loop. With conventional path we had

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute    100      0.02       0.00          0          0          0           0
Fetch      100    150.74    5103.34  131458857  194490912          0         100
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total      201    150.76    5103.34  131458857  194490912          0         100

So we seems to have 194142811 – 131458857 = 62683954 or around 32% less physical reads. Table tdash had 1,729,204 rows in 2,709,153 blocks and Table size as percent of buffer size was 123%. So with more caching and less waits that 20% performance gain may be explained now?

Cheers,

Mich

Many thanks for your input. To test the case again I decided to create the cacher table with PCTFREE = 0 as follows:

create table cacher(c1 char(2000), c2 char(2000), c3 char(2000), c4 char(2000))PCTFREE 0
nologging;
insert /*+ append */ into cacher
select
          RPAD('*',2000)
         ,RPAD('*',2000)
         ,RPAD('*',2000)
         ,RPAD('*',2000)
from dual
connect by level<=1350000;
exec dbms_stats.gather_table_stats(ownname=>user,tabname=>'CACHER',estimate_percent=>100);
commit;
alter system flush buffer_cache;
EXEC DBMS_MONITOR.SESSION_TRACE_ENABLE ( waits=>true, plan_stat=>'ALL_EXECUTIONS' );
ALTER SESSION SET EVENTS '10949 trace name context forever, level 1'  -- No Direct path read ;
select count(*) from cacher;
select count(*) from cacher;
ALTER SESSION SET TRACEFILE_IDENTIFIER = 'catcher';
EXEC DBMS_MONITOR.SESSION_TRACE_ENABLE ( waits=>true, plan_stat=>'ALL_EXECUTIONS' );
select count(*) from cacher;

With that in mind In the case above we have a table with the following properties. I also added the simple ratios of buffer cache to block size to give an indication of how many blocks fit in to this cache

Parameter                      buffer cache size/MB blocks fit in this cache
------------------------------ -------------------- ------------------------
buffer_cache                                 11,072                1,417,216


Small table threshold at 2% of buffer cache size/MB Small table block limit
--------------------------------------------------- -----------------------
                                                221                  28,344

name                                     value                isdefault            description
---------------------------------------- -------------------- -------------------- -------------------------------------------------------
_small_table_threshold                   27295                TRUE                 lower threshold level of table size for direct reads

My table details

OWNER                            TABLE_NAME                               rows block size/KB       blocks avg free space/KB Table size/MB
-------------------------------- -------------------------------- ------------ ------------- ------------ ----------------- -------------
SSDTESTER                        CACHER                              1,350,000             8    1,351,819                 0        10,552


Table size as percent of buffer size
------------------------------------
                               95.30

So almost one block per row and no free space. Tkprof reports shows:

select count(*)
from
 cacher


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch        2      1.31       1.32          2    1350039          0           1
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        4      1.31       1.32          2    1350039          0           1

Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 96

Rows     Row Source Operation
-------  ---------------------------------------------------
      1  SORT AGGREGATE (cr=1350039 pr=2 pw=0 time=0 us)
1350000   TABLE ACCESS FULL CACHER (cr=1350039 pr=2 pw=0 time=2925915 us cost=366427 size=0 card=1350000)

So there we go 135,0039 CR with 2 physical reads for a table that is around 95% of the buffer cache.

Obviously what really matters is the default optimizer behaviour when choosing ‘direct path read’. Let us what it shows now:

With direct path read the magic percentage of ~ 10% holds. Let us look at the below case. This table at 9.2% of buffer size in. Anything above is recycled

My table details

OWNER                            TABLE_NAME                               rows block size/KB       blocks avg free space/KB Table size/MB
-------------------------------- -------------------------------- ------------ ------------- ------------ ----------------- -------------
SSDTESTER                        CACHER                                130,000             8      130,612                 0         1,016


Table size as percent of buffer size
------------------------------------
                                9.18

Table block size/threshold limit 
------------------------------- 
                              5 



select count(*)
from
 cacher


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch        2      0.11       0.11          0     130022          0           1
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        4      0.11       0.11          0     130022          0           1

Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 96

Rows     Row Source Operation
-------  ---------------------------------------------------
      1  SORT AGGREGATE (cr=130022 pr=0 pw=0 time=0 us)
 130000   TABLE ACCESS FULL CACHER (cr=130022 pr=0 pw=0 time=252448 us cost=35406 size=0 card=130000)

I believe now that you assertions about 10% boundary below which repeating a scan WILL increment the touch count holds as per optimizer use of direct path reads. For conventional path reads if table fits into the cache then Oracle allows that.

In a way it perhaps explains why Oracle prefers direct path reads to conventional reads for FTS. I think because Oracle has more control in place on the use of PGA than SGA.. It also shows that threshold limit of 2% does not mean much.

Regards,

Mich

An important point to consider in this test case is that every row in your table is chained (row length is more than 8,000, blocksize is 8KB, pctfree seems to be at the default of 10, means you have about 7,200 bytes per block – and then you can see that your table of 700,000 rows has used 1.4M blocks).

You may have discovered yet another type of boundary case for full tablescans, direct path reads, and caching, but you’re not looking at the general case. You’re query is simply select count(*), so Oracle shouldn’t have to follow the chain for each row, but it’s possible (and please take this as a flight of fancy at this point) that when Oracle reads a block which holds nothing but the target of a chained, it doesn’t count it is a “proper” read, and therefore manages to exclude it from the more generic arithmetic.

I tried to do find out simply at what table size (as percent of buffer size), Oracle decides to cache table blocks. The approach I used was based on what Kyle Hailey devised. First

Let us get the environment: Oracle version installed is 11.2.0.1.0 -64 bit, block size is 8K. buffer_cache = 11,072 MB

Small table threshold at 2% of buffer cache size/MB Small table block limit
--------------------------------------------------- -----------------------
                                                221                  28,344
 
hidden parameter _small_table_threshold

name                                     value (blocks)               
---------------------------------------- -------------------
_small_table_threshold                   27295     

To force Oracle to choose the conventional path I disabled direct path read first

drop table cacher;
create table cacher(c1 char(2000), c2 char(2000), c3 char(2000), c4 char(2000)) nologging;
insert /*+ append */ into cacher
select
          RPAD('*',2000)
         ,RPAD('*',2000)
         ,RPAD('*',2000)
         ,RPAD('*',2000)
from dual
connect by level user,tabname=>'CACHER',estimate_percent=>100);
commit;
alter system flush buffer_cache;
ALTER SESSION SET EVENTS '10949 trace name context forever, level 1'  -- No Direct path read ;
select count(*) from cacher;
select count(*) from cacher;
set autot on stat
 select count(*) from cacher;
 set autot off;

When I run the above I get:

  COUNT(*)
----------
    700000
 
Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
    1400040  consistent gets
     224581  physical reads
          1  rows processed

OK so we are just above the threshold in here. At this stage table cacher details is

OWNER                            TABLE_NAME                               rows block size/KB       blocks avg free space/KB Table size/MB
-------------------------------- -------------------------------- ------------ ------------- ------------ ----------------- -------------
SSDTESTER                        CACHER                                700,000             8    1,401,868                 0         5,471


Table size as percent of buffer size
------------------------------------
                               49.42

So at almost 50% we are seeing some discard activity. If I relax the size of table to be ~45% of buffer size the table is fully cached.

 
 COUNT(*)
----------
    650000


Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
    1300038  consistent gets
          0  physical reads
 
OWNER                            TABLE_NAME                               rows block size/KB       blocks avg free space/KB Table size/MB
-------------------------------- -------------------------------- ------------ ------------- ------------ ----------------- -------------
SSDTESTER                        CACHER                                650,000             8    1,301,762                 0         5,081


Table size as percent of buffer size
------------------------------------
                               45.89

So with conventional path and FTS when table size is roughly around 45% we have fully caching and block promotion to hot area

Now if we leave the optimiser to choose the default direct path read with FTS, we notice that up around 4.8% of buffer size, the table is fully cached.

Example:

  COUNT(*)
----------
     67500


Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
     135022  consistent gets
          0  physical reads
 
  OWNER                            TABLE_NAME                               rows block size/KB       blocks avg free space/KB Table size/MB
-------------------------------- -------------------------------- ------------ ------------- ------------ ----------------- -------------
SSDTESTER                        CACHER                                 67,500             8      135,620                 0           528


Table size as percent of buffer size
------------------------------------
                                4.77

Above this figure, the table blocks are put at LRU end and discarded. At around 4.8% of buffer size the table is read in and discarded every time. Tkprof output shows this.

select count(*)
from
 cacher


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch        2      0.00       4.02     136000     136006          0           1
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        4      0.00       4.02     136000     136006          0           1

Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 96

Rows     Row Source Operation
-------  ---------------------------------------------------
      1  SORT AGGREGATE (cr=136006 pr=136000 pw=0 time=0 us)
  68000   TABLE ACCESS FULL CACHER (cr=136006 pr=136000 pw=0 time=179583 us cost=37034 size=0 card=68000)


Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  SQL*Net message to client                       2        0.00          0.00
  direct path read                             1073        0.00          3.79
  SQL*Net message from client                     2        0.00          0.00

So in summary as per default optimiser behaviour ‘direct path read’,the expectation is that any table below 5% of buffer cache is promoted to hot area. When the table size is larger than this figure blocks with full table scans go to LRU end and are fully discarded.

Regards,

Mich

 drop table cacher;
  create table cacher(c1 char(2000), c2 char(2000), c3 char(2000)) nologging;
  insert /*+ append */ into cacher
    select 'x', 'x', 'x'
    from dual
       connect by level <= 220000     ;
  EXEC DBMS_STATS.gather_table_stats(NULL, 'CACHER');
  commit;
  alter system flush buffer_cache;
  set autot on stat
  select count(*) from cacher;
  select count(*) from cacher;
  select count(*) from cacher;
  set autot off;

with
_db_block_buffers 273102
_small_table_threshold 5462
and the results were:

80% of buffer cache Statistics with stats gathered on table
----------------------------------------------------------
     220033  consistent gets
     220005  physical reads
----------------------------------------------------------
     220023  consistent gets
          0  physical reads
----------------------------------------------------------
     220023  consistent gets
          0  physical reads

Thanks for posting a follow up comment with a test case.

I noticed in the test case script that table statistics were not collected for the CACHER table. Do you think that skipping the statistics collection step might cause the runtime engine problems where it believes that the table is very small, resulting in too many of the blocks being cached, or do you think that the automatic dynamic sampling would have prevented problems like that? I am more than a little surprised by the results you posted for 10.2.0.4.

In the above comments I see that Jonathan had intended to write up a blog article about the _SMALL_TABLE_THRESHOLD parameter – it appears that he had a chance to finish up that article:
http://jonathanlewis.wordpress.com/2011/03/24/small-tables/