I ran the SSD test and new AWR report.

              Snap Id      Snap Time      Sessions Curs/Sess
            --------- ------------------- -------- ---------
Begin Snap:      4565 23-May-12 15:10:56        21       1.6
  End Snap:      4566 23-May-12 15:32:08        25       1.6
   Elapsed:               21.20 (mins)
   DB Time:               21.38 (mins)

Segments by Physical Writes               DB/Inst: MYDB/mydb  Snaps: 4565-4566
-> Total Physical Writes:       7,034,915
-> Captured Segments account for   63.9% of Total

           Tablespace                      Subobject  Obj.      Physical
Owner         Name    Object Name            Name     Type        Writes  %Total
---------- ---------- -------------------- ---------- ----- ------------ -------
SSDTESTER  APP1       TESTWRITES                      TABLE    4,493,979   63.88
SSDTESTER  APP1       TESTWRITES_PK                   INDEX        3,708     .05
SYS        SYSAUX     WRH$_ACTIVE_SESSION_ 49386_4552 TABLE           16     .00
SYS        SYSAUX     WRH$_PARAMETER_PK    49386_4552 INDEX           14     .00
SYS        SYSTEM     SEG$                            TABLE           13     .00
          -------------------------------------------------------------

Segments by Physical Write Requests       DB/Inst: MYDB/mydb  Snaps: 4565-4566
-> Total Physical Write Requestss:       5,735,831
-> Captured Segments account for   63.9% of Total

           Tablespace                      Subobject  Obj.    Phys Write
Owner         Name    Object Name            Name     Type      Requests  %Total
---------- ---------- -------------------- ---------- ----- ------------ -------
SSDTESTER  APP1       TESTWRITES                      TABLE    3,664,810   63.89
SSDTESTER  APP1       TESTWRITES_PK                   INDEX        2,144     .04
SYS        SYSTEM     SEG$                            TABLE           13     .00
SYSMAN     SYSAUX     MGMT_SYSTEM_PERFORMA            TABLE           11     .00
SYS        SYSAUX     WRH$_SQL_PLAN                   TABLE           11     .00
          -------------------------------------------------------------


Tablespace
------------------------------
                 Av       Av     Av                       Av     Buffer  Av Buf
         Reads Reads/s  Rd(ms) Blks/Rd       Writes Writes/s      Waits  Wt(ms)
-------------- ------- ------- ------- ------------ -------- ---------- -------
APP1
     2,837,745   2,230     0.3     1.1    3,667,136    2,882          0     0.0
UNDOTBS1
           104       0     2.4     1.0    2,067,857    1,625          0     0.0
SYSTEM
         5,993       5     0.2     1.3          141        0          2     0.0

Compared to the prtevious HDD

              Snap Id      Snap Time      Sessions Curs/Sess
            --------- ------------------- -------- ---------
Begin Snap:      4417 17-May-12 12:31:47        30       2.7
  End Snap:      4418 17-May-12 13:51:35        32       3.2
   Elapsed:               79.80 (mins)
   DB Time:               81.75 (mins)

Segments by Physical Writes               DB/Inst: MYDB/mydb  Snaps: 4417-4418
-> Total Physical Writes:       6,857,129
-> Captured Segments account for   62.9% of Total

           Tablespace                      Subobject  Obj.      Physical
Owner         Name    Object Name            Name     Type        Writes  %Total
---------- ---------- -------------------- ---------- ----- ------------ -------
TESTER     APP1_HDD   TESTWRITES                      TABLE    4,310,514   62.86
TESTER     APP1_HDD   TESTWRITES_PK                   INDEX        3,730     .05
SYSMAN     SYSAUX     MGMT_METRICS_RAW_PK             INDEX        1,488     .02
SYSMAN     SYSAUX     MGMT_METRICS_1HOUR_P            INDEX          126     .00
SYSMAN     SYSAUX     MGMT_SYSTEM_PERF_LOG            INDEX           66     .00
          -------------------------------------------------------------

Segments by Physical Write Requests       DB/Inst: MYDB/mydb  Snaps: 4417-4418
-> Total Physical Write Requestss:       1,912,076
-> Captured Segments account for   73.1% of Total

           Tablespace                      Subobject  Obj.    Phys Write
Owner         Name    Object Name            Name     Type      Requests  %Total
---------- ---------- -------------------- ---------- ----- ------------ -------
TESTER     APP1_HDD   TESTWRITES                      TABLE    1,395,810   73.00
TESTER     APP1_HDD   TESTWRITES_PK                   INDEX          770     .04
SYSMAN     SYSAUX     MGMT_METRICS_RAW_PK             INDEX          681     .04
SYSMAN     SYSAUX     MGMT_SYSTEM_PERF_LOG            INDEX           48     .00
SYSMAN     SYSAUX     MGMT_METRICS_1HOUR_P            INDEX           41     .00
          -------------------------------------------------------------

Tablespace
------------------------------
                 Av       Av     Av                       Av     Buffer  Av Buf
         Reads Reads/s  Rd(ms) Blks/Rd       Writes Writes/s      Waits  Wt(ms)
-------------- ------- ------- ------- ------------ -------- ---------- -------
APP1_HDD
     3,535,159     738     1.2     1.1    1,396,750      292          0     0.0
UNDOTBS1
           104       0     0.2     1.0      513,323      107          0     0.0
SYSTEM
         8,050       2     0.2     1.5          190        0          0     0.0
SYSAUX
         3,736       1     0.2     2.0        1,496        0          0     0.0

Stiil do not know what the issue could be:

Obviously we don’t have any info as to how many writes the OS does when we ask it to write 8k, we just ask it to write 8k and increment our metric stating that we wrote 8k.

Cheers,

Mich

That is an interesting observation. The Oracle Database statistics should NOT be able to see the extra writes to erase blocks on SSD – that should all be hidden in the operating system call to write to disk; Oracle Database would be able to determine the time required for the call to complete, but not the amount of internal disk-level work done.

Is there any chance that one of your test runs could have been performed during the time when an automatic AWR collection was initiated (by default at the start of every hour)?

I think that looking at the number of blocks written by DBWR as shown in the AWR report could cause some confusion. Oracle Database is fundamentally lazy… it procrastinates doing tasks that are time consuming and that do not need to be done immediately. When the commits in the PL/SQL block are executed, the redo log buffer *must* be flushed to disk, but DBWR does not need to flush the commited changes in the blocks to disk immediately. The uncommitted changes in the blocks could also be flushed to disk before a session issues a commit, as additional space is needed in the buffer cache. Throw delayed block cleanout and dynamically resized buffer caches into the mix, and you could find that the same block may be written to disk several times in one of the test runs, but not in the other. It is interesting to note that the number of blocks read from the APP1_HDD tablespace’s datafiles is nearly identical to the number of blocks written to the APP1 tablespace’s datafiles in the other AWR report.

http://docs.oracle.com/cd/E14072_01/server.112/e10713/memory.htm#sthref1580

“The database updates data blocks in the cache and stores metadata about the changes in the redo log buffer. After a COMMIT, the database writes the redo buffers to disk but does not immediately write data blocks to disk. Instead, database writer (DBWn) performs lazy writes in the background.”

You may also enjoy reading the following by Jonathan Lewis:
http://jonathanlewis.wordpress.com/2009/06/16/clean-it-up/
http://books.google.com/books?id=r86URDzs1TAC&pg=PA137 (the book “Oracle Core”, see “PL/SQL Optimization” on page 126, and “Database Writer” on pages 137-156)

When testing performance, you need to research what can change that is not under the control of your experiment. For instance, the server resynonchonrizing its time with an outside time source (NTP) could throw off time-delta calculations, non-instance related load placed on the drives or server CPU or physical memory, operating system level caching (seems to be enabled in your test case), automatic processing in the database (such as automatic statistics collection that typically starts around 10:00 PM) or automatic AWR collection, virus scanner activity (both Windows and Linux servers could be affected if a virus scanner is installed on the server), PL/SQL memory usage from a previous test run leaving Oracle Database believing that there is more competition for PGA memory – thus either automatically growing the PGA memory while decreasing the SGA memory (assuming that MEMORY_TARGET is used) or gradually permitting each session to use a smaller percentage of memory for PGA requirements, delayed block cleanouts caused by previous tests (or other activity) that affect later tests, etc.

In the end, it comes down to timing. After performing the test several times, what is the average elapsed time, and how much of the server’s resources were consumed to achieve that average elapsed time? In my experience, SATA drives in servers seem to drive up kernel mode CPU usage (and may not offer as effective native command queuing) when compared to SCSI (or SAS – serial attached SCSI), so that could also be a factor as you take a look at expanding the test to multiple sessions performing concurrent work at the same time.

Sorry, not a simple answer from me (I might be overlooking something painfully obvious). Maybe one of the other blog readers are able to better interpret the output that you provided.

Ok that was easy enough. However, I did that test because I wanted to load a typical OLTP simulation for solid state disks versus normal magnetic disks
(SSD vs HDD). I recall a while back you dis some tests on SSD as well. The challenge is whether it is possible for Oracle to write the same volume of data to disk with SSD against HDD.

If I rewrite the code to take AWR snapshots with checkpoints before and after, I should be able to verify the volume of writes.

alter system flush buffer_cache;
alter system flush shared_pool;
ALTER SYSTEM CHECKPOINT;
EXEC DBMS_WORKLOAD_REPOSITORY.CREATE_SNAPSHOT();
DECLARE
        type ObjIdArray is table of tdash.object_id%TYPE index by binary_integer;
        l_ids objIdArray;
        CURSOR c IS SELECT object_id FROM tdash;
BEGIN
        OPEN c;
        LOOP
                BEGIN
                        FETCH c BULK COLLECT INTO l_ids LIMIT 100;

                        FORALL rs in 1 .. l_ids.COUNT
                                UPDATE testwrites
                                  SET PADDING1 =  RPAD('y',4000,'y')
                                WHERE object_id = l_ids(rs);
                        commit;
                        FORALL rs in 1 .. l_ids.COUNT
                                DELETE FROM testwrites
                                WHERE object_id = l_ids(rs);
                        commit;
                        FORALL rs in 1 .. l_ids.COUNT
                                INSERT INTO testwrites
                                SELECT * FROM tdash t WHERE t.object_id = l_ids(rs);
                        commit;
                        EXIT WHEN C%NOTFOUND;
                EXCEPTION
                  WHEN NO_DATA_FOUND THEN
                    NULL;
                  WHEN OTHERS THEN
                    DBMS_OUTPUT.PUT_LINE('Transaction failed');
                END;
        END LOOP;
        CLOSE c;
END;
/
ALTER SYSTEM CHECKPOINT;
EXEC DBMS_WORKLOAD_REPOSITORY.CREATE_SNAPSHOT();

If we run the script against tables on SSD and again HDD it would be expected to see roughly the same volume of writes. However, is that the case? Look below. The first section is about HDD and tablescape is called APP1_HDD

 Tablespace
------------------------------
                 Av       Av     Av                       Av     Buffer  Av Buf
         Reads Reads/s  Rd(ms) Blks/Rd       Writes Writes/s      Waits  Wt(ms)
-------------- ------- ------- ------- ------------ -------- ---------- -------
APP1_HDD
     3,535,159     738     1.2     1.1    1,396,750      292          0     0.0
UNDOTBS1
           104       0     0.2     1.0      513,323      107          0     0.0
SYSTEM
         8,050       2     0.2     1.5          190        0          0     0.0
SYSAUX
         3,736       1     0.2     2.0        1,496        0          0     0.0
APP1
           144       0     0.0     1.0          117        0          0     0.0

Now the same stats for SSD tablespace APP1

Tablespace
------------------------------
                 Av       Av     Av                       Av     Buffer  Av Buf
         Reads Reads/s  Rd(ms) Blks/Rd       Writes Writes/s      Waits  Wt(ms)
-------------- ------- ------- ------- ------------ -------- ---------- -------
APP1
     2,860,666   2,268     0.3     1.1    3,564,644    2,826          0     0.0
UNDOTBS1
            90       0     9.8     1.0    1,990,759    1,578          0     0.0
SYSTEM
         7,190       6     0.2     1.3          204        0          0     0.0
SYSAUX
         1,938       2     0.3     2.1        1,047        1          0     0.0
APP1_HDD
           105       0     0.1     1.0          105        0          0     0.0
 

So I still do not see why the volumes of writes are higher for SSD except perhaps that updates (erase OS blocks) are causing extra writes for SSD.

The good news is that what you posted above exactly matches the output that I posted above from the 11.2.0.1 version of tkprof on my system, so there is some consistency. The bad news is that it is consistently bad – the output does NOT match the output from the 11.2.0.2 version of tkprof. The bad news probably extends also to the trace file that you generated and the processed with tkprof, and that probably explains why the figures in the tkprof output from your trace file are not self-consistent.

At this point, I recommend trying to pass your trace file through tkprof from 11.2.0.2 or 11.2.0.3 and compare the output. I have a strong feeling that you are encounting the same bug that you found when you processed my trace file.

It appears that there is a similar topic on the OTN forums. If you find that tkprof from 11.2.0.2 or 11.2.0.3 produces non-self-conflicting output, you might post that information in the OTN thread:
https://forums.oracle.com/forums/thread.jspa?messageID=10343194

This is the processed tkprof file with source from your zipped trc file

tkprof or1122p_ora_1808.trc mytrace.txt sys=no

SQL ID: 1ahnbczb20gdx
Plan Hash: 1751280057
UPDATE T2 SET PADDING1 = RPAD('y',4000,'y')
WHERE
 OBJECT_ID = :B1


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute    190      1.35       1.42        348      52552     165435       19000
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total      191      1.35       1.42        348      52552     165435       19000
..
..
Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file scattered read                         40        0.00          0.03
  db file sequential read                        32        0.00          0.01
  log file switch completion                      2        0.00          0.01
********************************************************************************

Now why am I getting 19000 rows as opposed to 50000 rows?

Thank you for the follow up.

Is there any way that you can pass the original trace file through tkprof from 11.2.0.2 or 11.2.0.3 so that you can confirm that there was a bug in 11.2.0.1′s tkprof? If you download the zip file at the end of my blog article (http://hoopercharles.files.wordpress.com/2012/05/or1122p_ora_1808-zip.doc ) and process the enclosed .trc file with tkprof, do you obtain the incorrect statistics from 11.2.0.1 shown above, or do you obtain the correct statistics from 11.2.0.2 above?

A search through Metalink (MOS) found an interesting bug fix in Exadata 11.2.0.1 bundle patch 7: “Bug 8328200 SHRD CRSRS: Tkprof does not aggregate STAT lines for ALL_EXECUTIONS”. If you have access to MOS, you can see the bug report here:
https://supporthtml.oracle.com/epmos/faces/ui/km/DocumentDisplay.jspx?_afrLoop=3303558429470000&id=8328200.8

SQL ID: 5qvgta8arnhya
Plan Hash: 2666452017
UPDATE TESTWRITES SET PADDING1 = RPAD('y',4000,'y')
WHERE
 OBJECT_ID = :B1


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute  17293     29.44    1070.20    1126609    2891927    8091191     1000000
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total    17294     29.44    1070.20    1126609    2891927    8091191     1000000

Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: ALL_ROWS
Parsing user id: 93     (recursive depth: 1)

Rows     Row Source Operation
-------  ---------------------------------------------------
      0  UPDATE  TESTWRITES (cr=214 pr=426 pw=0 time=0 us)
    100   INDEX UNIQUE SCAN TESTWRITES_PK (cr=110 pr=3 pw=0 time=0 us cost=2 size=4007 card=1)(object id 92592)


Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file sequential read                    686458        1.00        741.53
  Disk file operations I/O                        2        0.00          0.00
  db file scattered read                      55783        1.16        147.49
  log file switch completion                      5        0.15          0.47
  log file switch (private strand flush incomplete)
                                                  3        0.21          0.41
********************************************************************************

which gives Total Waited as 889.9 which looks more reasonable compared to the elasped time of 1070.20 sec. However, it begs the question if there is bug with the version of tkprof I am using (Release 11.2.0.1.0). I guess I can go directly to the trace file and use UNIX sed and awk to extract and sum data. However, that defeats the object. Well the issue we saw before is not a quantization error. Well I do not think one can attibute 49% differences to error in meassurement as in the original case.

Cheers,

Mich