Thank you for providing the link to that article. I recall reading about that feedback loop in your “Cost-Based Oracle Fundamentals” book. Speaking of which, I am still waiting for volumes 2 and 3 of the book series.

It is interesting that the performance problem is present in the test case script (found in the follow-up article) when the OPTIMIZER_FEATURES_ENABLE is set to 10.1.0.4 in both Oracle Database 10.2.0.2 and 11.2.0.1, but the problem is not present in Oracle Database 10.2.0.5 when the test is repeated in either the Standard Edition or the Enterprise Edition (that will be illustrated in part 3).

Just hogging things here, but I realised I missed an opportunity to advertise my extremely excellent blog where there’s a little note that shows this type of CVM. http://jonathanlewis.wordpress.com/2007/03/08/transformation-and-optimisation/

I forgot to say – after checking what was going on, I’d try to model it with a simpler example – “insert as select from two table join which could be satisfied by index-only acceses” (similar to the original nested loop join section.

Regards
Jonathan Lewis

I am a late arrival at this party – but my approach would be as follows:
Comparing lines 4 – 7 in the first plan with lines 4 – 10 in the second plan, there’s clearly an example of complex view merging taking place (“group by then join” turns into “join then group by”).

The complex view merging would have been unconditional in 9i – but should be costed in 10g: so (although I rarely use the 10053 trace this early in an investigation) I would assume in this case that there was a bug in the optimizer code somewhere in the handling of the “insert as select”.

I’d use the no_merge hint suggested by Flado to check that the behaviour changed to the desired behavious. And I’d look at three trace files (select, insert unhinted, insert hinted) to see if I could spot the place where CVM (complex view merging) failed to appear when expected, or appeared with an unsuitable cost; or where the cost of not merging appeared to be higher than the cost of merging.

For an experiment, I created the following script:

CREATE TABLE T1_DEST (
  PART_ID VARCHAR2(30),
  REQUIRED_DATE DATE,
  QTY NUMBER,
  PLAN_LEVEL NUMBER);

SET TIMING ON
SET ARRAYSIZE 1000

ALTER SESSION SET OPTIMIZER_FEATURES_ENABLE='10.1.0.4';
ALTER SESSION SET TRACEFILE_IDENTIFIER = 'SELECT_10.1.0.4';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';

SELECT /*+ LEADING(E) */
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM') PART_ID,
  E.FCDUEDATE REQUIRED_DATE,
  SUM(E.FCSTQTY) QTY,
  1 PLAN_LEVEL
FROM
  (SELECT /*+ LEADING(E) */
    E.CUSTPO,
    EL.FCSTQTY,
    EL.FCDUEDATE
  FROM
    (SELECT /*+  */
      E.CUSTPO,
      MAX(RELID) RELID
    FROM
      TESTUSER.EDI830 E
    GROUP BY
      E.CUSTPO) E,
    TESTUSER.EDI830_LINES EL
  WHERE  
    E.CUSTPO=EL.CUSTPO
    AND E.RELID=EL.RELID
    AND EL.FCDUEDATE BETWEEN SYSDATE-365 AND SYSDATE+1200
    AND EL.FCSTQTY>0) E,
  (SELECT /*+  */
    CO.CUSTOMER_PO_REF,
    MAX(COL.DESIRED_SHIP_DATE) DESIRED_SHIP_DATE,
    MAX(COL.PART_ID) PART_ID
  FROM
    CUSTOMER_ORDER CO,
    CUST_ORDER_LINE COL
  WHERE
    CO.ID=COL.CUST_ORDER_ID
    AND CUSTOMER_PO_REF IS NOT NULL
    AND COL.DESIRED_SHIP_DATE>=SYSDATE-365
  GROUP BY
    CO.CUSTOMER_PO_REF) CO
WHERE
  E.CUSTPO=CO.CUSTOMER_PO_REF(+)
  AND E.FCDUEDATE>COALESCE(CO.DESIRED_SHIP_DATE,SYSDATE-365)
GROUP BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE
ORDER BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE;

ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT OFF';

ALTER SESSION SET TRACEFILE_IDENTIFIER = 'INSERT_10.1.0.4';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';

INSERT INTO T1_DEST
SELECT /*+ LEADING(E) */
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM') PART_ID,
  E.FCDUEDATE REQUIRED_DATE,
  SUM(E.FCSTQTY) QTY,
  1 PLAN_LEVEL
FROM
  (SELECT /*+ LEADING(E) */
    E.CUSTPO,
    EL.FCSTQTY,
    EL.FCDUEDATE
  FROM
    (SELECT /*+  */
      E.CUSTPO,
      MAX(RELID) RELID
    FROM
      TESTUSER.EDI830 E
    GROUP BY
      E.CUSTPO) E,
    TESTUSER.EDI830_LINES EL
  WHERE  
    E.CUSTPO=EL.CUSTPO
    AND E.RELID=EL.RELID
    AND EL.FCDUEDATE BETWEEN SYSDATE-365 AND SYSDATE+1200
    AND EL.FCSTQTY>0) E,
  (SELECT /*+  */
    CO.CUSTOMER_PO_REF,
    MAX(COL.DESIRED_SHIP_DATE) DESIRED_SHIP_DATE,
    MAX(COL.PART_ID) PART_ID
  FROM
    CUSTOMER_ORDER CO,
    CUST_ORDER_LINE COL
  WHERE
    CO.ID=COL.CUST_ORDER_ID
    AND CUSTOMER_PO_REF IS NOT NULL
    AND COL.DESIRED_SHIP_DATE>=SYSDATE-365
  GROUP BY
    CO.CUSTOMER_PO_REF) CO
WHERE
  E.CUSTPO=CO.CUSTOMER_PO_REF(+)
  AND E.FCDUEDATE>COALESCE(CO.DESIRED_SHIP_DATE,SYSDATE-365)
GROUP BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE
ORDER BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE;

ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT OFF';

TRUNCATE TABLE T1_DEST;

ALTER SESSION SET OPTIMIZER_FEATURES_ENABLE='10.2.0.2';
ALTER SESSION SET TRACEFILE_IDENTIFIER = 'INSERT_10.2.0.2';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';

SELECT /*+ LEADING(E) */
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM') PART_ID,
  E.FCDUEDATE REQUIRED_DATE,
  SUM(E.FCSTQTY) QTY,
  1 PLAN_LEVEL
FROM
  (SELECT /*+ LEADING(E) */
    E.CUSTPO,
    EL.FCSTQTY,
    EL.FCDUEDATE
  FROM
    (SELECT /*+  */
      E.CUSTPO,
      MAX(RELID) RELID
    FROM
      TESTUSER.EDI830 E
    GROUP BY
      E.CUSTPO) E,
    TESTUSER.EDI830_LINES EL
  WHERE  
    E.CUSTPO=EL.CUSTPO
    AND E.RELID=EL.RELID
    AND EL.FCDUEDATE BETWEEN SYSDATE-365 AND SYSDATE+1200
    AND EL.FCSTQTY>0) E,
  (SELECT /*+  */
    CO.CUSTOMER_PO_REF,
    MAX(COL.DESIRED_SHIP_DATE) DESIRED_SHIP_DATE,
    MAX(COL.PART_ID) PART_ID
  FROM
    CUSTOMER_ORDER CO,
    CUST_ORDER_LINE COL
  WHERE
    CO.ID=COL.CUST_ORDER_ID
    AND CUSTOMER_PO_REF IS NOT NULL
    AND COL.DESIRED_SHIP_DATE>=SYSDATE-365
  GROUP BY
    CO.CUSTOMER_PO_REF) CO
WHERE
  E.CUSTPO=CO.CUSTOMER_PO_REF(+)
  AND E.FCDUEDATE>COALESCE(CO.DESIRED_SHIP_DATE,SYSDATE-365)
GROUP BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE
ORDER BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE;

ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT OFF';

ALTER SESSION SET TRACEFILE_IDENTIFIER = 'INSERT_10.2.0.2';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';

INSERT INTO T1_DEST
SELECT /*+ LEADING(E) */
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM') PART_ID,
  E.FCDUEDATE REQUIRED_DATE,
  SUM(E.FCSTQTY) QTY,
  1 PLAN_LEVEL
FROM
  (SELECT /*+ LEADING(E) */
    E.CUSTPO,
    EL.FCSTQTY,
    EL.FCDUEDATE
  FROM
    (SELECT /*+  */
      E.CUSTPO,
      MAX(RELID) RELID
    FROM
      TESTUSER.EDI830 E
    GROUP BY
      E.CUSTPO) E,
    TESTUSER.EDI830_LINES EL
  WHERE  
    E.CUSTPO=EL.CUSTPO
    AND E.RELID=EL.RELID
    AND EL.FCDUEDATE BETWEEN SYSDATE-365 AND SYSDATE+1200
    AND EL.FCSTQTY>0) E,
  (SELECT /*+  */
    CO.CUSTOMER_PO_REF,
    MAX(COL.DESIRED_SHIP_DATE) DESIRED_SHIP_DATE,
    MAX(COL.PART_ID) PART_ID
  FROM
    CUSTOMER_ORDER CO,
    CUST_ORDER_LINE COL
  WHERE
    CO.ID=COL.CUST_ORDER_ID
    AND CUSTOMER_PO_REF IS NOT NULL
    AND COL.DESIRED_SHIP_DATE>=SYSDATE-365
  GROUP BY
    CO.CUSTOMER_PO_REF) CO
WHERE
  E.CUSTPO=CO.CUSTOMER_PO_REF(+)
  AND E.FCDUEDATE>COALESCE(CO.DESIRED_SHIP_DATE,SYSDATE-365)
GROUP BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE
ORDER BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE;

ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT OFF';

TRUNCATE TABLE T1_DEST;

ALTER SESSION SET OPTIMIZER_FEATURES_ENABLE='10.2.0.2';
ALTER SESSION SET "_OPTIMIZER_UNDO_COST_CHANGE"='10.2.0.2';
ALTER SESSION SET TRACEFILE_IDENTIFIER = 'INSERT_UNDO_10.2.0.2';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';

SELECT /*+ LEADING(E) */
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM') PART_ID,
  E.FCDUEDATE REQUIRED_DATE,
  SUM(E.FCSTQTY) QTY,
  1 PLAN_LEVEL
FROM
  (SELECT /*+ LEADING(E) */
    E.CUSTPO,
    EL.FCSTQTY,
    EL.FCDUEDATE
  FROM
    (SELECT /*+  */
      E.CUSTPO,
      MAX(RELID) RELID
    FROM
      TESTUSER.EDI830 E
    GROUP BY
      E.CUSTPO) E,
    TESTUSER.EDI830_LINES EL
  WHERE  
    E.CUSTPO=EL.CUSTPO
    AND E.RELID=EL.RELID
    AND EL.FCDUEDATE BETWEEN SYSDATE-365 AND SYSDATE+1200
    AND EL.FCSTQTY>0) E,
  (SELECT /*+  */
    CO.CUSTOMER_PO_REF,
    MAX(COL.DESIRED_SHIP_DATE) DESIRED_SHIP_DATE,
    MAX(COL.PART_ID) PART_ID
  FROM
    CUSTOMER_ORDER CO,
    CUST_ORDER_LINE COL
  WHERE
    CO.ID=COL.CUST_ORDER_ID
    AND CUSTOMER_PO_REF IS NOT NULL
    AND COL.DESIRED_SHIP_DATE>=SYSDATE-365
  GROUP BY
    CO.CUSTOMER_PO_REF) CO
WHERE
  E.CUSTPO=CO.CUSTOMER_PO_REF(+)
  AND E.FCDUEDATE>COALESCE(CO.DESIRED_SHIP_DATE,SYSDATE-365)
GROUP BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE
ORDER BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE;

ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT OFF';

ALTER SESSION SET TRACEFILE_IDENTIFIER = 'INSERT_UNDO_10.2.0.2';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';

INSERT INTO T1_DEST
SELECT /*+ LEADING(E) */
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM') PART_ID,
  E.FCDUEDATE REQUIRED_DATE,
  SUM(E.FCSTQTY) QTY,
  1 PLAN_LEVEL
FROM
  (SELECT /*+ LEADING(E) */
    E.CUSTPO,
    EL.FCSTQTY,
    EL.FCDUEDATE
  FROM
    (SELECT /*+  */
      E.CUSTPO,
      MAX(RELID) RELID
    FROM
      TESTUSER.EDI830 E
    GROUP BY
      E.CUSTPO) E,
    TESTUSER.EDI830_LINES EL
  WHERE  
    E.CUSTPO=EL.CUSTPO
    AND E.RELID=EL.RELID
    AND EL.FCDUEDATE BETWEEN SYSDATE-365 AND SYSDATE+1200
    AND EL.FCSTQTY>0) E,
  (SELECT /*+  */
    CO.CUSTOMER_PO_REF,
    MAX(COL.DESIRED_SHIP_DATE) DESIRED_SHIP_DATE,
    MAX(COL.PART_ID) PART_ID
  FROM
    CUSTOMER_ORDER CO,
    CUST_ORDER_LINE COL
  WHERE
    CO.ID=COL.CUST_ORDER_ID
    AND CUSTOMER_PO_REF IS NOT NULL
    AND COL.DESIRED_SHIP_DATE>=SYSDATE-365
  GROUP BY
    CO.CUSTOMER_PO_REF) CO
WHERE
  E.CUSTPO=CO.CUSTOMER_PO_REF(+)
  AND E.FCDUEDATE>COALESCE(CO.DESIRED_SHIP_DATE,SYSDATE-365)
GROUP BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE
ORDER BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE;

ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT OFF';

(disconnect and reconnect)

SET TIMING ON
SET ARRAYSIZE 1000

TRUNCATE TABLE T1_DEST;

ALTER SESSION SET OPTIMIZER_INDEX_COST_ADJ=100;
ALTER SESSION SET OPTIMIZER_INDEX_CACHING=0;
ALTER SESSION SET OPTIMIZER_FEATURES_ENABLE='10.1.0.4';
ALTER SESSION SET TRACEFILE_IDENTIFIER = 'SELECT_10.1.0.4-2';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';

SELECT /*+ LEADING(E) */
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM') PART_ID,
  E.FCDUEDATE REQUIRED_DATE,
  SUM(E.FCSTQTY) QTY,
  1 PLAN_LEVEL
FROM
  (SELECT /*+ LEADING(E) */
    E.CUSTPO,
    EL.FCSTQTY,
    EL.FCDUEDATE
  FROM
    (SELECT /*+ 2 */
      E.CUSTPO,
      MAX(RELID) RELID
    FROM
      TESTUSER.EDI830 E
    GROUP BY
      E.CUSTPO) E,
    TESTUSER.EDI830_LINES EL
  WHERE  
    E.CUSTPO=EL.CUSTPO
    AND E.RELID=EL.RELID
    AND EL.FCDUEDATE BETWEEN SYSDATE-365 AND SYSDATE+1200
    AND EL.FCSTQTY>0) E,
  (SELECT /*+  */
    CO.CUSTOMER_PO_REF,
    MAX(COL.DESIRED_SHIP_DATE) DESIRED_SHIP_DATE,
    MAX(COL.PART_ID) PART_ID
  FROM
    CUSTOMER_ORDER CO,
    CUST_ORDER_LINE COL
  WHERE
    CO.ID=COL.CUST_ORDER_ID
    AND CUSTOMER_PO_REF IS NOT NULL
    AND COL.DESIRED_SHIP_DATE>=SYSDATE-365
  GROUP BY
    CO.CUSTOMER_PO_REF) CO
WHERE
  E.CUSTPO=CO.CUSTOMER_PO_REF(+)
  AND E.FCDUEDATE>COALESCE(CO.DESIRED_SHIP_DATE,SYSDATE-365)
GROUP BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE
ORDER BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE;

ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT OFF';

ALTER SESSION SET TRACEFILE_IDENTIFIER = 'INSERT_10.1.0.4-2';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';

INSERT INTO T1_DEST
SELECT /*+ LEADING(E) */
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM') PART_ID,
  E.FCDUEDATE REQUIRED_DATE,
  SUM(E.FCSTQTY) QTY,
  1 PLAN_LEVEL
FROM
  (SELECT /*+ LEADING(E) */
    E.CUSTPO,
    EL.FCSTQTY,
    EL.FCDUEDATE
  FROM
    (SELECT /*+ 2 */
      E.CUSTPO,
      MAX(RELID) RELID
    FROM
      TESTUSER.EDI830 E
    GROUP BY
      E.CUSTPO) E,
    TESTUSER.EDI830_LINES EL
  WHERE  
    E.CUSTPO=EL.CUSTPO
    AND E.RELID=EL.RELID
    AND EL.FCDUEDATE BETWEEN SYSDATE-365 AND SYSDATE+1200
    AND EL.FCSTQTY>0) E,
  (SELECT /*+  */
    CO.CUSTOMER_PO_REF,
    MAX(COL.DESIRED_SHIP_DATE) DESIRED_SHIP_DATE,
    MAX(COL.PART_ID) PART_ID
  FROM
    CUSTOMER_ORDER CO,
    CUST_ORDER_LINE COL
  WHERE
    CO.ID=COL.CUST_ORDER_ID
    AND CUSTOMER_PO_REF IS NOT NULL
    AND COL.DESIRED_SHIP_DATE>=SYSDATE-365
  GROUP BY
    CO.CUSTOMER_PO_REF) CO
WHERE
  E.CUSTPO=CO.CUSTOMER_PO_REF(+)
  AND E.FCDUEDATE>COALESCE(CO.DESIRED_SHIP_DATE,SYSDATE-365)
GROUP BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE
ORDER BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE;

ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT OFF';

TRUNCATE TABLE T1_DEST;

ALTER SESSION SET OPTIMIZER_FEATURES_ENABLE='10.2.0.2';
ALTER SESSION SET TRACEFILE_IDENTIFIER = 'INSERT_10.2.0.2-2';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';

SELECT /*+ LEADING(E) */
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM') PART_ID,
  E.FCDUEDATE REQUIRED_DATE,
  SUM(E.FCSTQTY) QTY,
  1 PLAN_LEVEL
FROM
  (SELECT /*+ LEADING(E) */
    E.CUSTPO,
    EL.FCSTQTY,
    EL.FCDUEDATE
  FROM
    (SELECT /*+ 2 */
      E.CUSTPO,
      MAX(RELID) RELID
    FROM
      TESTUSER.EDI830 E
    GROUP BY
      E.CUSTPO) E,
    TESTUSER.EDI830_LINES EL
  WHERE  
    E.CUSTPO=EL.CUSTPO
    AND E.RELID=EL.RELID
    AND EL.FCDUEDATE BETWEEN SYSDATE-365 AND SYSDATE+1200
    AND EL.FCSTQTY>0) E,
  (SELECT /*+  */
    CO.CUSTOMER_PO_REF,
    MAX(COL.DESIRED_SHIP_DATE) DESIRED_SHIP_DATE,
    MAX(COL.PART_ID) PART_ID
  FROM
    CUSTOMER_ORDER CO,
    CUST_ORDER_LINE COL
  WHERE
    CO.ID=COL.CUST_ORDER_ID
    AND CUSTOMER_PO_REF IS NOT NULL
    AND COL.DESIRED_SHIP_DATE>=SYSDATE-365
  GROUP BY
    CO.CUSTOMER_PO_REF) CO
WHERE
  E.CUSTPO=CO.CUSTOMER_PO_REF(+)
  AND E.FCDUEDATE>COALESCE(CO.DESIRED_SHIP_DATE,SYSDATE-365)
GROUP BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE
ORDER BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE;

ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT OFF';

ALTER SESSION SET TRACEFILE_IDENTIFIER = 'INSERT_10.2.0.2-2';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';

INSERT INTO T1_DEST
SELECT /*+ LEADING(E) */
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM') PART_ID,
  E.FCDUEDATE REQUIRED_DATE,
  SUM(E.FCSTQTY) QTY,
  1 PLAN_LEVEL
FROM
  (SELECT /*+ LEADING(E) */
    E.CUSTPO,
    EL.FCSTQTY,
    EL.FCDUEDATE
  FROM
    (SELECT /*+ 2 */
      E.CUSTPO,
      MAX(RELID) RELID
    FROM
      TESTUSER.EDI830 E
    GROUP BY
      E.CUSTPO) E,
    TESTUSER.EDI830_LINES EL
  WHERE  
    E.CUSTPO=EL.CUSTPO
    AND E.RELID=EL.RELID
    AND EL.FCDUEDATE BETWEEN SYSDATE-365 AND SYSDATE+1200
    AND EL.FCSTQTY>0) E,
  (SELECT /*+  */
    CO.CUSTOMER_PO_REF,
    MAX(COL.DESIRED_SHIP_DATE) DESIRED_SHIP_DATE,
    MAX(COL.PART_ID) PART_ID
  FROM
    CUSTOMER_ORDER CO,
    CUST_ORDER_LINE COL
  WHERE
    CO.ID=COL.CUST_ORDER_ID
    AND CUSTOMER_PO_REF IS NOT NULL
    AND COL.DESIRED_SHIP_DATE>=SYSDATE-365
  GROUP BY
    CO.CUSTOMER_PO_REF) CO
WHERE
  E.CUSTPO=CO.CUSTOMER_PO_REF(+)
  AND E.FCDUEDATE>COALESCE(CO.DESIRED_SHIP_DATE,SYSDATE-365)
GROUP BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE
ORDER BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE;

ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT OFF';

TRUNCATE TABLE T1_DEST;

ALTER SESSION SET OPTIMIZER_FEATURES_ENABLE='10.2.0.2';
ALTER SESSION SET "_OPTIMIZER_UNDO_COST_CHANGE"='10.2.0.2';
ALTER SESSION SET TRACEFILE_IDENTIFIER = 'INSERT_UNDO_10.2.0.2-2';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';

SELECT /*+ LEADING(E) */
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM') PART_ID,
  E.FCDUEDATE REQUIRED_DATE,
  SUM(E.FCSTQTY) QTY,
  1 PLAN_LEVEL
FROM
  (SELECT /*+ LEADING(E) */
    E.CUSTPO,
    EL.FCSTQTY,
    EL.FCDUEDATE
  FROM
    (SELECT /*+ 2 */
      E.CUSTPO,
      MAX(RELID) RELID
    FROM
      TESTUSER.EDI830 E
    GROUP BY
      E.CUSTPO) E,
    TESTUSER.EDI830_LINES EL
  WHERE  
    E.CUSTPO=EL.CUSTPO
    AND E.RELID=EL.RELID
    AND EL.FCDUEDATE BETWEEN SYSDATE-365 AND SYSDATE+1200
    AND EL.FCSTQTY>0) E,
  (SELECT /*+  */
    CO.CUSTOMER_PO_REF,
    MAX(COL.DESIRED_SHIP_DATE) DESIRED_SHIP_DATE,
    MAX(COL.PART_ID) PART_ID
  FROM
    CUSTOMER_ORDER CO,
    CUST_ORDER_LINE COL
  WHERE
    CO.ID=COL.CUST_ORDER_ID
    AND CUSTOMER_PO_REF IS NOT NULL
    AND COL.DESIRED_SHIP_DATE>=SYSDATE-365
  GROUP BY
    CO.CUSTOMER_PO_REF) CO
WHERE
  E.CUSTPO=CO.CUSTOMER_PO_REF(+)
  AND E.FCDUEDATE>COALESCE(CO.DESIRED_SHIP_DATE,SYSDATE-365)
GROUP BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE
ORDER BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE;

ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT OFF';

ALTER SESSION SET TRACEFILE_IDENTIFIER = 'INSERT_UNDO_10.2.0.2-2';
ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT FOREVER, LEVEL 1';

INSERT INTO T1_DEST
SELECT /*+ LEADING(E) */
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM') PART_ID,
  E.FCDUEDATE REQUIRED_DATE,
  SUM(E.FCSTQTY) QTY,
  1 PLAN_LEVEL
FROM
  (SELECT /*+ LEADING(E) */
    E.CUSTPO,
    EL.FCSTQTY,
    EL.FCDUEDATE
  FROM
    (SELECT /*+ 2 */
      E.CUSTPO,
      MAX(RELID) RELID
    FROM
      TESTUSER.EDI830 E
    GROUP BY
      E.CUSTPO) E,
    TESTUSER.EDI830_LINES EL
  WHERE  
    E.CUSTPO=EL.CUSTPO
    AND E.RELID=EL.RELID
    AND EL.FCDUEDATE BETWEEN SYSDATE-365 AND SYSDATE+1200
    AND EL.FCSTQTY>0) E,
  (SELECT /*+  */
    CO.CUSTOMER_PO_REF,
    MAX(COL.DESIRED_SHIP_DATE) DESIRED_SHIP_DATE,
    MAX(COL.PART_ID) PART_ID
  FROM
    CUSTOMER_ORDER CO,
    CUST_ORDER_LINE COL
  WHERE
    CO.ID=COL.CUST_ORDER_ID
    AND CUSTOMER_PO_REF IS NOT NULL
    AND COL.DESIRED_SHIP_DATE>=SYSDATE-365
  GROUP BY
    CO.CUSTOMER_PO_REF) CO
WHERE
  E.CUSTPO=CO.CUSTOMER_PO_REF(+)
  AND E.FCDUEDATE>COALESCE(CO.DESIRED_SHIP_DATE,SYSDATE-365)
GROUP BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE
ORDER BY
  COALESCE(CO.PART_ID,'NOT-IN-SYSTEM'),
  E.FCDUEDATE;

ALTER SESSION SET EVENTS '10053 TRACE NAME CONTEXT OFF';

The execution plans that were produced from the 10053 trace follow (note that if a hard parse was not performed, nothing will be written to a trace file, so that probably explains why some of the trace file results are missing):
select_10.1.0.4:

------------------------------------------------------+-----------------------------------+
| Id  | Operation                    | Name           | Rows  | Bytes | Cost  | Time      |
------------------------------------------------------+-----------------------------------+
| 0   | SELECT STATEMENT             |                |       |       |   157 |           |
| 1   |  SORT GROUP BY               |                |   204 |   17K |   157 |  00:00:01 |
| 2   |   FILTER                     |                |       |       |       |           |
| 3   |    HASH JOIN OUTER           |                |   374 |   31K |   156 |  00:00:01 |
| 4   |     NESTED LOOPS             |                |   194 |  9506 |    47 |  00:00:01 |
| 5   |      VIEW                    |                |  1993 |   35K |    46 |  00:00:01 |
| 6   |       SORT GROUP BY          |                |  1993 |   39K |    46 |  00:00:01 |
| 7   |        FILTER                |                |       |       |       |           |
| 8   |         INDEX FAST FULL SCAN | EDI830_PK      |  105K | 2092K |    33 |  00:00:01 |
| 9   |      INDEX RANGE SCAN        | EDI830_LINES_PK|     1 |    31 |     1 |  00:00:01 |
| 10  |     VIEW                     |                |  6261 |  220K |   108 |  00:00:01 |
| 11  |      SORT GROUP BY           |                |  6261 |  251K |   108 |  00:00:01 |
| 12  |       HASH JOIN              |                |  8185 |  328K |   106 |  00:00:01 |
| 13  |        TABLE ACCESS FULL     | CUSTOMER_ORDER |  7562 |  126K |    14 |  00:00:01 |
| 14  |        TABLE ACCESS FULL     | CUST_ORDER_LINE|  8550 |  200K |    92 |  00:00:01 |
------------------------------------------------------+-----------------------------------+
Predicate Information:
----------------------
2 - filter(("EL"."FCDUEDATE">COALESCE("CO"."DESIRED_SHIP_DATE",SYSDATE@!-365) AND COALESCE("CO"."DESIRED_SHIP_DATE",SYSDATE@!-365)<SYSDATE@!+1200))
3 - access("E"."CUSTPO"="CO"."CUSTOMER_PO_REF")
7 - filter(SYSDATE@!-365<=SYSDATE@!+1200)
9 - access("E"."CUSTPO"="EL"."CUSTPO" AND "E"."RELID"="EL"."RELID" AND "EL"."FCDUEDATE">=SYSDATE@!-365 AND "EL"."FCSTQTY">0 AND "EL"."FCDUEDATE"<=SYSDATE@!+1200)
9 - filter("EL"."FCSTQTY">0)
12 - access("CO"."ID"="COL"."CUST_ORDER_ID")
13 - filter("CUSTOMER_PO_REF" IS NOT NULL)
14 - filter("COL"."DESIRED_SHIP_DATE">=SYSDATE@!-365)

insert_10.1.0.4:

-------------------------------------------------------+-----------------------------------+
| Id  | Operation                     | Name           | Rows  | Bytes | Cost  | Time      |
-------------------------------------------------------+-----------------------------------+
| 0   | INSERT STATEMENT              |                |       |       |  272K |           |
| 1   |  SORT GROUP BY                |                |   204 |   13K |  272K |  00:22:07 |
| 2   |   FILTER                      |                |       |       |       |           |
| 3   |    MERGE JOIN OUTER           |                |   75K | 5046K |  272K |  00:22:07 |
| 4   |     VIEW                      |                |   69K | 2129K |  272K |  00:22:06 |
| 5   |      FILTER                   |                |       |       |       |           |
| 6   |       SORT GROUP BY           |                |   69K | 3502K |  272K |  00:22:06 |
| 7   |        FILTER                 |                |       |       |       |           |
| 8   |         HASH JOIN             |                |  104M | 5326M |  1154 |  00:00:06 |
| 9   |          INDEX FAST FULL SCAN | EDI830_PK      |  105K | 2092K |    33 |  00:00:01 |
| 10  |          TABLE ACCESS FULL    | EDI830_LINES   |  236K | 7311K |   941 |  00:00:05 |
| 11  |     SORT JOIN                 |                |  6261 |  220K |   110 |  00:00:01 |
| 12  |      VIEW                     |                |  6261 |  220K |   108 |  00:00:01 |
| 13  |       SORT GROUP BY           |                |  6261 |  251K |   108 |  00:00:01 |
| 14  |        HASH JOIN              |                |  8185 |  328K |   106 |  00:00:01 |
| 15  |         TABLE ACCESS FULL     | CUSTOMER_ORDER |  7562 |  126K |    14 |  00:00:01 |
| 16  |         TABLE ACCESS FULL     | CUST_ORDER_LINE|  8550 |  200K |    92 |  00:00:01 |
-------------------------------------------------------+-----------------------------------+
Predicate Information:
----------------------
2 - filter("E"."FCDUEDATE">COALESCE("CO"."DESIRED_SHIP_DATE",SYSDATE@!-365))
5 - filter("EL"."RELID"=MAX("RELID"))
7 - filter(SYSDATE@!-365<=SYSDATE@!+1200)
8 - access("E"."CUSTPO"="EL"."CUSTPO")
10 - filter(("EL"."FCDUEDATE">=SYSDATE@!-365 AND "EL"."FCSTQTY">0 AND "EL"."FCDUEDATE"<=SYSDATE@!+1200))
11 - access("E"."CUSTPO"="CO"."CUSTOMER_PO_REF")
11 - filter("E"."CUSTPO"="CO"."CUSTOMER_PO_REF")
14 - access("CO"."ID"="COL"."CUST_ORDER_ID")
15 - filter("CUSTOMER_PO_REF" IS NOT NULL)
16 - filter("COL"."DESIRED_SHIP_DATE">=SYSDATE@!-365)

insert_10.2.0.2:

------------------------------------------------------+-----------------------------------+
| Id  | Operation                    | Name           | Rows  | Bytes | Cost  | Time      |
------------------------------------------------------+-----------------------------------+
| 0   | SELECT STATEMENT             |                |       |       |   157 |           |
| 1   |  SORT GROUP BY               |                |   204 |   17K |   157 |  00:00:01 |
| 2   |   FILTER                     |                |       |       |       |           |
| 3   |    HASH JOIN OUTER           |                |   374 |   31K |   156 |  00:00:01 |
| 4   |     NESTED LOOPS             |                |   194 |  9506 |    47 |  00:00:01 |
| 5   |      VIEW                    |                |  1993 |   35K |    46 |  00:00:01 |
| 6   |       HASH GROUP BY          |                |  1993 |   39K |    46 |  00:00:01 |
| 7   |        FILTER                |                |       |       |       |           |
| 8   |         INDEX FAST FULL SCAN | EDI830_PK      |  105K | 2092K |    33 |  00:00:01 |
| 9   |      INDEX RANGE SCAN        | EDI830_LINES_PK|     1 |    31 |     1 |  00:00:01 |
| 10  |     VIEW                     |                |  6261 |  220K |   108 |  00:00:01 |
| 11  |      HASH GROUP BY           |                |  6261 |  251K |   108 |  00:00:01 |
| 12  |       HASH JOIN              |                |  8185 |  328K |   106 |  00:00:01 |
| 13  |        TABLE ACCESS FULL     | CUSTOMER_ORDER |  7562 |  126K |    14 |  00:00:01 |
| 14  |        TABLE ACCESS FULL     | CUST_ORDER_LINE|  8550 |  200K |    92 |  00:00:01 |
------------------------------------------------------+-----------------------------------+
Predicate Information:
----------------------
2 - filter(("EL"."FCDUEDATE">COALESCE("CO"."DESIRED_SHIP_DATE",SYSDATE@!-365) AND COALESCE("CO"."DESIRED_SHIP_DATE",SYSDATE@!-365)<SYSDATE@!+1200))
3 - access("E"."CUSTPO"="CO"."CUSTOMER_PO_REF")
7 - filter(SYSDATE@!-365<=SYSDATE@!+1200)
9 - access("E"."CUSTPO"="EL"."CUSTPO" AND "E"."RELID"="EL"."RELID" AND "EL"."FCDUEDATE">=SYSDATE@!-365 AND "EL"."FCSTQTY">0 AND "EL"."FCDUEDATE"<=SYSDATE@!+1200)
9 - filter("EL"."FCSTQTY">0)
12 - access("CO"."ID"="COL"."CUST_ORDER_ID")
13 - filter("CUSTOMER_PO_REF" IS NOT NULL)
14 - filter("COL"."DESIRED_SHIP_DATE">=SYSDATE@!-365)

insert_undo_10.2.0.2:

------------------------------------------------------+-----------------------------------+
| Id  | Operation                    | Name           | Rows  | Bytes | Cost  | Time      |
------------------------------------------------------+-----------------------------------+
| 0   | SELECT STATEMENT             |                |       |       |   157 |           |
| 1   |  SORT GROUP BY               |                |   204 |   17K |   157 |  00:00:01 |
| 2   |   FILTER                     |                |       |       |       |           |
| 3   |    HASH JOIN OUTER           |                |   375 |   31K |   156 |  00:00:01 |
| 4   |     NESTED LOOPS             |                |   194 |  9506 |    47 |  00:00:01 |
| 5   |      VIEW                    |                |  1993 |   35K |    46 |  00:00:01 |
| 6   |       HASH GROUP BY          |                |  1993 |   39K |    46 |  00:00:01 |
| 7   |        FILTER                |                |       |       |       |           |
| 8   |         INDEX FAST FULL SCAN | EDI830_PK      |  105K | 2092K |    33 |  00:00:01 |
| 9   |      INDEX RANGE SCAN        | EDI830_LINES_PK|     1 |    31 |     1 |  00:00:01 |
| 10  |     VIEW                     |                |  6261 |  220K |   108 |  00:00:01 |
| 11  |      HASH GROUP BY           |                |  6261 |  251K |   108 |  00:00:01 |
| 12  |       HASH JOIN              |                |  8185 |  328K |   106 |  00:00:01 |
| 13  |        TABLE ACCESS FULL     | CUSTOMER_ORDER |  7562 |  126K |    14 |  00:00:01 |
| 14  |        TABLE ACCESS FULL     | CUST_ORDER_LINE|  8550 |  200K |    92 |  00:00:01 |
------------------------------------------------------+-----------------------------------+
Predicate Information:
----------------------
2 - filter(("EL"."FCDUEDATE">COALESCE("CO"."DESIRED_SHIP_DATE",SYSDATE@!-365) AND COALESCE("CO"."DESIRED_SHIP_DATE",SYSDATE@!-365)<SYSDATE@!+1200))
3 - access("E"."CUSTPO"="CO"."CUSTOMER_PO_REF")
7 - filter(SYSDATE@!-365<=SYSDATE@!+1200)
9 - access("E"."CUSTPO"="EL"."CUSTPO" AND "E"."RELID"="EL"."RELID" AND "EL"."FCDUEDATE">=SYSDATE@!-365 AND "EL"."FCSTQTY">0 AND "EL"."FCDUEDATE"<=SYSDATE@!+1200)
9 - filter("EL"."FCSTQTY">0)
12 - access("CO"."ID"="COL"."CUST_ORDER_ID")
13 - filter("CUSTOMER_PO_REF" IS NOT NULL)
14 - filter("COL"."DESIRED_SHIP_DATE">=SYSDATE@!-365)

select_10.1.0.4-2:

------------------------------------------------------+-----------------------------------+
| Id  | Operation                    | Name           | Rows  | Bytes | Cost  | Time      |
------------------------------------------------------+-----------------------------------+
| 0   | SELECT STATEMENT             |                |       |       |  1101 |           |
| 1   |  SORT GROUP BY               |                |    11 |   935 |  1101 |  00:00:06 |
| 2   |   HASH JOIN                  |                |    11 |   935 |  1100 |  00:00:06 |
| 3   |    FILTER                    |                |       |       |       |           |
| 4   |     HASH JOIN OUTER          |                |  2186 |  115K |   155 |  00:00:01 |
| 5   |      VIEW                    |                |  1993 |   35K |    46 |  00:00:01 |
| 6   |       SORT GROUP BY          |                |  1993 |   39K |    46 |  00:00:01 |
| 7   |        FILTER                |                |       |       |       |           |
| 8   |         INDEX FAST FULL SCAN | EDI830_PK      |  105K | 2092K |    33 |  00:00:01 |
| 9   |      VIEW                    |                |  6261 |  220K |   108 |  00:00:01 |
| 10  |       SORT GROUP BY          |                |  6261 |  251K |   108 |  00:00:01 |
| 11  |        HASH JOIN             |                |  8185 |  328K |   106 |  00:00:01 |
| 12  |         TABLE ACCESS FULL    | CUSTOMER_ORDER |  7562 |  126K |    14 |  00:00:01 |
| 13  |         TABLE ACCESS FULL    | CUST_ORDER_LINE|  8550 |  200K |    92 |  00:00:01 |
| 14  |    TABLE ACCESS FULL         | EDI830_LINES   |  236K | 7311K |   941 |  00:00:05 |
------------------------------------------------------+-----------------------------------+
Predicate Information:
----------------------
2 - access("E"."CUSTPO"="EL"."CUSTPO" AND "E"."RELID"="EL"."RELID")
2 - filter("EL"."FCDUEDATE">COALESCE("CO"."DESIRED_SHIP_DATE",SYSDATE@!-365))
3 - filter(COALESCE("CO"."DESIRED_SHIP_DATE",SYSDATE@!-365)<SYSDATE@!+1200)
4 - access("E"."CUSTPO"="CO"."CUSTOMER_PO_REF")
7 - filter(SYSDATE@!-365<=SYSDATE@!+1200)
11 - access("CO"."ID"="COL"."CUST_ORDER_ID")
12 - filter("CUSTOMER_PO_REF" IS NOT NULL)
13 - filter("COL"."DESIRED_SHIP_DATE">=SYSDATE@!-365)
14 - filter(("EL"."FCDUEDATE">=SYSDATE@!-365 AND "EL"."FCSTQTY">0 AND "EL"."FCDUEDATE"<=SYSDATE@!+1200))

insert_10.1.0.4-2:

-------------------------------------------------------+-----------------------------------+
| Id  | Operation                     | Name           | Rows  | Bytes | Cost  | Time      |
-------------------------------------------------------+-----------------------------------+
| 0   | INSERT STATEMENT              |                |       |       |  272K |           |
| 1   |  SORT GROUP BY                |                |   204 |   13K |  272K |  00:22:07 |
| 2   |   FILTER                      |                |       |       |       |           |
| 3   |    MERGE JOIN OUTER           |                |   75K | 5046K |  272K |  00:22:07 |
| 4   |     VIEW                      |                |   69K | 2129K |  272K |  00:22:06 |
| 5   |      FILTER                   |                |       |       |       |           |
| 6   |       SORT GROUP BY           |                |   69K | 3502K |  272K |  00:22:06 |
| 7   |        FILTER                 |                |       |       |       |           |
| 8   |         HASH JOIN             |                |  104M | 5326M |  1154 |  00:00:06 |
| 9   |          INDEX FAST FULL SCAN | EDI830_PK      |  105K | 2092K |    33 |  00:00:01 |
| 10  |          TABLE ACCESS FULL    | EDI830_LINES   |  236K | 7311K |   941 |  00:00:05 |
| 11  |     SORT JOIN                 |                |  6261 |  220K |   110 |  00:00:01 |
| 12  |      VIEW                     |                |  6261 |  220K |   108 |  00:00:01 |
| 13  |       SORT GROUP BY           |                |  6261 |  251K |   108 |  00:00:01 |
| 14  |        HASH JOIN              |                |  8185 |  328K |   106 |  00:00:01 |
| 15  |         TABLE ACCESS FULL     | CUSTOMER_ORDER |  7562 |  126K |    14 |  00:00:01 |
| 16  |         TABLE ACCESS FULL     | CUST_ORDER_LINE|  8550 |  200K |    92 |  00:00:01 |
-------------------------------------------------------+-----------------------------------+
Predicate Information:
----------------------
2 - filter("E"."FCDUEDATE">COALESCE("CO"."DESIRED_SHIP_DATE",SYSDATE@!-365))
5 - filter("EL"."RELID"=MAX("RELID"))
7 - filter(SYSDATE@!-365<=SYSDATE@!+1200)
8 - access("E"."CUSTPO"="EL"."CUSTPO")
10 - filter(("EL"."FCDUEDATE">=SYSDATE@!-365 AND "EL"."FCSTQTY">0 AND "EL"."FCDUEDATE"<=SYSDATE@!+1200))
11 - access("E"."CUSTPO"="CO"."CUSTOMER_PO_REF")
11 - filter("E"."CUSTPO"="CO"."CUSTOMER_PO_REF")
14 - access("CO"."ID"="COL"."CUST_ORDER_ID")
15 - filter("CUSTOMER_PO_REF" IS NOT NULL)
16 - filter("COL"."DESIRED_SHIP_DATE">=SYSDATE@!-365)

insert_10.2.0.2-2:

--------------------------------------------------------+-----------------------------------+
| Id  | Operation                      | Name           | Rows  | Bytes | Cost  | Time      |
--------------------------------------------------------+-----------------------------------+
| 0   | INSERT STATEMENT               |                |       |       |  274K |           |
| 1   |  SORT GROUP BY                 |                |   204 |   13K |  274K |  00:22:15 |
| 2   |   FILTER                       |                |       |       |       |           |
| 3   |    MERGE JOIN OUTER            |                |   75K | 5046K |  274K |  00:22:15 |
| 4   |     SORT JOIN                  |                |   69K | 2129K |  274K |  00:22:14 |
| 5   |      VIEW                      |                |   69K | 2129K |  274K |  00:22:14 |
| 6   |       FILTER                   |                |       |       |       |           |
| 7   |        SORT GROUP BY           |                |   69K | 3502K |  274K |  00:22:14 |
| 8   |         FILTER                 |                |       |       |       |           |
| 9   |          HASH JOIN             |                |  104M | 5326M |  2876 |  00:00:14 |
| 10  |           INDEX FAST FULL SCAN | EDI830_PK      |  105K | 2092K |    33 |  00:00:01 |
| 11  |           TABLE ACCESS FULL    | EDI830_LINES   |  236K | 7310K |   941 |  00:00:05 |
| 12  |     SORT JOIN                  |                |  6261 |  220K |   110 |  00:00:01 |
| 13  |      VIEW                      |                |  6261 |  220K |   108 |  00:00:01 |
| 14  |       SORT GROUP BY            |                |  6261 |  251K |   108 |  00:00:01 |
| 15  |        HASH JOIN               |                |  8184 |  328K |   106 |  00:00:01 |
| 16  |         TABLE ACCESS FULL      | CUSTOMER_ORDER |  7562 |  126K |    14 |  00:00:01 |
| 17  |         TABLE ACCESS FULL      | CUST_ORDER_LINE|  8550 |  200K |    92 |  00:00:01 |
--------------------------------------------------------+-----------------------------------+
Predicate Information:
----------------------
2 - filter("E"."FCDUEDATE">COALESCE("CO"."DESIRED_SHIP_DATE",SYSDATE@!-365))
6 - filter("EL"."RELID"=MAX("RELID"))
8 - filter(SYSDATE@!-365<=SYSDATE@!+1200)
9 - access("E"."CUSTPO"="EL"."CUSTPO")
11 - filter(("EL"."FCDUEDATE">=SYSDATE@!-365 AND "EL"."FCSTQTY">0 AND "EL"."FCDUEDATE"<=SYSDATE@!+1200))
12 - access("E"."CUSTPO"="CO"."CUSTOMER_PO_REF")
12 - filter("E"."CUSTPO"="CO"."CUSTOMER_PO_REF")
15 - access("CO"."ID"="COL"."CUST_ORDER_ID")
16 - filter("CUSTOMER_PO_REF" IS NOT NULL)
17 - filter("COL"."DESIRED_SHIP_DATE">=SYSDATE@!-365)

insert_undo_10.2.0.2-2:

--------------------------------------------------------+-----------------------------------+
| Id  | Operation                      | Name           | Rows  | Bytes | Cost  | Time      |
--------------------------------------------------------+-----------------------------------+
| 0   | INSERT STATEMENT               |                |       |       |  274K |           |
| 1   |  SORT GROUP BY                 |                |   204 |   13K |  274K |  00:22:15 |
| 2   |   FILTER                       |                |       |       |       |           |
| 3   |    MERGE JOIN OUTER            |                |   75K | 5046K |  274K |  00:22:15 |
| 4   |     SORT JOIN                  |                |   69K | 2129K |  274K |  00:22:14 |
| 5   |      VIEW                      |                |   69K | 2129K |  274K |  00:22:14 |
| 6   |       FILTER                   |                |       |       |       |           |
| 7   |        SORT GROUP BY           |                |   69K | 3502K |  274K |  00:22:14 |
| 8   |         FILTER                 |                |       |       |       |           |
| 9   |          HASH JOIN             |                |  104M | 5326M |  2876 |  00:00:14 |
| 10  |           INDEX FAST FULL SCAN | EDI830_PK      |  105K | 2092K |    33 |  00:00:01 |
| 11  |           TABLE ACCESS FULL    | EDI830_LINES   |  236K | 7311K |   941 |  00:00:05 |
| 12  |     SORT JOIN                  |                |  6261 |  220K |   110 |  00:00:01 |
| 13  |      VIEW                      |                |  6261 |  220K |   108 |  00:00:01 |
| 14  |       SORT GROUP BY            |                |  6261 |  251K |   108 |  00:00:01 |
| 15  |        HASH JOIN               |                |  8184 |  328K |   106 |  00:00:01 |
| 16  |         TABLE ACCESS FULL      | CUSTOMER_ORDER |  7562 |  126K |    14 |  00:00:01 |
| 17  |         TABLE ACCESS FULL      | CUST_ORDER_LINE|  8550 |  200K |    92 |  00:00:01 |
--------------------------------------------------------+-----------------------------------+
Predicate Information:
----------------------
2 - filter("E"."FCDUEDATE">COALESCE("CO"."DESIRED_SHIP_DATE",SYSDATE@!-365))
6 - filter("EL"."RELID"=MAX("RELID"))
8 - filter(SYSDATE@!-365<=SYSDATE@!+1200)
9 - access("E"."CUSTPO"="EL"."CUSTPO")
11 - filter(("EL"."FCDUEDATE">=SYSDATE@!-365 AND "EL"."FCSTQTY">0 AND "EL"."FCDUEDATE"<=SYSDATE@!+1200))
12 - access("E"."CUSTPO"="CO"."CUSTOMER_PO_REF")
12 - filter("E"."CUSTPO"="CO"."CUSTOMER_PO_REF")
15 - access("CO"."ID"="COL"."CUST_ORDER_ID")
16 - filter("CUSTOMER_PO_REF" IS NOT NULL)
17 - filter("COL"."DESIRED_SHIP_DATE">=SYSDATE@!-365)

insert_undo_10.2.0.2-2:

--------------------------------------------------------+-----------------------------------+
| Id  | Operation                      | Name           | Rows  | Bytes | Cost  | Time      |
--------------------------------------------------------+-----------------------------------+
| 0   | INSERT STATEMENT               |                |       |       |  274K |           |
| 1   |  SORT GROUP BY                 |                |   204 |   13K |  274K |  00:22:15 |
| 2   |   FILTER                       |                |       |       |       |           |
| 3   |    MERGE JOIN OUTER            |                |   75K | 5046K |  274K |  00:22:15 |
| 4   |     SORT JOIN                  |                |   69K | 2129K |  274K |  00:22:14 |
| 5   |      VIEW                      |                |   69K | 2129K |  274K |  00:22:14 |
| 6   |       FILTER                   |                |       |       |       |           |
| 7   |        SORT GROUP BY           |                |   69K | 3502K |  274K |  00:22:14 |
| 8   |         FILTER                 |                |       |       |       |           |
| 9   |          HASH JOIN             |                |  104M | 5326M |  2876 |  00:00:14 |
| 10  |           INDEX FAST FULL SCAN | EDI830_PK      |  105K | 2092K |    33 |  00:00:01 |
| 11  |           TABLE ACCESS FULL    | EDI830_LINES   |  236K | 7310K |   941 |  00:00:05 |
| 12  |     SORT JOIN                  |                |  6261 |  220K |   110 |  00:00:01 |
| 13  |      VIEW                      |                |  6261 |  220K |   108 |  00:00:01 |
| 14  |       SORT GROUP BY            |                |  6261 |  251K |   108 |  00:00:01 |
| 15  |        HASH JOIN               |                |  8184 |  328K |   106 |  00:00:01 |
| 16  |         TABLE ACCESS FULL      | CUSTOMER_ORDER |  7562 |  126K |    14 |  00:00:01 |
| 17  |         TABLE ACCESS FULL      | CUST_ORDER_LINE|  8550 |  200K |    92 |  00:00:01 |
--------------------------------------------------------+-----------------------------------+
Predicate Information:
----------------------
2 - filter("E"."FCDUEDATE">COALESCE("CO"."DESIRED_SHIP_DATE",SYSDATE@!-365))
6 - filter("EL"."RELID"=MAX("RELID"))
8 - filter(SYSDATE@!-365<=SYSDATE@!+1200)
9 - access("E"."CUSTPO"="EL"."CUSTPO")
11 - filter(("EL"."FCDUEDATE">=SYSDATE@!-365 AND "EL"."FCSTQTY">0 AND "EL"."FCDUEDATE"<=SYSDATE@!+1200))
12 - access("E"."CUSTPO"="CO"."CUSTOMER_PO_REF")
12 - filter("E"."CUSTPO"="CO"."CUSTOMER_PO_REF")
15 - access("CO"."ID"="COL"."CUST_ORDER_ID")
16 - filter("CUSTOMER_PO_REF" IS NOT NULL)
17 - filter("COL"."DESIRED_SHIP_DATE">=SYSDATE@!-365)

According to the timing information, the SELECT statements with OPTIMIZER_FEATURES_ENABLE set to 10.1.0.4 completed in about 0.34 seconds, while the INSERT INTO completed in about 3 minutes and 16 seconds (I am not sure why I was seeing times of about 20 minutes before). With OPTIMIZER_FEATURES_ENABLE set to 10.2.0.2 with the default values for OPTIMIZER_INDEX_COST_ADJ and OPTIMIZER_INDEX_CACHING the SELECT statements completed in about 1.3 seconds and the INSERT INTO statements completed in about 2 minutes and 40 seconds.

Thanks for the explaination. Can’t wait to see your test cases to simulate this.
Would you mind validating my claim below?
While investigating this performance problem, if using hints (like NO_QUERY_TRANSFORMATION or NO_MERGE) result in original plan (for SELECT) being used and the estimated cost of the revised query is less than the estimated cost of the plan for the slow query, it is safe to conclude that it is an optimizer bug. This is because optimizer has clearly failed to select the cheapest plan to execute the query
While the estimated cost of the “fast” plan is visible above (157), the estimated cost of the “slow” plan is not visible. Would you mind sharing the estimated cost details for the “slow” plan above? Considering that the “slow” plan expects large TEMP usage as opposed to no TEMP usage expected by “fast” plan, I would be surprised if CBO has indeed considered the “slow” plan to be the cheapest plan.

Great idea to use the APPEND hint, and extra credit (bonus points) for mentioning one of the potential issues with implementing the APPEND hint (inserting above the high watermark, triggers, integrity constraints, etc.). In this particular case only about 900 rows were being inserted into the table (I intentionally withheld that piece of information so that people responding to this blog article, and specifically to those people who later read the blog article, would see a broader range of suggestions, and the potential warnings that should accompany the suggestions).

The ORDER BY clause was left in the INSERT statement for a couple of reasons:
* Initially I forgot to remove it when converting the SELECT statement into an INSERT INTO … SELECT type statement.
* I left the ORDER BY clause in the INSERT INTO … SELECT statement to take advantage of the nearly pre-sorted ordering of the rows in the table’s blocks. For the initial pass through the table when it is actually selected from, this will be slightly helpful. However, the INSERT INTO … SELECT is only the starting point of what actually ends up in that table. Immediately after the INSERT INTO … SELECT is performed, a recursive routine is performed that explodes up to 7 levels of bills of material for those parts, and the rows are inserted back into the same table. The CLUSTERING_FACTOR for the index that will eventually be created on the PART_ID column was a consideration.
* If I had removed the ORDER BY clause, it *might* be expected that the execution plan for the standard SELECT statement would differ from the execution plan for the INSERT INTO … SELECT SQL statement – so there is a risk that I might cause the performance to change.

I am once again impressed with the quality of the comments that are posted to this blog.