<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:georss="http://www.georss.org/georss" xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#" xmlns:media="http://search.yahoo.com/mrss/"
		>
<channel>
	<title>Comments on: Simple Query Generates Complex Execution Plan, the Mysterious 4063.88 Second Single Block Read Wait</title>
	<atom:link href="http://hoopercharles.wordpress.com/2010/01/22/simple-query-generates-complex-execution-plan-the-mysterious-4063-88-second-single-block-read-wait/feed/" rel="self" type="application/rss+xml" />
	<link>http://hoopercharles.wordpress.com/2010/01/22/simple-query-generates-complex-execution-plan-the-mysterious-4063-88-second-single-block-read-wait/</link>
	<description>Miscellaneous Random Oracle Topics: Stop, Think, ... Understand</description>
	<lastBuildDate>Mon, 13 May 2013 14:10:06 +0000</lastBuildDate>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.com/</generator>
	<item>
		<title>By: Log Buffer #176: a Carnival of the Vanities for DBAs &#124; The Pythian Blog</title>
		<link>http://hoopercharles.wordpress.com/2010/01/22/simple-query-generates-complex-execution-plan-the-mysterious-4063-88-second-single-block-read-wait/#comment-257</link>
		<dc:creator><![CDATA[Log Buffer #176: a Carnival of the Vanities for DBAs &#124; The Pythian Blog]]></dc:creator>
		<pubDate>Fri, 29 Jan 2010 19:11:38 +0000</pubDate>
		<guid isPermaLink="false">http://hoopercharles.wordpress.com/?p=940#comment-257</guid>
		<description><![CDATA[[...] From Charles Hooper comes this investigation: Simple Query Generates Complex Execution Plan, the Mysterious 4063.88 Second Single Block Read Wait. [...]]]></description>
		<content:encoded><![CDATA[<p>[...] From Charles Hooper comes this investigation: Simple Query Generates Complex Execution Plan, the Mysterious 4063.88 Second Single Block Read Wait. [...]</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Charles Hooper</title>
		<link>http://hoopercharles.wordpress.com/2010/01/22/simple-query-generates-complex-execution-plan-the-mysterious-4063-88-second-single-block-read-wait/#comment-239</link>
		<dc:creator><![CDATA[Charles Hooper]]></dc:creator>
		<pubDate>Sun, 24 Jan 2010 14:15:03 +0000</pubDate>
		<guid isPermaLink="false">http://hoopercharles.wordpress.com/?p=940#comment-239</guid>
		<description><![CDATA[Narendra, I think that your final sentence summarizes the entire thread.  The OP begins the thread with inaccurate information, and that causes confusion and inconsistencies that are brought up later in the thread.  But, in support of the OP, at times it is very difficult to determine what information may be relevant to solving a particular problem.  This is the case regardless of whether that problem is &quot;where did I leave my car keys&quot; or &quot;why is it taking a long time to retrieve 12 million rows from the database when I am fetching 15 rows at a time&quot;.  I suspect that in many cases prior experience helps a lot when trying to determine what information needs to be provided.  Suppose that the DBA on site (or possibly a consultant in the case of the OP) stated quite simply &quot;I have been a DBA for 25 years and have never needed to look at a raw 10046 trace file - just pass the file through TKPROF, or use explain plan.&quot;  In such a case, prior experience may make it very difficult to find any answers.  Jonathan definitely demonstrated considerable skill in that thread, and I think that I managed to extract a little bit of information from the OP to find some of the performance issues.

Gary, you make a good point that enabling a 10046 trace could have had an impact on the execution time.  The OP&#039;s disk sub-system seemed to be heavily loaded, causing long waits for disk reads, and the writing of the 10046 trace file could have further complicated the disk performance problem.  I had not thought about the OS suspending the process as a possible cause, but as I understand it, the session checks the server&#039;s current time, and then makes a disk read request - at that point operating system kernel level code is responsible for fetching the block - once the fetch completes the process again checks the server&#039;s current time to calculate the elapsed time of the disk read.  In the same response where I made the joke about the RAID 10 floppy array, I stated the following:

&quot;There is a possible explanation for the 4179109066us time for the SQL*Net message from client wait and the 4063882583us time for the db file sequential read. The clue is found on page 148 of &quot;Optimizing Oracle Performance&quot; in the pseudo-code for procedure wevent. What if the value of ela1 is less than ela0 due to a timer resolution problem with gettimeofday? If I recall correctly, I believe that it is possible that the operating system does not check the hardware clock on every time call (on some platforms), so it is theoretically possible that the operating system occasionally corrects its &quot;software&quot; clock with the hardware clock (on some Windows systems, this drift may not be corrected until a reboot is performed, or until the time is synchronized with the domain). 

This is just a guess, but a little mathematics with powers of 2, assuming that the OP is using a 32 bit system [note that if the software clock time drift was significant, the long duration wait time reported may be caused by an overflow condition that was caused by a negative value in the unsigned integer value):
4294967296us = 2^32us = 32 bit unsigned integer max value
4179109066us = OP&#039;s long SQL*Net time
4063882583us = OP&#039;s long single block read time

Browsing through Metalink finds a couple documents that may be of interest (yes these are old, but there may be others), that might identify other causes:
Bug No. 777326 USING POST WAIT THE CPU TIME STATISTICS ARE COMPLETELY WRONG
Bug No. 1704769 V$SESSION_WAIT.SECONDS_IN_WAIT SHOWS IMPOSSIBLY HIGH VALUES 
Bug No. 2717586 UNREALISTIC HUGE VALUES IN V$SYSTEM_EVENT AND V$SESSION_EVENT 

It is, however, odd that the tim= values were not thrown off in the same manner in that portion of the trace file.&quot;

With the above in mind, the actual wait time may have been very short, but the server&#039;s current time at the end of the wait may have been earlier than the server&#039;s current time at the start of the wait if the server re-synchronized its software clock with the hardware clock (or some external time source) between the time the disk read started and the disk read completed.]]></description>
		<content:encoded><![CDATA[<p>Narendra, I think that your final sentence summarizes the entire thread.  The OP begins the thread with inaccurate information, and that causes confusion and inconsistencies that are brought up later in the thread.  But, in support of the OP, at times it is very difficult to determine what information may be relevant to solving a particular problem.  This is the case regardless of whether that problem is &#8220;where did I leave my car keys&#8221; or &#8220;why is it taking a long time to retrieve 12 million rows from the database when I am fetching 15 rows at a time&#8221;.  I suspect that in many cases prior experience helps a lot when trying to determine what information needs to be provided.  Suppose that the DBA on site (or possibly a consultant in the case of the OP) stated quite simply &#8220;I have been a DBA for 25 years and have never needed to look at a raw 10046 trace file &#8211; just pass the file through TKPROF, or use explain plan.&#8221;  In such a case, prior experience may make it very difficult to find any answers.  Jonathan definitely demonstrated considerable skill in that thread, and I think that I managed to extract a little bit of information from the OP to find some of the performance issues.</p>
<p>Gary, you make a good point that enabling a 10046 trace could have had an impact on the execution time.  The OP&#8217;s disk sub-system seemed to be heavily loaded, causing long waits for disk reads, and the writing of the 10046 trace file could have further complicated the disk performance problem.  I had not thought about the OS suspending the process as a possible cause, but as I understand it, the session checks the server&#8217;s current time, and then makes a disk read request &#8211; at that point operating system kernel level code is responsible for fetching the block &#8211; once the fetch completes the process again checks the server&#8217;s current time to calculate the elapsed time of the disk read.  In the same response where I made the joke about the RAID 10 floppy array, I stated the following:</p>
<p>&#8220;There is a possible explanation for the 4179109066us time for the SQL*Net message from client wait and the 4063882583us time for the db file sequential read. The clue is found on page 148 of &#8220;Optimizing Oracle Performance&#8221; in the pseudo-code for procedure wevent. What if the value of ela1 is less than ela0 due to a timer resolution problem with gettimeofday? If I recall correctly, I believe that it is possible that the operating system does not check the hardware clock on every time call (on some platforms), so it is theoretically possible that the operating system occasionally corrects its &#8220;software&#8221; clock with the hardware clock (on some Windows systems, this drift may not be corrected until a reboot is performed, or until the time is synchronized with the domain). </p>
<p>This is just a guess, but a little mathematics with powers of 2, assuming that the OP is using a 32 bit system [note that if the software clock time drift was significant, the long duration wait time reported may be caused by an overflow condition that was caused by a negative value in the unsigned integer value):<br />
4294967296us = 2^32us = 32 bit unsigned integer max value<br />
4179109066us = OP&#8217;s long SQL*Net time<br />
4063882583us = OP&#8217;s long single block read time</p>
<p>Browsing through Metalink finds a couple documents that may be of interest (yes these are old, but there may be others), that might identify other causes:<br />
Bug No. 777326 USING POST WAIT THE CPU TIME STATISTICS ARE COMPLETELY WRONG<br />
Bug No. 1704769 V$SESSION_WAIT.SECONDS_IN_WAIT SHOWS IMPOSSIBLY HIGH VALUES<br />
Bug No. 2717586 UNREALISTIC HUGE VALUES IN V$SYSTEM_EVENT AND V$SESSION_EVENT </p>
<p>It is, however, odd that the tim= values were not thrown off in the same manner in that portion of the trace file.&#8221;</p>
<p>With the above in mind, the actual wait time may have been very short, but the server&#8217;s current time at the end of the wait may have been earlier than the server&#8217;s current time at the start of the wait if the server re-synchronized its software clock with the hardware clock (or some external time source) between the time the disk read started and the disk read completed.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Gary</title>
		<link>http://hoopercharles.wordpress.com/2010/01/22/simple-query-generates-complex-execution-plan-the-mysterious-4063-88-second-single-block-read-wait/#comment-238</link>
		<dc:creator><![CDATA[Gary]]></dc:creator>
		<pubDate>Sun, 24 Jan 2010 10:56:49 +0000</pubDate>
		<guid isPermaLink="false">http://hoopercharles.wordpress.com/?p=940#comment-238</guid>
		<description><![CDATA[&quot;the nearly 68 minute wait for the single block read &quot;. 
Haven&#039;t gone through the thread in exact detail, but it appears the initial report of 45 minutes was the first &#039;problem&#039; run and the trace was on a subsequent execution, which could have taken much longer.
Reading the thread, it appears the slow read was on a block in an UNDO tablespace. 

Didn&#039;t see any exact description of hardware, but I wonder if there is anything that would case a process to be completely de-prioritised /suspended by the OS.]]></description>
		<content:encoded><![CDATA[<p>&#8220;the nearly 68 minute wait for the single block read &#8220;.<br />
Haven&#8217;t gone through the thread in exact detail, but it appears the initial report of 45 minutes was the first &#8216;problem&#8217; run and the trace was on a subsequent execution, which could have taken much longer.<br />
Reading the thread, it appears the slow read was on a block in an UNDO tablespace. </p>
<p>Didn&#8217;t see any exact description of hardware, but I wonder if there is anything that would case a process to be completely de-prioritised /suspended by the OS.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Narendra</title>
		<link>http://hoopercharles.wordpress.com/2010/01/22/simple-query-generates-complex-execution-plan-the-mysterious-4063-88-second-single-block-read-wait/#comment-234</link>
		<dc:creator><![CDATA[Narendra]]></dc:creator>
		<pubDate>Sat, 23 Jan 2010 11:24:42 +0000</pubDate>
		<guid isPermaLink="false">http://hoopercharles.wordpress.com/?p=940#comment-234</guid>
		<description><![CDATA[Charles,

After posting my comment, I read the thread. Apart from technical details, I must say how much I appreciate you and Jonathan for your persistence. If what OP mentioned in the thread is true, I believe his(or her) organization was sitting on a time bomb. :) First, OP hardly knew anything about oracle and he (or she) claimed they do not have a DBA. Now coming to the problem, I would say the biggest root cause is poor design and/or poor capacity planning. The query is retrieving over 12 million records with 15 records at a time. For what ? Also, looking at the way OP was providing trace details, it appears he (or she) was executing some kind of procedure (from SQL*Plus) that involved many DML statements, before executing this query. It might be possible that OP was using some kind of logic to populate/update table data before querying it, which may result in stale statistics. Now, you did mention that OP might be using what appear to be some standard softwares but OP did not confirm the same. And as Hemant suggested (I guess), if the underlying tables were being updated simultaneously, I must say somebody has messed up the project big time.
I think this thread is a perfect example of how NOT to ask questions and how one can spend endless time, in vain, resolving performance issues due to lack of vital information.]]></description>
		<content:encoded><![CDATA[<p>Charles,</p>
<p>After posting my comment, I read the thread. Apart from technical details, I must say how much I appreciate you and Jonathan for your persistence. If what OP mentioned in the thread is true, I believe his(or her) organization was sitting on a time bomb. <img src='http://s0.wp.com/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' />  First, OP hardly knew anything about oracle and he (or she) claimed they do not have a DBA. Now coming to the problem, I would say the biggest root cause is poor design and/or poor capacity planning. The query is retrieving over 12 million records with 15 records at a time. For what ? Also, looking at the way OP was providing trace details, it appears he (or she) was executing some kind of procedure (from SQL*Plus) that involved many DML statements, before executing this query. It might be possible that OP was using some kind of logic to populate/update table data before querying it, which may result in stale statistics. Now, you did mention that OP might be using what appear to be some standard softwares but OP did not confirm the same. And as Hemant suggested (I guess), if the underlying tables were being updated simultaneously, I must say somebody has messed up the project big time.<br />
I think this thread is a perfect example of how NOT to ask questions and how one can spend endless time, in vain, resolving performance issues due to lack of vital information.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Charles Hooper</title>
		<link>http://hoopercharles.wordpress.com/2010/01/22/simple-query-generates-complex-execution-plan-the-mysterious-4063-88-second-single-block-read-wait/#comment-230</link>
		<dc:creator><![CDATA[Charles Hooper]]></dc:creator>
		<pubDate>Sat, 23 Jan 2010 01:57:41 +0000</pubDate>
		<guid isPermaLink="false">http://hoopercharles.wordpress.com/?p=940#comment-230</guid>
		<description><![CDATA[When you read the thread, you will find that the claim of there being 4 tables named Case, Item, Master, and Warehouse was quickly determined to be incorrect - for the very reasons that you identified in your comment.  

The RAID 10 floppy array was a joke that I wrote in the thread as an explanation for the nearly 68 minute wait for the single block read (DBA 1 to DBA2: &quot;quick, find and mount floppy #1765799 so that the query will finish executing&quot;).  

Considering the nearly 68 minute wait for the single block read written to the trace file, how could the OP in the thread explain the 45 minute query execution time.  Maybe the server was stuck in a time warp?  Maybe an accuracy warp?  Maybe someone was constantly re-syncing the clock on the server?]]></description>
		<content:encoded><![CDATA[<p>When you read the thread, you will find that the claim of there being 4 tables named Case, Item, Master, and Warehouse was quickly determined to be incorrect &#8211; for the very reasons that you identified in your comment.  </p>
<p>The RAID 10 floppy array was a joke that I wrote in the thread as an explanation for the nearly 68 minute wait for the single block read (DBA 1 to DBA2: &#8220;quick, find and mount floppy #1765799 so that the query will finish executing&#8221;).  </p>
<p>Considering the nearly 68 minute wait for the single block read written to the trace file, how could the OP in the thread explain the 45 minute query execution time.  Maybe the server was stuck in a time warp?  Maybe an accuracy warp?  Maybe someone was constantly re-syncing the clock on the server?</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Narendra</title>
		<link>http://hoopercharles.wordpress.com/2010/01/22/simple-query-generates-complex-execution-plan-the-mysterious-4063-88-second-single-block-read-wait/#comment-229</link>
		<dc:creator><![CDATA[Narendra]]></dc:creator>
		<pubDate>Fri, 22 Jan 2010 23:35:50 +0000</pubDate>
		<guid isPermaLink="false">http://hoopercharles.wordpress.com/?p=940#comment-229</guid>
		<description><![CDATA[The only reasons that are preventing me from saying &quot;This is PLAIN WRONG&quot; are
1) It is ORACLE after all ; no shortage of strange bugs
2) Your last statement (the concepts of which are Latin to me :) )

I have not read the forum post but I find it almost impossible to believe that the execution plan is of the query which is based on 4 &quot;tables&quot;. I do not think all of them are tables. At least, MASTER and CASE (or CASE_SIZE) look like (non-mergeable) views or materialized views. How else do you justify the presense of SYSTEM (steps 16 to 18), ITEM_SUPPLIER_COUNTRY(Steps 20, 24 to 27) ?
Also, the plan from step 22 onwards looks impossible with plain tables.]]></description>
		<content:encoded><![CDATA[<p>The only reasons that are preventing me from saying &#8220;This is PLAIN WRONG&#8221; are<br />
1) It is ORACLE after all ; no shortage of strange bugs<br />
2) Your last statement (the concepts of which are Latin to me <img src='http://s0.wp.com/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' />  )</p>
<p>I have not read the forum post but I find it almost impossible to believe that the execution plan is of the query which is based on 4 &#8220;tables&#8221;. I do not think all of them are tables. At least, MASTER and CASE (or CASE_SIZE) look like (non-mergeable) views or materialized views. How else do you justify the presense of SYSTEM (steps 16 to 18), ITEM_SUPPLIER_COUNTRY(Steps 20, 24 to 27) ?<br />
Also, the plan from step 22 onwards looks impossible with plain tables.</p>
]]></content:encoded>
	</item>
</channel>
</rss>
