System/Kernel Mode CPU Usage – What is Wrong with this Quote?

16 06 2010

June 16, 2010

While reading the June 2010 printing of the “Oracle Performance Firefighting” book I found an interesting paragraph that left me wondering if something more, or perhaps something else should be stated.  This quote is from page 104 (this is a long quote, required to not lose the context of the discussion topic - please excuse any typos):

“System time: This is when a core is spending time processing operating system kernel code. Virtual memory management, process scheduling, power management, or essentially any activity not directly related to a user task is classified as system time. From an Oracle-centric perspective, system time is pure overhead. It’s like paying taxes. It must be done, and there are good reasons (usually) for doing it, but it’s not under the control of the business—it’s for the government. Normally, Oracle database CPU subsystems spend about 5% to 40% of their active time in what is called system mode. If you’re from a non-Unix background, you may be more familiar with the term kernel mode or privileged mode, which is, in essence, system time.”

What, if anything, is wrong with the above quote from the book? 

The point of blog articles like this one is not to insult authors who have spent thousands of hours carefully constructing an accurate and helpful book, but instead to suggest that readers investigate when something stated does not exactly match what one believes to be true.  It could be that the author “took a long walk down a short pier”, or that the author is revealing accurate information which simply cannot be found through other resources (and may in the process be directly contradicting information sources you have used in the past).  If you do not investigate in such cases, you may lose an important opportunity to learn something that could prove to be extremely valuable.



5 responses

17 06 2010
Marcin Przepiorowski


As far as I know system time is calculated when kernel is doing any system call –
In that case if software is using any kernel interfaces (disk, network, memory allocation, etc) a system time is calculated and it
is related to program activity. Can it be named pure overhead ? I’m not sure but I’m open to discuss about it.


17 06 2010
Charles Hooper


Nice article. Just a couple of quotes from the article:

“Most operations interacting with the system require permissions not available to a user level process, e.g. I/O performed with a device present on the system, or any form of communication with other processes requires the use of system calls… On Unix, Unix-like and other POSIX-compatible operating systems, popular system calls are open, read, write, close, wait, exec, fork, exit, and kill. Many of today’s operating systems have hundreds of system calls. For example, Linux has 319 different calls, and FreeBSD has almost 500. Tools such as strace and truss allow a process to execute from start and report all system calls the process invokes, or can attach to an already running process and intercept any system call made by said process if the operation does not violate the permissions of the user.”

A quote from the same “Oracle Performance Firefighting” book, in this blog article indicates that the book author used strace to demonstrate that Oracle database used readv to read multiple blocks from disk, which the author described as being associated with a db file scattered read wait event. I believe that readv is considered a system call, and I would assume that call would motivate at least some CPU usage in kernel mode. To state that essentially any activity not directly related to a user task is considered kernel CPU time might be considered “missing the boat”.

Page 200 of the Expert Oracle Practices book shows interesting output that is related to this blog article. The output shows output from 64 bit Windows where a database instance was consuming (roughly per second):

User Time: 0.31S    Sys Time: 0.75S
User Time: 0.24S    Sys Time: 0.86S
User Time: 0.22S    Sys Time: 0.84S

That is roughly 80% of the CPU time consumed by the database instance in kernel mode. What happened during that time period? The instance was running a SQL statement that was performing 100% (or very close to it) logical I/O, requiring over an hour at 100% CPU utilization (100% of 1 CPU) to output the first row. The troubling part? In the next chapter this was the same SQL statement that was fixed in the next chapter with the fix results summary information appearing on page 313. This query that started at 1 hour, 6 minutes, and 44 seconds completed in a low of 2.65 seconds just by trying to help the Oracle optimizer understand the intent of the query, and was further dropped to 0.28 seconds by switching from STATISTICS_LEVEL=’ALL’ to STATISTICS_LEVEL=’TYPICAL’

In short, if you just throw up your hands when you see excessive kernel mode CPU usage, thinking that it is just like taxes, completely out of your control, you might be missing an opportunity.

My notes suggest that the following activities that needed to be performed for an Oracle database instance require kernel mode CPU time:
* network I/O
* disk I/O (especially with buffered file systems)
* timer calls (typically gettimeofday calls)
* memory access requests
* memory management
* process creation and termination (think session creation and termination)

It looks like my analysis seems to agree with what you stated above.

Any other discussion?

18 06 2010
Charles Hooper

It looks like Kevin Closson, who is far more familiar with this topic, set up a parallel blog article to dig further into the details. That is part of the point of this blog article series, as mentioned in the blue text in this article. If you read something that does not agree with your prior knowledge, do you:
* Stop reading.
* Insult the author, rather than discuss the written material.
* Assume that the author is wrong and keep reading.
* Assume that the author is right and keep reading.
* Stop, think, and understand (and probably set up a couple of test cases too).

Kevin’s parallel article on the topic may be found here.

18 06 2010
from google


The kernel mode concept is very easy to understand, even, for the beginners too. do you think the author is not aware of it ?

You had enforced to find something wrong. It looks there is a wording game here.

18 06 2010
Charles Hooper

“From Google”,

Please review the following before posting comments to this blog:

It is simply illogical to advance that kernel mode CPU usage is very easy to understand, and therefore suggest that the author’s knowledge is inferior to that of a beginner.

What an author shares in a book does not necessarily communicate the depth of knowledge that an author has about a particular sub-topic in the book. Some of the sections of the book appear as though the author planned to circle back to fill in the limited details provided in some of the paragraphs, while other sections show deep knowledge.

To be absolutely clear, I believe that this book’s author is very knowledgable about Oracle Databases, and demonstrates that repeatedly in the book. He mentions in the book that he worked with Cary Millsap at Oracle Corp (I believe in the same department), and that likely means he has at least 20 years of in-depth knowledge in the functioning of Oracle Databases. The author’s “Forecasting Oracle Performance” book did not appear to have the same pattern of drifting between the internals of modifying the hidden Oracle parameters in one page and then a couple of pages later to simply glossing over problems – essentially stating that “there is nothing the DBA can do when this happens.”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

Join 137 other followers

%d bloggers like this: