Huge CPU load due to high %SYS usage: Oracle
Came across one of the interesting performance issues
related to System CPU(%System) going very high< better to call Kernel
CPU> on Oracle Db server box.After
lot of hard work and tracing,
It was realized that the Oracle as a part of
system call calling "acpi_pm_read".
Oracle
database relies on gettimeofday(2) for timing everything from I/O
calls to latch sleeps.The preferred clock source is the Time Stamp Counter
(TSC) but in my case it was acpi_pm_read.
Such
context switches consumed lot of CPU and hence why the %Kernel CPU went high.
Just
a brief:
Multiprocessor
systems such as NUMA or SMP have multiple instances of clock sources. The way
clocks interact among themselves and the way they react to system events, such
as CPU frequency scaling or entering energy economy modes, determine whether
they are suitable clock sources for the real-time kernel.
During
boot time the kernel discovers the available clock sources and selects one to
use. The preferred clock source is the Time Stamp Counter (TSC), but if it
is not available the High Precision Event Timer (HPET) is the second best
option. However, not all systems have HPET clocks and some HPET clocks can be
unreliable.
In
the absence of TSC and HPET, other options include the ACPI Power Management
Timer (ACPI_PM), the Programmable Interval Timer (PIT) and the Real Time Clock
(RTC).
The
last two options are either costly to read or have a low resolution (time
granularity), therefore they are sub-optimal for the real-time kernel.
In
my case as i mentioned the read call shows acpi_pm_read call which means it was
using the non-preferred timer< ACPI Power Management Timer
(ACPI_PM)>.
Best
of the lot is TSC and most of the preferred system uses the same.
You
can always see what your hardware system is using by below command. Tsc is the
likely outcome.
cat
/sys/devices/system/clocksource/clocksource0/current_clocksource
Once
set %Kernel CPU went back normal.
Note: This was my observation, It may not be same for everyone. So be careful.
