Oracle 19c Database: High Priority Processes

 

In this best practice we used an Oracle configuration parameter to set database processes to high priority. The Linux operating system enables setting priorities on processes to ensure critical processes gain priority of the operating systems. 

Category	Oracle 19c Database
Product	Oracle 19c
Type of best practice	Performance Optimization
Day and value	Day 3, Fine Tuning

 Overview

The Oracle configuration parameter _high_priority_processes enables the database administrator to set related database processes to a higher priority. The following command shows the default value of the parameter:

_high_priority_processess = ’LMS*|LM1*|LM2*|LM3*|LM4*|LM5*|LM6*|LM7*|LM8*|LM9*’

• LMSn and LMn - relates to the global cache service process.

• VKTM – is the virtual keeper of time process and provides a wall clock time for measurements. 
• LGWR – is the log writer processes and writes data to the online redo logs. 

The LMSn and LMn process relates to the global cache service process. These processes receive, process, and send requests for the global cache service and the buffer cache resources. 

To optimize the database configuration, we configured the parameter to prioritize these processes:

_high_priority_processess = ’LMS*|VKTM|LGWR’

Changing the _high_priority_processess parameter does require a database restart. 

Recommendation

Updating the _high_priority_processess database parameter slightly increased performance across the following metrics:

• New Orders per Minute (NOPM) 
• Transactions per Minute (TPM) 
• PowerMax IOPS.
• Server CPU utilization. 
• DB File Sequential Read 
• Log File Parallel Write 

We recommend changing the _high_priority_processess parameter as a Day 3, Fine Tuning activity as it can provide the database with a slight performance boost. We also recommend testing this thoroughly before applying to production systems. Refer to Oracle support DOC: Doc ID 1373500.1

Implementation Steps

Use the following steps to update the parameter related to high priority processes:

1. Log in to the database node/VM (rp2vm2) as oracle user
2. Connect to sqlplus
3. Update the priority of intended processes

            SQL>  alter system set "_high_priority_processes"='LMS*|VKTM|LGWR' scope=spfile;

SQL> Exit 

1. Restart the database

               [oracle@rp2vm2 ~]$ srvctl stop database -d rdpp1
               [oracle@rp2vm2 ~]$ srvctl start database -d rdpp1

1. Verfiy the updated priority using the following steps
2. Connect to sqlplus

SQL> select ksppstvl from x$ksppi join x$ksppcv using (indx) where ksppinm='_high_priority_processes';

Follow these steps for all the eight database nodes/VMs (rp2vm2, rp2vm3 to rp2vm9). 

Increasing the performance of global cache service(LMSn) in Oracle

 

The global cache service (LMSn) processes copy consistent copies of blocks in the buffer cache to the requesting instance’s foregound process.

LMS also performs a rollback on uncommitted transactions in the requested block.

The number of LMS processes is 2, unless otherwise specified. This number is not sufficient for RAC databases with heavy transaction.

Because it is not sufficient, “gc current block 2-way” or “gc current block 3-way” wait events can be seen in the database.

The number of LMS processes is managed by the parameter GCS_SERVER_PROCESSES. 

The default value is 2 and the maximum value is 36. By increasing the value of this parameter, you can avoid wait events such as “gc current block 2-way”.

At the same time, block requests between intances in RAC databases can be completed in less time. This means serious performance gain.

Change the parameter:

1	SQL> alter system set GCS_SERVER_PROCESSES=10 scope=spfile;ile sid='*';

The database must be restarted after the parameter change.

GCS_SERVER_PROCESSES - LMS process - RAC

 

GCS_SERVER_PROCESSES

GCS_SERVER_PROCESSES specifies the number of background GCS server processes (LMSn and LMnn) to serve the inter-instance traffic among Oracle RAC instances.

If there is high Cache Fusion congestion on the system, the number of GCS server processes could be increased to reduce Cluster wait time. Additionally, the number of GCS server processes could be increased to reduce the application brownout time associated with Cluster reconfiguration and Dynamic Re-mastering (DRM).

Property	Description
Parameter type	Integer
Default value	Oracle calculates the default value as follows (in order of precedence): If CLUSTER_DATABASE is set to false, then 0 If Oracle ASM, then 1 If 1 - 3 CPUS, then 1 If 4 - 15 CPUs, then 2 If 128 or more CPUs and SGA is 100 GB or more, then (CPUs / 6). If the result includes a fraction, then the fraction is disregarded. Otherwise, the value is 2 + (CPUs / 32). If the result includes a fraction, then the fraction is disregarded. For example, if you have 20 CPUs, then 2 + (20 / 32) would equal 2 GCS processes. On NUMA-enabled systems with 32 or more CPUs, the value is rounded up to a multiple of the number of NUMA processor groups, with a limit of (CPUs / 4) rounded down to a multiple of the number of NUMA processor groups.
Modifiable	No
Modifiable in a PDB	No
Range of values	0 if Oracle RAC is disabled (CLUSTER_DATABASE is set to false) 1 to 100 if Oracle RAC is enabled (CLUSTER_DATABASE is set to true)
Basic	No
Oracle RAC	Multiple instances can have different values.

GCS server processes are only seen in an Oracle RAC environment.

Oracle RAC background processes

 

The GCS and GES processes, and the GRD collaborate to enable Cache Fusion. The Oracle RAC processes and their identifiers are as follows:

1. ACMS: Atomic Controlfile to Memory Service (ACMS)

In an Oracle RAC environment, the ACMS per-instance process is an agent that contributes to ensuring a distributed SGA memory update is either globally committed on success or globally aborted if a failure occurs.

2. GTX0-j: Global Transaction Process

The GTX0-j process provides transparent support for XA global transactions in an Oracle RAC environment. The database autotunes the number of these processes based on the workload of XA global transactions.

3. RMSn: Oracle RAC Management Processes (RMSn)

The RMSn processes perform manageability tasks for Oracle RAC. Tasks accomplished by an RMSn process include creation of resources related to Oracle RAC when new instances are added to the clusters.

4. RSMN: Remote Slave Monitor manages background slave process creation and communication on remote instances. These background slave processes perform tasks on behalf of a coordinating process running in another instance.

5. Lock Monitor Processes ( LMON)

• It Maintains GCS memory structures.
• Handles the abnormal termination of processes and instances.
• Reconfiguration of locks & resources when an instance joins or leaves the cluster are handled by LMON ( During reconfiguration LMON generate the trace files)
• It responsible for executing dynamic lock remastering every 10 mins ( Only in 10g R2 & later versions).
• LMON Processes manages the global locks & resources.
• It monitors all instances in cluster, primary for dictionary cache locks,library cache locks & deadlocks on deadlock sensitive on enqueue & resources.
• LMON also provides cluster group services.
• Also called Global enqueue service monitor.

6. Lock Monitor Services (LMS)

• LMS is most very active background processes.
• Consuming significant amount of CPU time. ( 10g R2 – ensure that LMS process does not encounter the CPU starvation).
• Its primary job is to transport blocks across the nodes for cache-fusion requests.
• If there is a consistent-read request, the LMS process rolls back the block, makes a Consistent-Read image of the block and then ship this block across the HSI (High Speed Interconnect) to the process requesting from a remote node.
• LMS must also check constantly with the LMD background process (or our GES process) to get the lock requests placed by the LMD process.
• Each node have 2 or more LMS processes.
• GCS_SERVER_PROCESSES –> no of LMS processes specified in init. ora parameter.
• Above parameter value set based on number of cpu’s ( MIN(CPU_COUNT/2,2))
• 10gR2, single CPU instance,only one LMS processes started.
• Increasing the parameter value,if global cache activity is very high.
• Also called the GCS (Global Cache Services) processes.

Internal View: X$KJMSDP

7. Lock Monitor Daemon Process ( LMDn)

• LMD process performs global lock deadlock detection.
• Also monitors for lock conversion timeouts.
• Also sometimes referred to as the GES (Global Enqueue Service) daemon since its job is to manage the global enqueue and global resource access.
• LMD process also handles deadlock detection and remote enqueue requests.
• Remote resource requests are the requests originating from another instance.

Internal View: X$KJMDDP

8. LCKn ( Lock Process)

• Manages instance resource requests & cross instance calls for shared resources.
• During instance recovery,it builds a list of invalid lock elements and validates lock elements.

9. DIAG (Diagnostic Daemon)

• Oracle 10g – this one new background processes ( New enhanced diagnosability framework).
• Regularly monitors the health of the instance.
• Also checks instance hangs & deadlocks.
• It captures the vital diagnostics data for instance & process failures.

Noteworthy changes in Oracle RAC 18c and 19c

 

Customers monitoring Oracle RAC Databases running on Oracle 18c, Oracle19c with the latest RUs will notice the following changes. These are documented here for information only and there is no manual action to be taken.

New RS process

$ps -ef |grep rs00 |grep -v grep
S oracle 438809   1 0 20 19 0  - 64439 964916 pipe_wait     18:09 ?    00:00:00 oracle+ASM1_rs00_anair1 

Starting with Oracle 18c and higher releases, RMV process has been renamed to RS. RS process helps LMS* processes during remastering. Please refer to my blog post for more details on remastering. Information about the new rs00 process is currently missing from the Oracle documentation as of today (22-Nov-2019). The documentation is planned to be updated in the next refresh cycle, until then the current description of RMV processes in the documentation can be used to understand the functionality of RS process.

LMS is not running in Higher Priority

Customers may notice that the LMS* processes are not running in high priority mode. For example, in Oracle RAC 12c, ps command shows that LMS process is running in "RR" mode

$ ps -o user,pid,cls,priority,cmd -e |grep -v grep|egrep '^USER|lms'|grep -v ASM 

USER       PID CLS PRI CMD
racusr   12912  RR  -2 ora_lms0_acct2

While in Oracle RAC 19c, the same Oracle RAC 19c LMS process is running in "TS" mode

$ ps -o user,pid,cls,priority,cmd -e |grep -v grep |egrep '^USER|lms'|grep -v ASM
USER       PID CLS PRI CMD
racusr   341887  TS  20 ora_lms0_V19c1
racusr   341889  TS  20 ora_lms1_V19c1
racusr   341891  TS  20 ora_lms2_V19c1

However checking the Oracle Database alert log shows that the LMS was indeed started in higher priority

LMS0 started with pid=24, OS id=341887 at elevated (RT) priority

This discrepancy is caused due to a change in the architecture whereby Oracle RAC LMS process now runs as multiple threads. The command therefore to get the details of individual threads and their scheduling class is

$ps -eLo user,pid,cls,priority,cmd -e |grep -v grep |grep 'lms'|grep -v ASM

USER      PID   CLS  PRI  CMD
racusr   341887  TS  20 ora_lms0_V19c1
racusr   341887  RR  -2 ora_lms0_D19c1
racusr   341887  TS  20 ora_lms0_D19c1
racusr   341887  TS  20 ora_lms0_D19c1
racusr   341889  TS  20 ora_lms1_D19c1
racusr   341889  RR  -2 ora_lms1_D19c1
racusr   341889  TS  20 ora_lms1_D19c1
racusr   341889  TS  20 ora_lms1_D19c1
racusr   341891  TS  20 ora_lms2_D19c1
racusr   341891  RR  -2 ora_lms2_D191c
racusr   341891  TS  20 ora_lms2_D19c1
racusr   341891  TS  20 ora_lms2_D19c1

Lot more information about LMS process can be found in My Oracle Support Note 558185.1 **(requires login)

USM processes

It was a coincidence that during my visit to a customer site, I was asked about these processes. Incidentally, Liron also reached out to me that same evening to ask about these USM processes. The following ps command can be used to see the details of these processes as shown below.

ps -ef |grep -i USM
root      6931     2  0 Sep30 ?        00:00:02 [UsmGeneral]
root      6932     2  0 Sep30 ?        00:00:02 [UsmGeneral]
root      6933     2  0 Sep30 ?        00:01:49 [UsmMonitor]
root      6934     2  0 Sep30 ?        00:00:02 [UsmGeneral]
root      6939     2  0 Sep30 ?        00:00:02 [UsmGeneral]
root      6943     2  0 Sep30 ?        00:00:02 [UsmGeneral]
root      7333     2  0 Sep30 ?        00:00:00 [USM:RD:02]
root      7335     2  0 Sep30 ?        00:00:00 [USM:RD:01]
root      7337     2  0 Sep30 ?        00:01:49 [UsmMonitor]
root      7338     2  0 Sep30 ?        00:00:00 [USM:RD:00]
root      7340     2  0 Sep30 ?        00:00:00 [usmfqp1]
root      7341     2  0 Sep30 ?        00:00:00 [usmfqp0]
root      7343     2  0 Sep30 ?        00:01:47 [UsmMonitor]
root      7358     2  0 Sep30 ?        00:00:00 [USM:RD:03]
root      7374     2  0 Sep30 ?        00:00:00 [USM:RD:04]
...
...
root      7665     2  0 Sep30 ?        00:00:00 [USM:WR:29]
root      7666     2  0 Sep30 ?        00:00:00 [USM:WR:30]
root      7667     2  0 Sep30 ?        00:00:00 [USM:WR:31]

These processes are created for various functions in the ACFS, ADVM and OKS driver layer. They are created on demand, generally idle and only intervene during specific events. There is no user action needed at this time. Also note that the USM processes are not involved in the Oracle RAC Global Cache Management, I added this section here so there is a centralized location for changes that customers can refer to while working with Oracle RAC Database.