The Basic 20 Things you have to know when managing RAC Database

 The Basic 20 Things you have to know when managing RAC Database

 

1. What is OHASD
2. What is OCR
3. What is VOTING DISK
4. What is CRS
5. What is CSSD
6. What is CTSSD
7. What is VIP
8. What is SCAN IP and Listener
9. What is OLOGGERD
10. What is EVMD
11. What is MDNSD
12. What is ONS
13. What is OPROCD
14. What is FAN
15. What is TAF
16. What is FCF
17. What is GCS(LMSn)
18. What is GES
19. What is GNPD
20. What is DISKMON

1. What is OHASD

Ohasd stands for Oracle High Availability Services Daemon. It's implemented via a new daemon process which also call ohasd on the OS layer. It's the lower part of the Oracle Clusterware stack, which consists of processes that facilitate cluster operations in RAC databases. This includes :

1. GPNPD (Grid Plug and Play) provides access to the Grid Plug and Play profile, and coordinates updates to the profile among the nodes of the cluster to ensure that all of the nodes have the most recent profile.
2. GIPC (Grid Interprocess Communication) is a helper daemon for the communication infrastructure
3. MDNS (Multicast Domain Name Service) allows DNS requests.
4. GNS (Oracle Grid Naming Service) is a gateway

Useful Commands:

• crsctl enable has --> To start has services after reboot
• crsctl disable has --> has services should not start after reboot
• crsctl config has --> Check configuration whether autostart is enabled or not
• cat /etc/oracle/scls/<Node_name>/root/ohasdstr --> Check whether it is enabled or not
• cat /etc/oracle/scls/<Node_name>/root/ohasdrun --> whether restart enabled if node fails

2. What is OCR, OLR

OCR stands for Oracle Cluster Registry. It holds information on it such as node membership (which nodes are part of this cluster), Software Version, Location of voting disk, Status of RAC Database, Listeners, Instances and services. OCR is placed in ASM or OCFS (Oracle Cluster File System).

ASM can be brought up only if we have access to OCR. But OCR is accessible only after the ASM is up. So how will CRS services coming up?

For this OLR (Oracle Local Registry) is there. This is a multiplexing of OCR file which was places in local file system.

OLR holds information on it such as CRS_HOME, GPnP details, active session, localhost version, OCR latest backup, Node name…

• cat /etc/oracle/ocr.loc --> OCR file Details

ocrconfig_loc=<+ASM_Location>

local_only=FALSE

• cat /etc/oracle/ocr.loc --> OLR file Details

olrconfig_log=<file_name_with_location.olr>

crs_home=<CRS_HOME_Location>

Useful Commands:

• ocrconfig -showbackup --> OCR file backup location
• ocrconfig -export <File_name_with_Full_Location.ocr> --> OCR Backup
• ocrconfig -restore <File_name_with_Full_Location.ocr> --> Restore OCR
• ocrconfig -import <File_name_with_Full_Location.dmp> --> Import Metadata specifically for OCR
• ocrcheck -details --> Gives the OCR info in detail
• ocrcheck -local --> Gives OLR infor in detail
• ocrdump -local <File_name_with_Full_Location.olr> --> Take the dump of OLR
• ocrdump <File_name_with_Full_Location.ocr> --> Take the dump of OCR

3. What is VOTING DISK

If a node joins cluster, if a node fails (may be evicted), if VIP need to be assigned in case of GNS is configured etc, voting disks comes into picture. Voting disk saves the information of which nodes were part on cluster. While starting the crs services, with the help of OCR, it will vote in the voting disk.

We need to take the backup of the voting disk periodically like our cron jobs. There are two different jobs done by voting disk:

• Dynamic - Heart beat information
• Static - Node information in the cluster

Useful commands:

• dd if=Name_Of_Voting_Disk=Name_Of_Voting_Disk_Backup --> Taking backup of voting disk
• crsctl query css votedisk --> Check voting disk details
• crsctl add css votedisk Path_to_voting_disk --> To add voting disk
• crsctl add css votedisk -force --> If the cluster is down
• crsctl delete css votedisk <File_Name_With_Password_With_File_Name> --> Delete voting disk
• crsctl delete css votedisk -force --> If the cluster is down
• crsctl replace votedisk <+ASM_Disl_Group> --> Replace the voting disk

4. What is CRS

CRSD stands for Cluster Resource Service Daemon. It is a process which is responsible to monitor, stop, start and failover the resources. This process maintains OCR and this is responsible for restarting resource when any failover is about to take place.

Useful Commands :

• crs_stat -t -v --> Check crs resources
• crsctl stat res -t --> Check in a bit detail view
• crsctl enable crs --> Enable automatic start of Services after reboot
• crsctl check crs --> Check crs services
• crsctl disable crs --> Diable automatic start of services after reboot
• crsctl stop crs --> Stop the crs services forcefully
• crsctl start crs --> To start the crs services on respective node
• crsctl start crs -excl --> To start the crs services in exclusive mode when you lost voting disk (You must replace the voting disk after you start the crs)
• crsctl stop cluster -all --> Stop the crs services on the cluster nodes
• crsctl start cluster -all --> Start the crs services on all the cluster nodes
• olsnodes --> Find all the nodes relative to the cluster
• oclumon manage -get master --> With this you will get master node information
• cat $CRS_HOME/crs/init/<node_name>.pid --> Find PID from which crs is running

5. What is CSSD

CSSD stands for Cluster Synchronization Service Daemon. This is responsible for communicating the nodes each other, This will monitor the heart beat messages from all nodes.

Useful Commands

• crsctl stop css --> For stopping the css
• crsctl disable css --> Disabling automatic startup after reboot

6. What is CTSSD

CTSSD stands for Cluster Time Synchronization Service Daemon. This service by default will be in observer mode. If time difference is there, it won't be taking any action. To run this service in active mode, you need to disable all the time synchronization services like NTP (Network Time Protocol). But, it is recommended to keep this service in observer mode.

Useful commands:

• cluvfy comp clocksync -n all -verbose --> To check the clock synchronization across all the nodes
• crsctl check ctss --> Check the service status and time offset in msecs

7. What is VIP

VIP stands for Virtual IP Address. Oracle uses VIP for Databases level access, Basically, when a connection comes from application end. Then using this IP address, I will connect. The VIP is used for RAC failover and RAC management. Suppose if IP for one node is down. As per protocol timeout, it need to wait 90 seconds to get a session. In this scenario, VIP comes into picture. If one of the VIP is down, connections will be routed only to the active node. The VIP must be ion the same address as public address.

Useful Commands:

• Srvctl start vip -n <node_name> -i <VIP_Name> --> To Start VIP
• Srvctl stop vip -n <node_name> -i <VIP_Name> --> To Stop VIP

8. What is SCAN IP and Listener

SCAN stands for Single Client Access Name. Scan IP's must be on the same sub net mask. Three SCAN IP's is a recommended number of count which redirects user sessions to the scan listeners. Load balancing on scan listener will be done by least_recently_loaded_algorithm.

When a connection is initiated from the application end, listener verifies the load balancing, and once it gets information, it will assign the connection to the node listener and user can do his transaction.

Main use is that we need not change the connection string in the application servers if any changes on the cluster are done like adding node, deleting node and other modification basing on requirement.

Useful Commands:

• srvctl config scan --> Retrieves scan listener configuration
• srvctl config scan listener --> List of scan listeners with Port Number
• srvctl add scan -n <node_name> --> Add a scan listener to the cluster
• srvctl add scan_listener -p <Desired_port_number> --> To add scan listener on specific port
• SQL> show parameter remote_listener --> Find the list of scan listeners
• srvctl stop scan --> Stops all scan listeners when ysed without -i option
• srvctl stop scan_listener --> Stops one or mode services in the cluster
• srvctl start scan --> To start the scan VIP
• Srvctl start scan_listener --> Start the scan listener
• Srvctl status scan --> Verify scan listener status
• Srvctl modify scan_listener --> Modify the scan listener
• Srvctl relocate scan_listener -i <Ordinal_Number> -n <node_name> --> Relocate the scan listener to another node

9. What is OLOGGERD

Ologgerd stands for cluster logger service Daemon. This is otherwise called as cluster logger service. This logger services writes the data in the master node, and chooses other nodes as standby. If any network issue occurs between the nodes, and if it is unable to contact the master, the other node takes ownership and chooses a node as standby node.

Useful Commands:

• oclumon manage -get master --> Find which is the master
• oclumon manage -get reppath --> Will get the path of the repository logs
• oclumon -get repsize -->This will give you the limitations on repository size
• oclumon showobjects --> Find which nodes are connected to loggerd
• oclumon dumpnodevirw --> This will give a detail view including system, topconsumers, processes, devices. Nics, filesystems status, protocol errors.
• oclumon dumpnodeview -n <node_1 node_2 node_3> -last "HH:MM:SS" --> You can view all the details in column from a specific time you mentionned
• oclumon dumpnodeviw allnodes - last "HH:MM:SS" --> If we need information from all nodes

10. What is EVMD

EVMD stands for Event Volume Manager Daemon. This handles event messaging for the processes. It send and receives actions regarding resource state changes to and from all other nodes in a cluster. This will take the helps of ONS (Oracle Notification Services)

Useful Commands:

• evmwatch -A -t "@timestamp @@" --> Get events generated in evmd
• evmpost -u "Message Here" -h <node_name> --> This will post message in evmd log in the mentioned node

11. What is MDNSD

Mdnsd stands for Multicast Domain Name Service Daemon. This process is used by gpndp to locate profiles in the cluster as well as by GNS to perform name resolutions. Mdnsd updates the pid file in init directory.

12. What is ONS

ONS stands for Oracle Notification Service. It will allow users to send SMS, emails, voice messages and Fax messages in an easy way. ONS will send the state of database, instance. This state information is used for load balancing. ONS will also communicate with daemons in other nodes for informing state of database.

Useful Commands:

• srvctl status nodeapps --> Status of nodeapps
• cat $ORACLE_HOME/opmn/conf/ons.config --> Check ons configuration
• $ORACLE_HOME_HOME/opmn/logs --> ONS logs will be in this location

13. What is OPROCD

OPROCD stands for Oracle Process Monitor Daemon. It monitors the system state of cluster nodes.

Useful Commands:

• CRS_HOME/oprocod stop --> To stop the processon single mode

14. What is FAN

FAN stands for Fast Application Notification. If any state change occurs in cluster/instance/node, an event is triggered by the event manager and it is propagated by ONS. The event is known as FAN event.

Useful Commands:

• onsctl ping --> Tocheck wheteher ons is running or not
• onsclt debug --> Will get detail view of ons
• onsctl start --> Start the daemon
• onsctl stop --> Stop the daemon

15. What is TAF

TAF stands for Transparent Application Failover. When any RAC node is down, the select statements need to failover to the active node. Insert, Delete, Update and also Alter session statements are not supported by TAF. Temporary objects and PL/SQL are lost during the failover.

There are two types of failover methods used in TAF:

1. Basic Failover: It will connect to single node and no overload will be there. End user experiences delay in completing the transaction
2. Preconnect Failover: It will connect to primary and backup node at a time.This offers faster failover. An overload will be

Useful Commands:

SELECT machine, failover_type, failover_method, fail_over, Count(*)

FROM gv$session GROUP BY machine,failover_type, failover; --> Check Failover Status

16. What is FCF

FCF stands for Fast Connection Failover. It is an application level failover process. This will automatically subscribes to FAN events and this will help in immediate reaction on the up and down events from the database cluster. All the failure applications are cleaned up immediately, so that the application will receive a failure message. After cleanup, If new connection is received then with load balancing, it will reach active node.

17. What is GCS

GCS stands for Global Cache Service. It caches the information of data blocks, and access privileges of various instances. Integrity is maintained by maintaining global access. It is responsible for transferring blocks from instance to another instance when needed.

18. What is GES

GES stands for Global Enqueue Service. It controls library and dictionary caches on all the nodes. GES manages transaction locks, table locks, library cache locks, dictionary cache locks and database locks.

19. What is GPNPD

GPNPD stands for Grid Plug aNd Play Daemon. A file is located in CRS_HOME/gpnp/<node_name>/profile/peer/profile.xml which is known as GPNP profile. And this profile consists of cluster name, hostname, network profiles with IP addresses, OCR. If we need to do any modifications for voting disk, profile will be updated.

20. What is DISKMON

Disk Monitor daemon continuously runs when ocssd starts and it monitors and perform I/O fencing for Exadata Storage Server (This term is named as Cell as  per Exadata). This process will run since the ocssd starts because exadata cell can be added to any cluster at any time.