As per defined IP Addresses in /etc/hosts, Let’s find the same information using various commands available from GRID and OS.
Let’s verify the
same if Grid (cluster) picked up the correct IP’s.
NODE-1
=========
mgracsolsrv64bit1:/export/home/grid: cat /etc/hosts
#
# Internet host table
#
::1 localhost
127.0.0.1 localhost
192.168.56.99
mgsrv-dns mgsrv-dns.mgdom.com
loghost
#PUBLIC
192.168.56.20 mgracsolsrv64bit1 mgracsolsrv64bit1.mgdom.com
192.168.56.21 mgracsolsrv64bit2 mgracsolsrv64bit2.mgdom.com
#PRIVATE
192.168.05.01 mgracsolsrv64bit1-priv mgracsolsrv64bit1-priv.mgdom.com
192.168.05.02 mgracsolsrv64bit2-priv mgracsolsrv64bit2-priv.mgdom.com
#VIRTUAL
192.168.56.30 mgracsolsrv64bit1-vip mgracsolsrv64bit1-vip.mgdom.com
192.168.56.31 mgracsolsrv64bit2-vip mgracsolsrv64bit2-vip.mgdom.com
###################################################################################
1) To Find Cluster
Name
###################################################################################
$ /u01/app/11.2.0.1/grid/bin/olsnodes -c
mgrac-cluster
###################################################################################
2) Find the PUBLIC and
VIP Node name
###################################################################################
$ /u01/app/11.2.0.1/grid/bin/olsnodes -n -i
mgracsolsrv64bit1
1 mgracsolsrv64bit1-vip
mgracsolsrv64bit2
2 mgracsolsrv64bit2-vip
$ /u01/app/11.2.0.1/grid/bin/srvctl config nodeapps -a
VIP exists.:mgracsolsrv64bit1
VIP exists.: /mgracsolsrv64bit1-vip/192.168.56.30/255.255.255.0/e1000g0
VIP exists.:mgracsolsrv64bit2
VIP exists.: /mgracsolsrv64bit2-vip/192.168.56.31/255.255.255.0/e1000g0
$ /u01/app/11.2.0.1/grid/bin/crsctl stat res
ora.mgracsolsrv64bit1.vip -p |egrep
‘^NAME|TYPE|USR_ORA_VIP|START_DEPENDENCIES|SCAN_NAME|VERSION’
NAME=ora.mgracsolsrv64bit1.vip
TYPE=ora.cluster_vip_net1.type
START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network)
USR_ORA_VIP=mgracsolsrv64bit1-vip
VERSION=11.2.0.1.0
###################################################################################
3) To find Private IP
Details
###################################################################################
$ /u01/app/11.2.0.1/grid/bin/olsnodes -n -i -l -p
mgracsolsrv64bit1
1 192.168.5.1
mgracsolsrv64bit1-vip => (Middle one is Private IP – 192.168.5.1)
The Oracle
Interface Configuration Tool (oifcfg) is used define and administer network
interfaces such as the public and private interfaces.
———————————————————————————————————————————————
Oracle only
store network interface name and subnet ID in OCR, not the netmask. Oifcfg command can be used for such
change, oifcfg commands only require to run on 1 of the cluster node, not all. When netmask is changed,
the associated subnet ID is also changed.
Interfaces used
are : e1000g0 , e1000g1 , oifcfg would not show exact IP’s, instead it shows
SubnetID in OCR.
Subnet Info in Oracle Clusterware – OCR – To find out what’s in
OCR:
Both public and
private network information are stored in OCR.
$
/u01/app/11.2.0.1/grid/bin/oifcfg getif
e1000g0 192.168.56.0 global public
e1000g1 192.168.0.0 global cluster_interconnect —-> Private Interface
Note:
The first column is the network adapter name
The second column is the subnet ID
The third column is always “global” and should not be changed
The last column indicates whether it’s public or cluster_interconnect(private)
in Oracle Clusterware
Shows the available interfaces that you can configure with setif.
The –p parameter displays the type of interface which can be PRIVATE, PUBLIC or UNKNOWN.
The iflist command queries the operating system to find which network interfaces are present on this node.
$ ifconfig -a
...
...
e1000g1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4>
mtu 1500 index 3
==> Private IP
inet 192.168.5.1 netmask ffff0000 broadcast 192.168.255.255
...
$
/u01/app/11.2.0.1/grid/bin/oifcfg iflist -p -n
e1000g0 192.168.56.0 PUBLIC 255.255.255.0
e1000g1 192.168.0.0
PUBLIC 255.255.0.0 —–> Private interface on different Subnet
255.255.0.0
e1000g2 192.168.56.0 PUBLIC 255.255.255.0
Note :
oifcfg getif => Does not works, if Oracle
Clusterware (CRS) is not running.
$ ps -ef |grep
d.bin |wc -l
0
$ oifcfg getif
PRIF-10: failed to initialize the cluster registry
oifcfg iflist => Works, even if Oracle
Clusterware (CRS) is not running.
$ oifcfg iflist -p -n
e1000g0 192.168.56.0 PUBLIC 255.255.255.0
e1000g1 192.168.0.0 PUBLIC 255.255.0.0
$ cat /etc/hosts |grep 192.168.5.1
192.168.5.1 mgracsolsrv64bit1-priv mgracsolsrv64bit1-priv.mgdom.com
Note:
The first column is the network adapter name
The second column is the subnet ID
The third column indicates whether it’s private, public or unknown according to
RFC standard, it has NOTHING to do whether it’s used as
private or public network in Oracle Clusterware
The last column is
the netmask
$ netstat -i
Name Mtu Net/Dest
Address
Ipkts Ierrs Opkts Oerrs Collis Queue
lo0 8232 loopback
localhost
51795
0 51795 0
0 0
e1000g0 1500 mgracsolsrv64bit1
mgracsolsrv64bit1 6288
0 4673 0
0 0
e1000g1 1500 mgracsolsrv64bit1-priv mgracsolsrv64bit1-priv 121009
0 83308 0
0 0
e1000g2 1500 mgsrv-dns
mgsrv-dns
4493 0 201
0 0 0
$ netstat -in
Name Mtu
Net/Dest
Address Ipkts Ierrs Opkts
Oerrs Collis Queue
lo0
8232 127.0.0.0 127.0.0.1
52598 0 52598 0
0 0
e1000g0 1500 192.168.56.0 192.168.56.20
6356 0 4734
0 0 0
e1000g1 1500
192.168.0.0 192.168.5.1 123476 0 85456
0 0 0
e1000g2 1500 192.168.56.0
192.168.56.99 4562
0 202 0
0 0
mgracsolsrv64bit1:/export/home/grid: ifconfig -a
lo0:
flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232
index 1
inet 127.0.0.1 netmask ff000000
e1000g0:
flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index
2 ==> Public IP
inet 192.168.56.20 netmask ffffff00 broadcast
192.168.56.255
e1000g0:1:
flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500
index 2 ==>
SCAN IP 1
inet 192.168.56.81 netmask
ffffff00 broadcast 192.168.56.255
e1000g0:3:
flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500
index 2 ==>
SCAN IP 2
inet 192.168.56.82 netmask
ffffff00 broadcast 192.168.56.255
e1000g0:5:
flags=1040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4> mtu 1500
index 2 ==>
VIP
inet 192.168.56.30 netmask
ffffff00 broadcast 192.168.56.255
e1000g1:
flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index
3
==> Private IP
inet 192.168.5.1 netmask ffff0000 broadcast 192.168.255.255
e1000g2:
flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index
4
==> DNS Server IP
inet 192.168.56.99 netmask
ffffff00 broadcast 192.168.56.255
$ netstat -r
Routing Table: IPv4
Destination
Gateway
Flags
Ref Use Interface
--------------------
-------------------- ----- ----- ---------- ---------
default
mgsrv-dns
UG
1 0
192.168.0.0
mgracsolsrv64bit1-priv U
1 97 e1000g1
192.168.56.0
mgracsolsrv64bit1
U
1 90 e1000g0
192.168.56.0
mgsrv-dns
U
1 0 e1000g2
192.168.56.0
mgracsolsrv64bit1 U
1 0 e1000g0:1
-> SCANIP 1 -
192.168.56.81
192.168.56.0
mgracsolsrv64bit1
U
1 0 e1000g0:3
-> SCANIP 2 -
192.168.56.82
192.168.56.0
mgracsolsrv64bit1
U
1 0 e1000g0:5
->
VIP - 192.168.56.30
224.0.0.0
mgracsolsrv64bit1
U
1 0 e1000g0
localhost
localhost
UH 59
21132 lo0
$ netstat -rnv
IRE Table: IPv4
Destination
Mask Gateway
Device Mxfrg
Rtt Ref Flg Out In/Fwd
--------------------
--------------- -------------------- ------ ----- ----- ---
--- ----- ------
default
0.0.0.0
192.168.56.99
1500* 0 1
UG 0 0
192.168.0.0
255.255.0.0 192.168.5.1
e1000g1 1500* 0 1
U 100 0
192.168.56.0
255.255.255.0 192.168.56.20
e1000g0 1500* 0 1
U 93 0
192.168.56.0
255.255.255.0 192.168.56.99
e1000g2 1500* 0 1
U 0 0
192.168.56.0
255.255.255.0
192.168.56.20 e1000g0:1
1500* 0 1 U
0 0
192.168.56.0
255.255.255.0
192.168.56.20 e1000g0:3
1500* 0 1
U 0 0
192.168.56.0
255.255.255.0
192.168.56.20 e1000g0:5
1500* 0 1 U
0 0
224.0.0.0
240.0.0.0
192.168.56.20 e1000g0
1500* 0 1
U 0 0
127.0.0.1
255.255.255.255
127.0.0.1
lo0 8232* 0 59
UH 21725 0
###########################################################################################################
4) To Find SCAN IP
###########################################################################################################
$ nslookup mgrac-scan
Server: 192.168.56.99
Address: 192.168.56.99#53
Name: mgrac-scan.mgdom.com
Address: 192.168.56.82
Name: mgrac-scan.mgdom.com
Address: 192.168.56.83
Name: mgrac-scan.mgdom.com
Address: 192.168.56.81
$ dig mgrac-scan.mgdom.com +noall +answer
; <<>>
DiG 9.6-ESV-R8 <<>> mgrac-scan.mgdom.com +noall +answer
;; global options: +cmd
mgrac-scan.mgdom.com. 600
IN A 192.168.56.81
mgrac-scan.mgdom.com. 600
IN A 192.168.56.82
mgrac-scan.mgdom.com. 600
IN A 192.168.56.83
$
/u01/app/11.2.0.1/grid/bin/srvctl config scan
SCAN name:
mgrac-scan, Network: 1/192.168.56.0/255.255.255.0/e1000g0
SCAN VIP name: scan1, IP: /192.168.56.83/192.168.56.83
SCAN VIP name: scan2, IP: /192.168.56.81/192.168.56.81
SCAN VIP name: scan3, IP: /192.168.56.82/192.168.56.82
When /etc/hosts is
configured with SCAN names:
$ cat /etc/hosts |grep -i scan
# SCAN
192.168.56.81 mgracscan1
192.168.56.82 mgracscan2
192.168.56.83 mgracscan3
$ /u01/app/11.2.0.1/grid/bin/srvctl config scan
SCAN name: mgrac-scan, Network: 1/192.168.56.0/255.255.255.0/e1000g0
SCAN VIP name: scan1, IP: /mgracscan3/192.168.56.83
SCAN VIP name: scan2, IP: /mgracscan1/192.168.56.81
SCAN VIP name: scan3, IP: /mgracscan2/192.168.56.82
$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521
###################################################################################
Before You Start Grid
Installation Please follow below to configure the required IP Addresses.
###################################################################################
Even if you are
using a DNS,Oracle recommends that you add lines to the /etc/hosts file on each
node,specifying the public IP, VIP and private addresses.
Configure the
/etc/hosts file so that it is similar to the following example:
NODE-1
=========
mgracsolsrv64bit1:/export/home/grid: cat /etc/hosts
#
# Internet host table
#
::1 localhost
127.0.0.1 localhost
192.168.56.99
mgsrv-dns mgsrv-dns.mgdom.com
loghost
#PUBLIC
192.168.56.20 mgracsolsrv64bit1 mgracsolsrv64bit1.mgdom.com
192.168.56.21 mgracsolsrv64bit2 mgracsolsrv64bit2.mgdom.com
#PRIVATE
192.168.05.01 mgracsolsrv64bit1-priv mgracsolsrv64bit1-priv.mgdom.com
192.168.05.02 mgracsolsrv64bit2-priv mgracsolsrv64bit2-priv.mgdom.com
#VIRTUAL
192.168.56.30 mgracsolsrv64bit1-vip mgracsolsrv64bit1-vip.mgdom.com
192.168.56.31 mgracsolsrv64bit2-vip mgracsolsrv64bit2-vip.mgdom.com
=================================================================================
NODE-2
=========
mgracsolsrv64bit2:/export/home/grid: cat /etc/hosts
#
# Internet host table
#
::1 localhost
127.0.0.1 localhost
#PUBLIC
192.168.56.20 mgracsolsrv64bit1 mgracsolsrv64bit1.mgdom.com
192.168.56.21 mgracsolsrv64bit2 mgracsolsrv64bit2.mgdom.com
#PRIVATE
192.168.05.01 mgracsolsrv64bit1-priv mgracsolsrv64bit1-priv.mgdom.com
192.168.05.02 mgracsolsrv64bit2-priv mgracsolsrv64bit2-priv.mgdom.com
#VIRTUAL
192.168.56.30 mgracsolsrv64bit1-vip mgracsolsrv64bit1-vip.mgdom.com
192.168.56.31 mgracsolsrv64bit2-vip mgracsolsrv64bit2-vip.mgdom.com
=================================================================================
Public – Requires one
interface or network Adapter/card
=================================================================================
Determine the public host name for each node in the cluster.
1) For the public host name, use the primary host name of each node.
2) In other words, use the name displayed by the hostname command for example:
mgracsolsrv64bit1
3) Static IP address
4) Configured before installation for each node, and resolvable to that node
before installation.
5) It should be On the same subnet as all other public IP addresses, VIP
addresses, and SCAN addresses.
6) A public IP is registered on DNS
7) Assign One IP address with an associated host name (or network name)
registered in the DNS for the public interface.
8) Record the host name and IP address in the system hosts file also,
/etc/hosts.
9) Each node needs a public IP address This can be an interface bonded using
IPMP.
=================================================================================
Private – Requires one
interface or network Adapter/card
=================================================================================
Determine the private hostname for each node in the cluster.
1) This private hostname does not need to be resolvable through DNS and should
be entered in the /etc/hosts file.
2) The private IP should NOT be accessable to servers not participating in the
local cluster. (Only RAC Nodes should ping each other using Priv IP)
3) The private network should be on standalone dedicated switch(es).
4) The private network should NOT be part of a larger overall network topology.
5) The private network should be deployed on Gigabit Ethernet or better
6) It is recommended that redundant NICs are configured For Solaris either Sun
Trunking (OS based) or Sun IPMP (OS based)
More information: <<Note: 283107.1>>
7) IPMP in general. When IPMP is used for the interconnect: <<Note:
368464.1>>
NOTE: If IPMP is used for public
and/or cluster interconnect, critical merge patch 9729439 should be applied to
both Grid Infrastructure and RDBMS Oracle homes.
8) Static IP address
9) Configured before installation, but on a separate, private network, with its
own subnet, that is not resolvable except by other cluster member nodes.
10) A private IP only known to the servers in the RAC configuration, to be used
by the interconnect.
11) Configure the private interconnect network interface cards to have a
private node name and a private IP address.
12) This can be an interface bonded using IPMP.
13) Choose a private IP address that is in the address range 10.*.*.* or
192.168.*.*
Private 10.0.0.1 Hosts file
Private 10.0.0.2 Hosts file
14) The private IP address should be on a separate subnet than the public IP
address.
15) Oracle strongly recommends using a physically separate, private network
(You can use a vlan on a shared switch if you must).
16) You should ensure that the private IP addresses are reachable only by the
cluster member nodes.
17) Use Non Routable network addresses for private interconnect:
Class A: 10.0.0.0 to 10.255.255.255
Class B: 172.16.0.0 to 172.31.255.255
Class C: 192.168.0.0 to 192.168.255.255
http://www.subnet-calculator.com/
http://www.subnetonline.com/pages/subnet-calculators/ip-subnet-calculator.php
Or Google search –
“subnet calculator”
VIP – NOT required
separate Network adapter/card , it uses existing public interface.
===================================================================================
1) The virtual IP address and the network name must not be currently in use,
(But should not be accessible by a ping command or NOT Pingable)
2) The virtual IP address must be on the same subnet as your public IP address.
3) The virtual host name for each node should be registered with your DNS.
4) Static IP address
5) Configured before installation for each node, The IP address and host name
are currently unused (it can be registered in a DNS, but should not be
accessible by a ping command).
6) It should On the same subnet as all other public IP addresses, VIP
addresses, and SCAN addresses.
7) A virtual IP is registered on DNS, but NOT defined in the servers. It will
be defined later during Oracle Clusterware Install
8) Assign One virtual IP (VIP) address with an associated host name registered
in a DNS.
9) Record the host name and VIP address in the system hosts file, /etc/hosts.
===================================================================================
SCAN – VIP –
Define in corporate DNS (Domain Name Service), this again uses existing public
Interface
===================================================================================
1) Determine your cluster name. The cluster name should satisfy the following
conditions:
2) The cluster name is globally unique throughout your host domain.
3) The cluster name is at least 1 character long and less than 15 characters
long.
4) The cluster name must consist of the same character set used for host names:
single-byte alphanumeric
characters (a to z, A to Z, and 0 to 9) and hyphens (-).
5) Define the SCAN
in your corporate DNS (Domain Name Service), You must ask your network
administrator to create a single name,
that resolves to three IP addresses using a round robin algorithm.
6) The IP addresses must be on the same subnet as your public network in the
cluster.
7) SCAN VIPs must NOT be in the /etc/hosts file, it must be resolved by DNS.
$
/u01/app/11.2.0.1/grid/bin/olsnodes -c
mgrac-cluster
$ nslookup mgrac-scan
Server: 192.168.56.99
Address: 192.168.56.99#53
Name: mgrac-scan.mgdom.com
Address: 192.168.56.81
Name: mgrac-scan.mgdom.com
Address: 192.168.56.82
Name: mgrac-scan.mgdom.com
Address: 192.168.56.83
$ dig mgrac-scan.mgdom.com +noall +answer
; <<>>
DiG 9.6-ESV-R8 <<>> mgrac-scan.mgdom.com +noall +answer
;; global options: +cmd
mgrac-scan.mgdom.com. 600
IN A
192.168.56.83
mgrac-scan.mgdom.com. 600
IN A
192.168.56.81
mgrac-scan.mgdom.com. 600
IN A
192.168.56.82
$
/u01/app/11.2.0.1/grid/bin/srvctl config scan
SCAN name: mgrac-scan, Network: 1/192.168.56.0/255.255.255.0/e1000g0
SCAN VIP name: scan1, IP: /mgracscan3/192.168.56.83
SCAN VIP name: scan2, IP: /mgracscan1/192.168.56.81
SCAN VIP name: scan3, IP: /mgracscan2/192.168.56.82
$
/u01/app/11.2.0.1/grid/bin/srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521
Verify SCAN Ip’s
Registered with resource
===========================================
$ /u01/app/11.2.0.1/grid/bin/crsctl stat res ora.scan1.vip -p
|egrep ‘^NAME|TYPE|USR_ORA_VIP|START_DEPENDENCIES|SCAN_NAME|VERSION’
NAME=ora.scan1.vip
TYPE=ora.scan_vip.type
SCAN_NAME=mgrac-scan
START_DEPENDENCIES=hard(ora.net1.network) dispersion:active(type:ora.scan_vip.type)
pullup(ora.net1.network)
USR_ORA_VIP=192.168.56.83
VERSION=11.2.0.1.0
Refer few notes:
=================
http://technet.microsoft.com/en-us/library/cc958825.aspx
http://technet.microsoft.com/en-us/library/cc958832.aspx
How to Modify Private Network Information in Oracle Clusterware (Doc ID
283684.1)
How to Modify Public Network Information including VIP in Oracle Clusterware
(Doc ID 276434.1)
How to Modify SCAN Setting or SCAN Listener Port after Installation (Doc ID
972500.1)
No comments:
Post a Comment