Preparation for Oracle 11gR2 on Enterprise Linux 64 bit install

Here is the procedure I used for my last 11gr2 install.

Requirements for Installing Oracle 11g r2 RDBMS on Oracle Unbreakable Linux

Document sources for procedure:
Requirements for Installing Oracle 11gR2 on RHEL 5 on IBM: Linux [ID 1306889.1]
How to Configure the DNS Server for SCAN VIP on Linux for 11gR2 [ID 1107295.1]
Defining a "default RPMs" installation of the RHEL OS [ID 376183.1]
Upon startup of Linux database get ORA-27102: out of memory Linux [ID 301830.1]
RAC Assurance Support Team: RAC Starter Kit and Best Practices (Linux) [ID 811306.1]

You can obtain more information about the Oracle Unbreakable Enterprise Kernel for
Linux at the following URL:
http://www.oracle.com/us/technologies/linux/index.html
The Oracle Unbreakable Enterprise Linux kernel installs directly on top of Oracle
Linux 5, or Red Hat Enterprise Linux 5, starting with Update 5, so you are not
required to upgrade to a new major release of the operating system to obtain the
benefits and features of this new kernel. You can obtain additional information and
download the Oracle Unbreakable Enterprise Kernel for Linux at the following URL:
http://public-yum.oracle.com/
Oracle Linux 5
http://oss.oracle.com/el5/oracle-validated/
If the Linux distribution is Oracle Linux, or Red Hat Enterprise Linux, and you are an Unbreakable Linux customer, then you can complete most preinstallation
configuration tasks by using the Oracle Validated Configurations Setup RPM,
available from the Unbreakable Linux Network (ULN), or available on the Oracle
Linux DVD disks. When it is installed, the Oracle Validated Configuration RPM does the following:Automatically installs any additional packages needed for installing Oracle Grid Infrastructure and Oracle Database.

Creates an oracle user, and creates the oraInventory (oinstall) and OSDBA
(dba) groups for that user

Sets and verifies sysctl.conf settings, system startup parameters, user limits,
and driver parameters to values based on recommendations from the Oracle

Validated Configurations program

To become an Unbreakable Linux Network customer, contact your sales representative
or purchase a license from the Unbreakable Linux store:
https://shop.oracle.com/product/oraclelinux
To register your server on the Unbreakable Linux Network or to find out more
information, go to:
https://linux.oracle.com
If you are using Oracle Linux 4.7 and later, or Oracle Linux 5.2 and later, then the
Oracle Validated RPM is included on the installation media.

Installing the Oracle Validated Configuration RPM from Unbreakable Linux Network
Use the following procedure to subscribe to Oracle Unbreakable Linux channels and to
add the Oracle Software for the Enterprise Linux channel that distributes the Oracle
Validated Configurations Setup RPM:

1. Complete a default Oracle Linux workstation installation or a default Red Hat
Enterprise Linux installation.
2. Register the server with the Unbreakable Linux Network (ULN). By default, you
are registered for the Oracle Linux Latest channel for the operating system and
hardware.
3. Log in to ULN at:
https://linux.oracle.com
4. Click the Systems tab, and in the System Profiles list, select a registered server. The
System Details window opens and displays the subscriptions for the server.
5. From the Available Channels list, select the _base and _patch channels
corresponding to your Oracle Linux distribution. For example, if your distribution
is Oracle Linux 5 Update 5 for x86_64, then select the following:
Oracle Linux 5 Update 5 installation media copy (x86_64)
Oracle Linux 5 Update 5 Patch (x86_64)
6. Click Subscribe.
7. From a terminal session, as a root user, enter the following command:
# up2date --nox --show-channels
You should see output indicating that you have subscribed to the Oracle Linux
channel, for example:
el5_u5_i386_base
el5_u5_x86_64_patch
8. Open a terminal session as a root user, and install the Oracle Validated
Configurations Setup RPM with up2date using the following command:
# up2date --install oracle-validated
Oracle Linux automatically creates a standard (not role-allocated) Oracle
installation owner and groups, and sets up other kernel configuration settings as
required for Oracle installations.
Check the Oracle Validated Configuration RPM log file to
review the system configuration changes:
/var/log/oracle-validated/results/orakernel.log

Minimum Hardware Requirements

a.) At least 4.0 GB (4096 MB) of physical RAM
b.) Swap disk space proportional to the system's physical memory as follows:

grep MemTotal /proc/meminfo 32940856 kB 32 gb
grep SwapTotal /proc/meminfo 34996216 Kb 32 gb

RAM Swap Space
Between 4 GB and 8 GB 2 x RAM
Between 8 GB and 32 GB 1.5 x RAM
greater than 32 Gb 32 Gb

c.) approximately 5.5 GB of disk space for the Oracle Grid Infrastructure for a Grid
Home for Cluster and ASM Home (more than 1GB and less than 2Tb of disk space) in the /tmp directory.
d.) approximately 4.6 GB of local disk space for the database software.
e.) approximately 1.7 GB of disk space for a preconfigured database that uses file system storage (optional)


SELinux may need to be disabled Please refer Note 454196.1 for more information.
To check whether SELinux is Enabled or Disabled in a system, please refer Note 432988.1
To disable SELinux, please refer Note 457458.1
vi /etc/selinux/config
SELINUX=disabled


RPM Packages
The following packages are required to be installed to have a supported to installation. It is recommended to choose a "default list of RPMs" listed in the document taken from 11gR2 linux install pdf chapter 2.
Asianux Server 3, Oracle Linux 5, and Red HatEnterprise Linux 5
binutils-2.17.50.0.6
compat-libstdc++-33-3.2.3
elfutils-libelf-0.125
elfutils-libelf-devel-0.125
elfutils-libelf-devel-static-0.125
gcc-4.1.2
gcc-c++-4.1.2
glibc-2.5-24
glibc-common-2.5
glibc-devel-2.5
glibc-headers-2.5
kernel-headers-2.6.18
ksh-20060214
libaio-0.3.106
libaio-devel-0.3.106
libgcc-4.1.2
libgomp-4.1.2
libstdc++-4.1.2
libstdc++-devel-4.1.2
make-3.81
numactl-devel-0.9.8.i386
sysstat-7.0.2

Note:
Starting with Oracle Database 11g Release 2 (11.2.0.2), all the 32-bit
packages, except for gcc-32bit-4.3, listed in the following
table are no longer required for installing a database on Linux
x86-64. Only the 64-bit packages are required. However, for any
Oracle Database 11g release prior to 11.2.0.2, both the 32-bit and
64-bit packages listed in the following table are required.
If you are using Oracle Unbreakable Enterprise Kernel, then all
required kernel packages are installed as part of the Oracle
Unbreakable Enterprise Kernel installation.

For example: rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n" | grep glibc-devel.
# rpm --query --queryformat "%{NAME}-%{VERSION}.%{RELEASE} (%{ARCH})\n" binutils compat-db compat-libstdc++ control-center gcc gcc-c++ glibc glibc-devel glibc-headers glibc-common ksh libaio libgcc libgnome libgnomeui libgomp libstdc++ libXp make sysstat
up2date -i libXp
up2date -i glibc-headers
up2date -i gcc
up2date -i gcc-c++

up2date -i glibc-devel --arch=i386
up2date -i libaio-devel --arch=i386
up2date -i unixODBC
up2date -i unixODBC-devel
up2date -i unixODBC --arch=i386
up2date -i unixODBC-devel --arch=i386
up2date -i libXp --arch=i386
up2date -i unixODBC-devel --arch=x86_64
up2date -i oracleasm-support oracleasmlib oracleasm-`uname -r`

cd /var
mkdir /var/oracle_packages
tar -C /var/oracle_packages/ -xvf /var/oracle_packages/oracle_linux_packages.tar

rpm -i --nodeps /var/oracle_packages/cvuqdisk-1.0.7-1.rpm
rpm -i --nodeps /var/oracle_packages/elfutils-libelf-devel-0.137-3.el5.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/elfutils-libelf-devel-static-0.137-3.el5.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/gcc-4.1.2-48.el5.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/gcc-c++-4.1.2-48.el5.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/glibc-devel-2.5-49.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/glibc-devel-2.5-49.i386.rpm
rpm -i --nodeps /var/oracle_packages/glibc-headers-2.5-49.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/kernel-headers-2.6.18-194.el5.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/libaio-devel-0.3.106-5.i386.rpm
rpm -i --nodeps /var/oracle_packages/libaio-devel-0.3.106-5.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/libgomp-4.4.0-6.el5.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/libstdc++-devel-4.1.2-48.el5.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/numactl-devel-0.9.8-11.el5.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/oracleasm-2.6.18-194.el5-2.0.5-1.el5.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/oracleasmlib-2.0.4-1.el5.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/oracleasm-support-2.1.3-1.el5.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/strace-4.5.18-5.el5_4.1.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/sysstat-7.0.2-3.el5.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/unixODBC-2.2.11-7.1.i386.rpm
rpm -i --nodeps /var/oracle_packages/unixODBC-2.2.11-7.1.x86_64.rpm
rpm -i --nodeps /var/oracle_packages/unixODBC-devel-2.2.11-7.1.i386.rpm
rpm -i --nodeps /var/oracle_packages/unixODBC-devel-2.2.11-7.1.x86_64.rpm

Creating Grid and Oracle Users and directory for oracle software
1) Create grid user
a. groupadd –g 501 oinstall
b. groupadd –g 502 dba
c. groupadd –g 502 asmadmin
d. groupadd –g 503 asmdba
e. groupadd –g 504 asmoper
f. useradd –u 504 -g500 -G501,502,503,504 -s /bin/bash -d /home/grid grid
i. or useradd -u 504 -g oinstall -G asmadmin,asmdba,asmoper grid
g. passwd grid < keypass >
2) Create oracle user
a. useradd –u 500 -g500 -G501,503 -s /bin/bash -d /home/oracle oracle
i. or useradd -u 500 -g oinstall -G dba,asmdba oracle
b. passwd oracle < keypass >
3) mkdir -p /u00/app/
a. chown -R grid:oinstall /u00/app/
b. chmod -R 775 /u00/app/
c. mkdir /u00/app/oracle
d. mkdir /u00/app/oracle/zipfiles
e. mkdir /u00/app/oracle/cluster_verify_util
4) ssh presence setup
a. NoteID: SSH without a password [ID 464238.1]
b. Can also use /u00/app/grid/grid/sshsetup/sshUserSetup.sh
i. sshUserSetup.sh -hosts "host1 host2" -user grid -advanced
c. Also you can let installer setup ssh
d. If doing ssh setup manually here are the steps
i. as oracle and grid cd $HOME
ii. mkdir .ssh
iii. chmod 700 .ssh
iv. cd .ssh
v. ssh-keygen -t rsa
vi. ssh-keygen -t dsa
vii. scp ~/.ssh/id_dsa.pub grid@host1:~/.ssh/id_dsa.pub
viii. scp ~/.ssh/id_rsa.pub grid@host1:~/.ssh/id_rsa.pub
ix. ssh grid@carz0dbxd02
x. cd .ssh
xi. cat id_dsa.pub >> authorized_keys
xii. cat id_rsa.pub >> authorized_keys
xiii. chmod 640 authorized_keys

Kernel Settings
Modify your kernel settings in /etc/sysctl.conf as follows. If the current value for any parameter is higher than the value listed in this table, do not change the value of that parameter. Range values (such as net.ipv4.ip_local_port_range) must match exactly. Use the below cat commands to validate the values.

Automatic Memory Management
Starting with Oracle Database 11g, the Automatic Memory Management feature
requires more shared memory (/dev/shm)and file descriptors. The size of the shared
memory must be at least the greater of the MEMORY_MAX_TARGET and MEMORY_
TARGET parameters for each Oracle instance on the computer.
Note: The MEMORY_MAX_TARGET and MEMORY_TARGET parameters
cannot be used when the LOCK_SGA parameter is enabled, or with
HugePages on Linux.

In dev we are using 12928Mfor database memory, in prod 28928M
Prod
memory_max_target 28928M
memory_target 28928M
Dev
memory_max_target 12928M
memory_target 12928M

To determine the amount of shared memory available, enter the following command:
# df -h /dev/shm/

Prod
Filesystem Size Used Avail Use% Mounted on
tmpfs 36G 17G 19G 48% /dev/shm

Dev
Filesystem Size Used Avail Use% Mounted on
tmpfs 16G 6.5G 9.4G 41% /dev/shm


cat /proc/sys/net/core/rmem_default
cat /proc/sys/net/core/rmem_max
cat /proc/sys/net/core/wmem_default
cat /proc/sys/net/core/wmem_max
cat /proc/sys/kernel/shmmax
cat /proc/sys/kernel/shmmni
cat /proc/sys/kernel/shmall
cat /proc/sys/fs/file-max
cat /proc/sys/kernel/sem
cat /proc/sys/net/ipv4/ip_local_port_range

kernel.shmall = 6291456
kernel.shmall = 3309568 ( can go as low as this since dev has 12 gb memory target )
kernel.shmall = 4294967296 ( current set at this but this appears to high )
Set shmall equal to the sum of all the SGAs on the system, divided by the page size, getconf PAGE_SIZE 4096

kernel.shmmax = 68719476736 ( 1/2 of physical RAM )
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
If these parameters need to be modified, edit /etc/sysctl.conf. To activate these new settings into the running kernel space, run the “sysctl –p” command as this will propagate the changes to the /proc/sys files listed above. If you modify the individual files, and not sysctl.conf, the changes will be lost at the next reboot.
Network changes require reboot or restart of network service: /etc/rc.d/init.d/network restart
Examine /etc/sysctl.conf for the following entries:
#cat /etc/sysctl.conf

limits.conf
- Add the following settings to /etc/security/limits.conf

oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle hard stack 10240
#
grid hard nofile 65536
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024

pam.d/login
- Add or edit the following line in the /etc/pam.d/login file, if it does not already exist:
session required pam_limits.so

# pam_selinux.so close should be the first session rule
session required pam_selinux.so close
session include system-auth
session required pam_loginuid.so
session optional pam_console.so
# pam_selinux.so open should only be followed by sessions to be executed in the user context
session required pam_selinux.so open
session optional pam_keyinit.so force revoke

profile
- Verify that the below ulimits are set in either /etc/profile or in $ORACLE_HOME/oracle_env

if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi

Vi /etc/sudoers
%dba ALL=(ALL) ALL
%asmadmin ALL=(ALL) ALL



Storage Setup
MultiPath or EMC PowerPath
rpm -q device-mapper-multipath
cat /etc/multipath.conf
device {
vendor "DGC"
product ".*"
product_blacklist "LUNZ"
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout "/sbin/mpath_prio_emc /dev/%n"
features "1 queue_if_no_path"
hardware_handler "1 emc"
path_grouping_policy group_by_prio
failback immediate
rr_weight uniform
no_path_retry 60
rr_min_io 1000
path_checker emc_clariion

modprobe dm-multipath
service multipathd start
multipath -v2
chkconfig multipathd on
service multipathd reload
ls /dev | grep sd
cd /sys/class/scsi_host/
cat ./host3/device/fc_host\:host3/
ls ./host3/device/fc_host\:host3/
ls ./host3/device/fc_host\:host3/port_name
cat ./host3/device/fc_host\:host3/port_name
cat ./host3/device/fc_host\:host4/port_name
cat ./host4/device/fc_host\:host4/port_name
multipath -l
multipathd -l
service multipathd reload
service multipathd restart
multipath -v2
reboot
multipath -l
chown grid /dev/dm-*
chgrp asmadmin /dev/dm-*
ll /dev | grep dm
multipath –l

Disk Setup
Reference Doc Page 10 of RACGuides_Rac11gR2OnLinux.pdf
Once the LUNs have been presented from the SAN to ALL servers in the cluster, partition the LUNs from one node only, run fdisk to create a single whole-disk partition with exactly 1 MB offset on each LUN to be used as ASM Disk. From the fdisk prompt, type "u" to switch the display unit from cylinder to sector. Then create a single primary partition starting on sector 2048 (1MB offset assuming sectors of 512 bytes per unit).
fdisk /dev/dm
Command (m for help): u
Changing display/entry units to sectors
Command (m for help): n
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First sector (61-1048575, default 61): 2048
Last sector or +size or +sizeM or +sizeK (2048-1048575, default 1048575):
Using default value 1048575
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Load the updated block device partition tables by running the following on ALL servers participating in the cluster: sbin/partprobe

ASM Setup
The latest ASM RPMs can be downloaded from Oracle at:
http://www.oracle.com/technology/tech/linux/asmlib/index.html
NOTE: The below query listing the ASM RPMs is an example and is not meant to provide the correct version.
# rpm -qa | grep oracleasm | sort

Do asm under grid control user ( grid )
cd /etc/init.d/
./oracleasm stop
./oracleasm disable
./oracleasm configure
/etc/init.d/oracleasm enable
./oracleasm configure

Configuring the Oracle ASM library driver.
Default user to own the driver interface []: grid
Default group to own the driver interface []:asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
/etc/init.d/oracleasm createdisk ARCHIVE1 /dev/dm-6
/etc/init.d/oracleasm createdisk DATA1 /dev/dm-7
/etc/init.d/oracleasm createdisk OCR_VOTE1 /dev/dm-8
/etc/init.d/oracleasm createdisk OCR_VOTE2 /dev/dm-2
/etc/init.d/oracleasm createdisk OCR_VOTE3 /dev/dm-3
/etc/init.d/oracleasm listdisks
I should see disks now in /dev/oracleasm/disks/