Oracle Cloud Infrastructure
Coming soon ...
Apps R12.2
  • R12.2 Architecture
  • Cloning Error (RC-50208)
    		•
    Apps R12.1
  • Major changes from 11i
  • R12:HTTP Debug Logging
  • Compile Apps Schema invalid objects in R12
    		•
    Apps 11i
  • Apps Components and Architecture
  • Concurrent managers
  • Patching
  • Using AD Patch
  • Using AD Control
  • FNDCPASS utility
  • Single-node Installation
  • Multi-node Installation
  • Find Apps Version
  • Cloning
  • Upgrade 11.5.9 to 11.5.10.2
  • Upgrade from 11.5.10.2 to R12
  • Upgrading 9i to 10g with 11i
    		•
    11i/R12 General Topics
  • AppsDBA System Management Utilities Guide
  • Identifying Long Idle Sessions
  • Identifying High Active Sessions
  • Change hostname for EBS
    		•
    Oracle 12c Database
  • Oracle12c PSU Apply
  • Oracle12c Datafile moved Online
    		•
    Oracle 11g Database
  • Upgrade 10g to 11g R1
  • Upgrade 11.2.0.2 to 11.2.0.3
    • Database 9i-10g
  • Top 99 Responsibilities of a DBA
  • General Info
  • Database Patching
  • 10g:ASM
  • 10g:Data Pump
  • 10g:Data Guard Installing
  • 10g:Rollback Monitoring
  • 10g:Flashback Table
  • Tablespace Management
  • Materialized Views
  • 10g:Enterprise Manager
  • 10g:Upgrade
  • Error:Ora-01631
    		•
    DBA Scripts
  • Disk I/O,Events,Waits
  • Tablespace Information
  • Session Statistics
  • Hit/Miss Ratios
  • User Information
  • Rollback Segments
  • Full Table Scans
  • Contention/Locking
  • Redo Log Buffer
  • Data Dictionary Info
    		•
    Oracle10g Application Server
  • Oracle10g Application Installation
  • (Re)Securing OAS Control
  • Oracle AS10g null protocol issue
    		•
    Oracle Backup & Recovery
  • RMAN
  • RMAN Setup
  • Backup Recovery
  • Flash Recovery Area
    		•
    Oracle10g Discoverer with Apps
    Coming soon ..............
    Troubleshooting
  • Discoverer Troubleshooting
  • Access EBS in mozile
    • Linux and Unix Platforms
  • How To configure NFS
  • Unix/Linux Command
  • Change hostname in Linux
  • SENDMAIL configuration
    		•
    This Oracle Application DBA Portal is the biggest knowledge gateway for the people in the world of Oracle...
    Friday, April 3, 2009
    A Quick Reference for Oracle Database 10g RAC on Linux and Unix Platforms
    Introduction
    This paper will discuss and demonstrate some of the basic RAC management commands you might use to manage your Oracle 10g RAC components. The material presented will be applicable to most if not all Linux and Unix platforms. It will not cover RAC on the Microsoft Windows operating system. We will begin with the basics of checking out a RAC system to identify whether or not the appropriate services and resources are running or not. Then we will go through the basic startup and shutdown commands for all services and resources. Lastly, we will go over the startup of individual resources and checking the status of each step as we progress. The shutdown of individual resources can be done by reversing the individual startup order.
    About our Environment

    The RAC environment that we will reference throughout this paper is composed of two nodes running Red Hat Enterprise Linux 4 ES with a shared disk, and has two ASM instances named ASM1 and ASM2, two database instances named orcl1 and orcl2, and a service named RAC that is used for transparent application failover (TAF) and load balancing.

    Overview of Basic RAC Management Commands
    The commands we will use are listed below. Remember that this document is a quick reference, and not an exhaustive list of all commands for managing your RAC environment.
    Cluster Related Commands
    crs_stat -t Shows HA resource status (hard to read)
    crsstat Ouptut of crs_stat -t formatted nicely
    ps -ef|grep d.bin crsd.bin evmd.bin ocssd.bin
    crsctl check crs CSS,CRS,EVM appears healthy
    crsctl stop crs Stop crs and all other services
    crsctl disable crs* Prevents CRS from starting on reboot
    crsctl enable crs* Enables CRS start on reboot
    crs_stop -all Stops all registered resources
    crs_start -all Starts all registered resources

    * These commands update the file /etc/oracle/scls_scr//root/crsstart which contains the string “enable” or “disable” as appropriate.
    Database Related Commands
    srvctl start instance -d -i Starts an instance
    srvctl start database -d Starts all instances
    srvctl stop database -d Stops all instances, closes database
    srvctl stop instance -d -i Stops an instance
    srvctl start service -d -s Starts a service
    srvctl stop service -d -s Stops a service
    srvctl status service -d Checks status of a service
    srvctl status instance -d -i Checks an individual instance
    srvctl status database -d Checks status of all instances
    srvctl start nodeapps -n Starts gsd, vip, listener, and ons
    srvctl stop nodeapps -n Stops gsd, vip and listener

    Keep in mind that some resources will not start unless other resources are already online. We will now look at the general dependency list in greater detail.
    There are three main background processes you can see when doing a ps –ef|grep d.bin. They are normally started by init during the operating system boot process. They can be started and stopped manually by issuing the command /etc/init.d/init.crs {start|stop|enable|disable}
    1. /etc/rc.d/init.d/init.evmd
    2. /etc/rc.d/init.d/init.cssd
    3. /etc/rc.d/init.d/init.crsd
    Once the above processes are running, they will automatically start the following services in the following order if they are enabled. This list assumes you are using ASM and have a service set up for TAF/load balancing.
    1. The nodeapps (gsd, VIP, ons, listener) are brought online.
    2. The ASM instances are brought online.
    3. The database instances are brought online.
    4. Any defined services are brought online.

    Basic RAC Management Commands
    Now that we know the dependency tree and have some commands at our disposal, let’s have a look at them one at a time, starting with the cluster commands and processes.
    crs_stat -t
    This command shows us the status of each registered resource in the cluster. I generally avoid this command because its output is hard to read since the names are truncated as you can see in the sample output below. You can download a helpful script called crsstat from http://www.dbspecialists.com/specialists/specialist2007-05.html to make it easy on your eyes.
    [oracle@green ~]$ crs_stat -t
    Name Type Target State Host
    ------------------------------------------------------------
    ora....SM1.asm application ONLINE ONLINE green
    ora....EN.lsnr application ONLINE ONLINE green
    ora.green.gsd application ONLINE ONLINE green
    ora.green.ons application ONLINE ONLINE green
    ora.green.vip application ONLINE ONLINE green
    ora.....RAC.cs application ONLINE ONLINE red
    ora....cl1.srv application ONLINE ONLINE green
    ora....cl2.srv application ONLINE ONLINE red
    ora.orcl.db application ONLINE ONLINE red
    ora....l1.inst application ONLINE ONLINE green
    ora....l2.inst application ONLINE ONLINE red
    ora....SM2.asm application ONLINE ONLINE red
    ora....ED.lsnr application ONLINE ONLINE red
    ora.red.gsd application ONLINE ONLINE red
    ora.red.ons application ONLINE ONLINE red
    ora.red.vip application ONLINE ONLINE red

    [oracle@green ~]$
    crsstat
    The output of this script is much better. You can learn more about this script and download it at http://www.dbspecialists.com/specialists/specialist2007-05.html.
    [oracle@green ~]$ crsstat
    HA Resource Target State
    ----------- ------ -----
    ora.green.ASM1.asm ONLINE ONLINE on green
    ora.green.LISTENER_GREEN.lsnr ONLINE ONLINE on green
    ora.green.gsd ONLINE ONLINE on green
    ora.green.ons ONLINE ONLINE on green
    ora.green.vip ONLINE ONLINE on green
    ora.orcl.RAC.cs ONLINE ONLINE on red
    ora.orcl.RAC.orcl1.srv ONLINE ONLINE on green
    ora.orcl.RAC.orcl2.srv ONLINE ONLINE on red
    ora.orcl.db ONLINE ONLINE on red
    ora.orcl.orcl1.inst ONLINE ONLINE on green
    ora.orcl.orcl2.inst ONLINE ONLINE on red
    ora.red.ASM2.asm ONLINE ONLINE on red
    ora.red.LISTENER_RED.lsnr ONLINE ONLINE on red
    ora.red.gsd ONLINE ONLINE on red
    ora.red.ons ONLINE ONLINE on red
    ora.red.vip ONLINE ONLINE on red

    [oracle@green ~]$
    ps -ef|grep d.bin
    We can use this command to verify that the CRS background processes are actually running. It is implicit that they are running if the crs_stat command and crsstat script work. If they do not work, you will want to verify the background processes are really running.
    [root@green ~]# ps -ef|grep d.bin
    oracle 5335 3525 0 Jul11 ? 00:00:05 /u01/app/oracle/product/10.2.0/crs/bin/evmd.bin
    root 5487 3817 0 Jul11 ? 00:00:00 /u01/app/oracle/product/10.2.0/crs/bin/crsd.bin reboot
    oracle 5932 5392 0 Jul11 ? 00:00:00 /u01/app/oracle/product/10.2.0/crs/bin/ocssd.bin
    root 30486 30177 0 18:23 pts/1 00:00:00 grep d.bin

    [root@green ~]#
    crsctl check crs
    This command verifies that the above background daemons are functioning.
    [oracle@green ~]$ crsctl check crs
    CSS appears healthy
    CRS appears healthy
    EVM appears healthy
    [oracle@green ~]$
    crsctl stop crs
    We’ll need to be logged onto the server as the root user to run this command. It will stop all HA resources on the local node, and it will also stop the above mentioned background daemons.

    [oracle@green ~]$ crsctl stop crs
    Insufficient user privileges.
    [oracle@green ~]$ su
    Password:
    [root@green oracle]# crsctl stop crs
    Stopping resources. This could take several minutes.
    Successfully stopped CRS resources.
    Stopping CSSD.
    Shutting down CSS daemon.
    Shutdown request successfully issued.
    [root@green oracle]#
    crsctl disable crs
    This command will prevent CRS from starting on a reboot. Note there is no return output from the command.
    [root@green oracle]# crsctl disable crs

    [root@green oracle]#
    We did a reboot after this and verified that CRS did not come back online because we wanted to do some operating system maintenance. Let’s check the status by running some of the commands we’ve just discussed.
    [oracle@green ~]$ crsstat
    HA Resource Target State
    ----------- ------ -----
    error connecting to CRSD at [(ADDRESS=(PROTOCOL=ipc)(KEY=ora_crsqs))] clsccon 184

    [oracle@green ~]$ crsctl check crs
    Failure 1 contacting CSS daemon
    Cannot communicate with CRS
    Cannot communicate with EVM

    [oracle@green ~]$ ps -ef|grep d.bin
    oracle 6149 5582 0 15:54 pts/1 00:00:00 grep d.bin
    [oracle@green ~]$
    Everything appears to be down on this node as expected.
    Now let’s start everything back up. We will need to be root for this, unless you have been given permissions or sudo to run crsctl start crs.
    [root@green oracle]# crsctl start crs
    Attempting to start CRS stack
    The CRS stack will be started shortly
    [root@green oracle]#
    After a few minutes the registered resources for this node should come online. Let’s check to be sure:
    [oracle@green ~]$ crsstat
    HA Resource Target State
    ----------- ------ -----
    ora.green.ASM1.asm ONLINE ONLINE on green
    ora.green.LISTENER_GREEN.lsnr ONLINE ONLINE on green
    ora.green.gsd ONLINE ONLINE on green
    ora.green.ons ONLINE ONLINE on green
    ora.green.vip ONLINE ONLINE on green
    ora.orcl.RAC.cs ONLINE ONLINE on red
    ora.orcl.RAC.orcl1.srv ONLINE ONLINE on green
    ora.orcl.RAC.orcl2.srv ONLINE ONLINE on red
    ora.orcl.db ONLINE ONLINE on red
    ora.orcl.orcl1.inst ONLINE ONLINE on green
    ora.orcl.orcl2.inst ONLINE ONLINE on red
    ora.red.ASM2.asm ONLINE ONLINE on red
    ora.red.LISTENER_RED.lsnr ONLINE ONLINE on red
    ora.red.gsd ONLINE ONLINE on red
    ora.red.ons ONLINE ONLINE on red
    ora.red.vip ONLINE ONLINE on red

    [oracle@green ~]$
    Let’s not forget to enable CRS on reboot:
    [root@green oracle]# crsctl enable crs
    crs_stop -all
    This is a handy script that stops the registered resources and leaves the CRS running. This includes all services in the cluster, so it will bring down all registered resources on all nodes.
    [oracle@green ~]$ crs_stop -all
    Attempting to stop `ora.green.gsd` on member `green`
    Attempting to stop `ora.orcl.RAC.orcl2.srv` on member `red`
    Stop of `ora.orcl.TEST.orcl1.srv` on member `green` succeeded.
    Attempting to stop `ora.orcl.RAC.orcl1.srv` on member `green`
    Attempting to stop `ora.green.ons` on member `green`
    Attempting to stop `ora.orcl.RAC.cs` on member `red`
    Stop of `ora.green.gsd` on member `green` succeeded.
    Stop of `ora.orcl.RAC.orcl1.srv` on member `green` succeeded.
    Stop of `ora.orcl.RAC.orcl2.srv` on member `red` succeeded.
    Stop of `ora.orcl.TEST.orcl2.srv` on member `red` succeeded.
    Stop of `ora.green.ons` on member `green` succeeded.
    --snip--

    CRS-0216: Could not stop resource 'ora.orcl.orcl2.inst'.
    [oracle@green ~]$
    Occasionally you will get the CRS-0216 error message shown above. This is usually bogus, but you should re-check with crsstat and ps –ef|grep smon or similar to be sure everything has died off.
    Let’s verify that crs_stop -all worked as expected:
    [root@green oracle]# crsstat
    HA Resource Target State
    ----------- ------ -----
    ora.green.ASM1.asm OFFLINE OFFLINE
    ora.green.LISTENER_GREEN.lsnr OFFLINE OFFLINE
    ora.green.gsd OFFLINE OFFLINE
    ora.green.ons OFFLINE OFFLINE
    ora.green.vip OFFLINE OFFLINE
    ora.orcl.RAC.cs OFFLINE OFFLINE
    ora.orcl.RAC.orcl1.srv OFFLINE OFFLINE
    ora.orcl.RAC.orcl2.srv OFFLINE OFFLINE
    ora.orcl.db OFFLINE OFFLINE
    ora.orcl.orcl1.inst OFFLINE OFFLINE
    ora.orcl.orcl2.inst OFFLINE OFFLINE
    ora.red.ASM2.asm OFFLINE OFFLINE
    ora.red.LISTENER_RED.lsnr OFFLINE OFFLINE
    ora.red.gsd OFFLINE OFFLINE
    ora.red.ons OFFLINE OFFLINE
    ora.red.vip OFFLINE OFFLINE

    [root@green oracle]#
    Let’s move on to working with srvctl and managing individual resources. We will begin with the crs background daemons already running, and all registered resources being offline from the last step above. We will first start the nodeapps, then the ASM instances, followed by the database instances, and lastly the services for TAF and load balancing. This is the dependency order in our particular environment. You may or may not have ASM or TAF and load balancing services to start in your environment.
    srvctl start nodeapps -n (node)
    This will bring up the gsd, ons, listener, and vip. The same command can shut down the nodeapps by replacing start with stop.
    [oracle@green ~]$ srvctl start nodeapps -n green
    [oracle@green ~]$ srvctl start nodeapps -n red
    Now we will check with crsstat again to be sure the nodeapps have started.
    [oracle@green ~]$ crsstat
    HA Resource Target State
    ----------- ------ -----
    ora.green.ASM1.asm OFFLINE OFFLINE
    ora.green.LISTENER_GREEN.lsnr ONLINE ONLINE on green
    ora.green.gsd ONLINE ONLINE on green
    ora.green.ons ONLINE ONLINE on green
    ora.green.vip ONLINE ONLINE on green
    ora.orcl.RAC.cs OFFLINE OFFLINE
    ora.orcl.RAC.orcl1.srv OFFLINE OFFLINE
    ora.orcl.RAC.orcl2.srv OFFLINE OFFLINE
    ora.orcl.db OFFLINE OFFLINE
    ora.orcl.orcl1.inst OFFLINE OFFLINE
    ora.orcl.orcl2.inst OFFLINE OFFLINE
    ora.red.ASM2.asm OFFLINE OFFLINE
    ora.red.LISTENER_RED.lsnr ONLINE ONLINE on red
    ora.red.gsd ONLINE ONLINE on red
    ora.red.ons ONLINE ONLINE on red
    ora.red.vip ONLINE ONLINE on red

    [root@green oracle]#
    Now we need to start our ASM instances before we bring up our database and services.
    srvctl start asm -n (node)
    This will bring up our ASM instances on nodes green and red. Again, the same command will stop the ASM instances by replacing start with stop.
    [oracle@green ~]$ srvctl start asm -n green
    [oracle@green ~]$ srvctl start asm -n red
    [oracle@green ~]$ crsstat
    HA Resource Target State
    ----------- ------ -----
    ora.green.ASM1.asm ONLINE ONLINE on green
    ora.green.LISTENER_GREEN.lsnr ONLINE ONLINE on green
    ora.green.gsd ONLINE ONLINE on green
    ora.green.ons ONLINE ONLINE on green
    ora.green.vip ONLINE ONLINE on green
    ora.orcl.RAC.cs OFFLINE OFFLINE
    ora.orcl.RAC.orcl1.srv OFFLINE OFFLINE
    ora.orcl.RAC.orcl2.srv OFFLINE OFFLINE
    ora.orcl.db OFFLINE OFFLINE
    ora.orcl.orcl1.inst OFFLINE OFFLINE
    ora.orcl.orcl2.inst OFFLINE OFFLINE
    ora.red.ASM2.asm ONLINE ONLINE on red
    ora.red.LISTENER_RED.lsnr ONLINE ONLINE on red
    ora.red.gsd ONLINE ONLINE on red
    ora.red.ons ONLINE ONLINE on red
    ora.red.vip ONLINE ONLINE on red

    [oracle@green ~]$
    Now let’s bring up our orcl1 and orcl2 instances, and verify they are up with crsstat. Once more we can replace start with stop and shutdown an individual instance if we so choose.
    srvctl start instance -d (database) -I (instance)
    [oracle@green ~]$ srvctl start instance –d orcl –i orcl1
    [oracle@green ~]$ srvctl start instance -d orcl -i orcl2
    [oracle@green ~]$ crsstat
    HA Resource Target State
    ----------- ------ -----
    ora.green.ASM1.asm ONLINE ONLINE on green
    ora.green.LISTENER_GREEN.lsnr ONLINE ONLINE on green
    ora.green.gsd ONLINE ONLINE on green
    ora.green.ons ONLINE ONLINE on green
    ora.green.vip ONLINE ONLINE on green
    ora.orcl.RAC.cs OFFLINE OFFLINE
    ora.orcl.RAC.orcl1.srv OFFLINE OFFLINE
    ora.orcl.RAC.orcl2.srv OFFLINE OFFLINE
    ora.orcl.db ONLINE ONLINE on red
    ora.orcl.orcl1.inst ONLINE ONLINE on green
    ora.orcl.orcl2.inst ONLINE ONLINE on red
    ora.red.ASM2.asm ONLINE ONLINE on red
    ora.red.LISTENER_RED.lsnr ONLINE ONLINE on red
    ora.red.gsd ONLINE ONLINE on red
    ora.red.ons ONLINE ONLINE on red
    ora.red.vip ONLINE ONLINE on red

    [oracle@green ~]$
    srvctl start service -d (database) -s (service)
    Now we will finish up by bringing our load balanced/TAF service named RAC online.
    [oracle@green ~]$ srvctl start service -d orcl -s RAC
    [oracle@green ~]$ crsstat
    HA Resource Target State
    ----------- ------ -----
    ora.green.ASM1.asm ONLINE ONLINE on green
    ora.green.LISTENER_GREEN.lsnr ONLINE ONLINE on green
    ora.green.gsd ONLINE ONLINE on green
    ora.green.ons ONLINE ONLINE on green
    ora.green.vip ONLINE ONLINE on green
    ora.orcl.RAC.cs ONLINE ONLINE on red
    ora.orcl.RAC.orcl1.srv ONLINE ONLINE on green
    ora.orcl.RAC.orcl2.srv ONLINE ONLINE on red
    ora.orcl.db ONLINE ONLINE on red
    ora.orcl.orcl1.inst ONLINE ONLINE on green
    ora.orcl.orcl2.inst ONLINE ONLINE on red
    ora.red.ASM2.asm ONLINE ONLINE on red
    ora.red.LISTENER_RED.lsnr ONLINE ONLINE on red
    ora.red.gsd ONLINE ONLINE on red
    ora.red.ons ONLINE ONLINE on red
    ora.red.vip ONLINE ONLINE on red

    [oracle@green ~]$
    There we have it; all of our resources are now online. The next steps would be to verify you can connect via SQL*Plus or your favorite application.

    Conclusion
    When a product or process is new to you, as Oracle 10g RAC is to many people, it can be an intimidating and possibly disastrous experience. This paper has hopefully given you the elementary commands you will need to manage your Oracle 10g RAC system. While it is not a complete dissection of RAC and its total command set, it should be enough to get you on your feet. You can always get the basic syntax of srvctl by typing srvctl –help. For a complete list of all options, type srvctl –h. You can also get the complete syntax for crsctl by typing crsctl at the command line. Also, do have a peek at the Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide, publication number B14197-04. You can find it on Oracle’s website at http://download.oracle.com/docs/cd/B19306_01/rac.102/b14197/toc.htm.

    Labels:

    posted by Srinivasan .R @ 1:35 AM  
    1 Comments:
    Post a Comment
    << Home
     
    About Me

    Name: Srinivasan .R
    Home: Chennai, India

    About Me:
     I am working as an Oracle Applications DBA specializing in EBS 11i/R12 with Over 14+ years of experience, mainly in different versions of Oracle Database & Application administration on various platforms like HP-UX, SOLARIS, AIX, Red hat Linux & Windows
    See my complete profile
    High Availability
  • Oracle10g RAC Installation
  • A Quick Reference for Oracle Database 10g RAC on Linux and Unix Platforms
  • Implementing Oracle 10g RAC with ASM on AIX
  • Locked objects for whole RAC
  • Monitor Memory RAC
  • Sessions RAC
  • Install Oracle 11g RAC On Linux
  • Migrating Oracle10g DB to ASM
    		•
    Helpful Links
  • Good Metalink Notes
  • Discoverer:Metalink Notes
  • Logs Scripts:Metalink Notes
  • Support:Metalink Notes
    		•
    Previous Post
  • Implementing Oracle 10g RAC with ASM on AIX
  • DBA
    		•
    Archives
    Download Software
  • Oracle 11g
  • Oracle 10g
  • 10g Express Edition
  • Oracle 9i
  • Oracle Apps
  • Oracle Linux
  • Oracle VM
  • App Server
  • Solaris
  • Fedora
  • Fedora
  • OpenSUSE
  • Ubuntu
    		•
    Advertisement Links
    INTUIT Technology

    MACHS DATA
    Add Ons
    Locations of visitors to this page

    Add to Google Reader or Homepage
    Template by
    Sreene



    Oracle Application DBA Portal
    © 2009 Oracle Application DBA Portal All rights reserved Presented by Sreene