Oracle Grid Infrastructure - reconfiguration

There are some situations we need to reconfigure the Grid Home like IP-address change/hostname changer,
fresh GI installation due to binary issue etc..
Please follow the below steps


0. Please capture the below details before you do reconfigurations
      crsctl stat res -p
      crsctl stat res -t
      cluster Name:cemutlo -n
      scan name :  srvctl confif scan
      listener port :  srvctl confif listener
      host vip names
      if you are doing it in exadata   . please do the step 3 before you do the reconfiguration


1. Run the below command to deconfig the GI from all the nodes except the last node.

 $ORACLE_HOME/crs/install/rootcrs.pl -verbose -deconfig -force

2. Run the below command in the last node ( keepdg to retain the DG configuration)
 $ORACLE_HOME\crs\install\rootcrs.pl -deconfig -force -verbose -lastnode -keepdg





3. During reconfiguration, incase if the OCR disks are not listed as candidates for OCR DG creations, we may need to delete and add the disks

    ORACLE ASM:
            we can use either Plan A or Plan B         
          
           Plan A:
                   oracleasm createdisk <label> <dev_mapper>
                   oracleasm createdisk OCR_DK1  /dev/mapper/ASM_OCR_0001
                  
          Plan B:
                 incase the above is faling with "Unable to open device "/dev/mapper/ASM_OCR_0001": Device or resource busy "
          /usr/sbin/asmtool -C -l /dev/oracleasm -n OCR_DK1 -s  /dev/mapper/ASM_OCR_0001 -a force=yes

          Post task:
                   oracleasm scandisks
                   oracleasm listdisks
     AFD:
            ( BS = block size) and count= 1000 * BS = size of the disk
     
              dd if=/dev/zero of=/dev/mapper/ASM_REG_OCR_0001 bs=1M count=1000
        Post task:
       asmcmd afd_scan
       asmcmd afd_lsdsk
    Exadata:
             delete 3 disks from a DG from different cell to make them as a candidate for the OCR DG ( this step need to be done before the deconfigruation run)


4. Cleanup the gpnp files from all the nodes
  find <GRID_HOME>/gpnp/* -type f -exec rm -rf {} \;




5. run config.sh
 cd $ORACLE_HOME/crs/config
 ./config.sh

 feed the below values
    => cluster_name, scan name, port & Disable GNS if it is not used.
    => ensure all the host and VIPs are fetched automatically, if not, please add them manually.
    => Storage for OCR: create new DG and choose the disks candidates.
   

ACFS on Exadata

Oracle Automatic Storage Management Cluster File System (ACFS) on Exadata Database Machine:

Starting with Oracle Grid Infrastructure version 12.1.0.2, Oracle ACFS supports all database files and general purpose files on Oracle Exadata Database Machine running Oracle Linux on database servers.

The following database versions are supported by Oracle ACFS on Exadata Database Machine:

• Oracle Database 10g Rel. 2 (10.2.0.4 and 10.2.0.5)
• Oracle Database 11g (11.2.0.4 and higher)
• Oracle Database 12c (12.1.0.1 and higher)

The below steps are duly tested in Exadata X7 machines

Pre-requisite for ACFS creation

1. Verify if ACFS/ADVM module is loaded ( oracleacfs & oracleadvm )

dcli -g /home/oracle/dbs_group -l oracle 'lsmod | grep oracle'

 

Step by step instruction to create ACFS on Exadata

 

1. As ORAGRID : Create ASM volume to be used for ACFS file systems on FLASH01 DG
    

asmcmd volcreate -G FLASH01 -s 10G acfsvol

 

 

        2) As ORAGRID:  Capture the volume device name

asmcmd volinfo -G FLASH01 acfsvol

 

For example this case:

 

Diskgroup Name: FLASH01

 

         Volume Name: ACFSVOL

         Volume Device: /dev/asm/acfsvol-118

         State: ENABLED

         Size (MB): 10240

         Resize Unit (MB): 512

         Redundancy: MIRROR

         Stripe Columns: 8

         Stripe Width (K): 1024

         Usage:

         Mountpath: 

 

 

        3) As ORAGRID:  verify if the volume device name created on all the nodes

dcli -g /home/oracle/dbs_group -l oracle ‘s -l /dev/asm/acfsvol-118’

 

        4) As ORAGRID:  Verify if ADVM is enabled

crsctl stat res ora.FLASH01.ACFSVOL.advm -t

 

For example this case:

 

crsctl stat res ora.FLASH01.ACFSVOL.advm -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.FLASH01.ACFSVOL.advm

               ONLINE  ONLINE       fraespou0101             STABLE

               ONLINE  ONLINE       fraespou0102             STABLE

               ONLINE  ONLINE       fraespou0103             STABLE

               ONLINE  ONLINE       fraespou0104             STABLE

--------------------------------------------------------------------------------

 

    

For example this case:

 

 

5) AS ORAGRID:  Create ACFS on ADVM  

/sbin/mkfs -t acfs /dev/asm/acfsvol-321 –b 4K

 

 

 

 6) As root - Create the mount points for the file systems and change ownership

 

dcli -g ~/dbs_group -l root ‘mkdir /u01/app/oracle/admin/common’   dcli -g ~/dbs_group -l root 'chown oracle:oinstall /u01/app/oracle/admin/common’   dcli -g ~/dbs_group -l root 'chmod 775 /u01/app/oracle/admin/common’

 

7) As ROOT:  Add filesystem  

/u01/app/oragrid/product/18.0/bin/srvctl add filesystem -d /dev/asm/acfsvol-321 -m /u01/app/oracle/admin/common  -u oracle -fstype ACFS -autostart ALWAYS

 

 

8) As ROOT/Oracle:  Start  filesystem  

/u01/app/oragrid/product/18.0/bin/srvctl start filesystem -d /dev/asm/acfsvol-321

 

 

 

  

Verifications

 

 

9) root/oragrid:  Verify if change ownership and permission  

crsctl stat res ora.FLASH01.ACFSVOL.acfs -t

 

crsctl stat res ora.FLASH01.ACFSVOL.advm –t

 

 

 

 

10) as Oracle:  list the ACFS FS  from all the nodes  

dcli -g  /home/oracle/dbs_group -l oracle df –h /u01/app/oracle/admin/common

 

 

 

 

 

11) as Oracle:  Touch a file on the ACFS FS  from all the nodes  

dcli -g  /home/oracle/dbs_group -l oracle touch /u01/app/oracle/admin/common/a.txt dcli -g  /home/oracle/dbs_group -l oracle cat /u01/app/oracle/admin/common/a.txt

 

 

 

How to restore a corrupted/missing datafile in standby:

How to restore a corrupted/missing datafile in standby:


Before 12.2
1. Cancel the recovery
    dgmgrl > edit database <db> set state='APPLY-OFF';
     (or) alter database recover managed standby database cancel;

2. From Primary take a datafile copy
rman target /
 backup as copy datafile 2 format "/u01/app/oracle/data01/sysaux_01.dbk" ;

3. scp the file from primay server to standby server ( may be to the same location in standby server)
4. catalog the datafile copy ( in sby server)
rman target /
  catalog datafilecopy '/u01/app/oracle/data01/sysaux_01.dbk'

5. Either do a switch copy or restore the datafile to the same location mentioned in the controlfile( in sby server)
   rman target /
 switch datafile 2 to copy;
 report schema

  alternatively
RUN {
  ALLOCATE CHANNEL ch00 TYPE disk ;
  restore datafile 2  ;
  }

6. enable recovery
    dgmgrl > edit database <db> set state='APPLY-ON';
     (or) alter database recover managed standby database disconnect from session ;


After 12
rman target sys/oracle@prod
RMAN> run
{
set newname for datafile 4 to '/u01/oracle/data02/users01.dbf' section size 1G;
restore (datafile 4 from service prodservice) using compressed backupset;
catalog datafilecopy '/u01/oracle/data02/users01.dbf';
}

OPatch lsinventory command doesnot show RAC details

Please use the below command to get the remote nodes patching details


 $ORACLE_HOME/OPatch/opatchauto report -format xml -type patches -remote


===============================================================
This is applicable to all releases of OPatch.
 The 12.2.0.1.13 release (and later) of OPatch, which is used for all releases 12.1.0.x and later
 The 11.2.0.3.18 release (and later) of OPatch, which is used for all releases 11.2.0.1 - 11.2.0.4
 Beginning with these releases of OPatch,  OPatch will only support patching/listing inventory for local node  of a RAC cluster. There will be no propagation to other nodes in cluster.


1. In Opatch 12.2.0.1.13 and Opatch 11.2.0.3.18, OPatch command option “-all_nodes” will be no-op and existed in -help.
     If opatch command is being called with option “-all_nodes”, Opatch will print out the warning on console msg as:
     ""OPatch was called with -all_nodes option. The -all_nodes option is being deprecated. Please remove it while calling OPatch."
 2.  In future release Opatch 12.2.0.1.14 (and later) and Opatch 11.2.0.3.20 (and later) , Option “-all_nodes” will be removed from -help, user will get syntax error if specify the option.
Alternative Feature of -all_node is available  for Multi-Node GI/RAC :

 $ORACLE_HOME/OPatch/opatchauto report -format xml -type patches -remote
(The remote command will get a list of patches from all nodes)
===========================================================


Refer:
GI/RAC/Single Instance Announcing Deprecation of OPatch Command Option "-all_nodes" (Doc ID 2331762.1)

Near Zero down time for relocating a PDB from one CDB to another CDB - a 12.2 new feature

This new feature significantly reduces downtime by leveraging the clone functionality to relocate a pluggable database (PDB) from one multitenant container database (CDB) to another CDB. The source PDB is still open and fully functional while the actual cloning operation is taking place. The application outage is reduced to a very small window while the source PDB is quiesced and the destination PDB is brought online after applying the incremental redo. The source PDB is subsequently dropped. 

With this feature, you can now adjust to changing workloads and comply with Service Level Agreement (SLA) requirements in near real time.




In Oracle Database 12.1, unplugging and plugging a PDB requires several steps such as unplugging the PDB from the source CDB, copying the database files to a new location, creating the new PDB by plugging the source PDB at the target CDB, and finally dropping the PDB from the source CDB.
A single DDL statement can relocate a PDB, using the “pull” mode, connected to the CDB where the PDB will be relocated to pull it from the CDB where the PDB exists, managing draining existing connections and migrating new connections without requiring any changes to the application.
There are two relocation methods:


 • Normal availability mode

• −  When the newly created PDB is opened in read-write mode for the first time, the source PDB is automatically closed and dropped, and the relocation operation is completed with the relocated PDB being fully available. This is the “normal availability” default mode.
• −  This method can be used to relocate application PDBs too.

• Maximum availability mode
  

  • −  The maximum availability mode reduces application impact by handling the migration of connections, preserving the source CDB in mount state to guarantee connection forwarding of the listener to the remote listener where the PDB is relocated. In this case, you cannot create a PDB with the same name as the source PDB because it will conflict with the listener forwarding. It is expected that connect strings are updated at a time that is convenient for the application. After this is done and all the clients connect to the new host without forwarding, the DBA can drop the source PDB.
  • −  If AVAILABILITY MAX is specified during the CREATE PLUGGABLE DATABASE RELOCATE command, additional handling is performed to ensure smooth migration of workload and persistent connection forwarding from the source to the target. The PDB is always first opened in read-only mode. This makes the PDB available as a target for new connections before the source PDB is closed. During this operation, listener information of the target CDB is automatically sent to the source and a special forwarding registration is performed with the source PDB’s current listener. New connections to the existing listener are automatically forwarded to connect to the new target. This forwarding persists even after the relocation operation has been completed, and effectively allows for no changes to connect strings.
  • −  It is still recommended that connect strings are updated eventually at a time that is convenient for the application, but availability is not dependent on when this action is performed.
    PDB relocation requires enabling the local UNDO mode and ARCHIVELOG mode in both CDBs.

Generic Python DB Module to connect Oracle Database

Please refer my post [ http://jeyaseelan-m.blogspot.com/2018/11/how-to-connect-oracle-database-using.html ] for basic setup or python to oracle connection

import sys,os,datetime,cx_Oracle,subprocess,traceback,logging

from pprint import pprint

# Global Variables declarations

global CONFIG_FILE

CONFIG_FILE=".BLACKOUT_CONFIG"

CONFIG_TEMPLATE = { 'SOURCE_DIR' : 'SOURCE_DIR', 'USERID' : 'USERID', 'PASSWORD' : 'PASSWORD', 'TNS_NAME' : 'TNS_NAME' }

# Print messages

def printError(module,message):

    logging.error("Error at Module - {}: {} " .format(module,message))

    sys.exit(1)

def printWarning(module,message):

    print("Wanning at Module - {}: {} " .format(module,message))

# Get config values

def getConfig():

    global configDict

    configDict = {}

    if os.path.exists(CONFIG_FILE):

       with open(CONFIG_FILE) as config_file:

            for row in config_file:

                key,value = row.split(":")

                configDict[key]=value.strip()

    else:

       printError("getConfig","{} doesnot exist".format(CONFIG_FILE))

    CONFIG_TEMPLATE_set=set(CONFIG_TEMPLATE)

    config_set         =set(configDict)

    if CONFIG_TEMPLATE_set ^ config_set :

       printError("getConfig"," {} parameter is missing in the {} confgile file ".format(CONFIG_TEMPLATE_set ^ config_set,CONFIG_FILE) )

# Get DB details

def getDBconfig():

    global cmdb_dBuser ,cmdb_dBpass,cmdb_dBname,cmdb_dBconection

    cmdb_dBuser=configDict["USERID"]

    cmdb_dBpass=configDict["PASSWORD"]

    cmdb_dBconection=configDict["TNS_NAME"]

# Connect Database

def connectDB():

    global cmdb_dBconection,cmdb_dBcursor

    try:

        cmdb_dBconection = cx_Oracle.connect(cmdb_dBuser,cmdb_dBpass,cmdb_dBconection)

        cmdb_dBcursor=cmdb_dBconection.cursor()

    except  cx_Oracle.DatabaseError as e:

        error, = e.args

        printError(connectDB,error.message)

# run query  

def runQuery(sql):

    cmdb_dBcursor.execute(sql, REFERENCE_NUMBER=str(sys.argv[1]) )

    result=cmdb_dBcursor.fetchall ()

    for row in result:

        print ( " Task Number              : {0} ". format(row[0]))

        print ( " Assigned To              : {0} ". format(row[1]))

        print ( " Assigment Group          : {0} ". format(row[2]))

        print ( " Change Number            : {0} ". format(row[3]))

        print ( " Task State               : {0} ". format(row[4]))

        print ( " Planned Start            : {0} ". format(row[5]))

        print ( " Planned End              : {0} ". format(row[6]))

        print ( " Task details             : {0} ". format(row[11]))

 

 

sqlplus - new features ( 12c to 18c)

The following are important new sqlplus  features  in 12c & 18c






1)SET FEEDBACK ON SQL_ID


It will show sql_id for the sql statements we execute from sqlplus prompt


SQL> create table test_invisible_cols (emp_id number, emp_info char(20),
                                  emp_acc_no number invisible);  2
Table created.


SQL_ID: cw7wvs1x99d72




2) SET ROWLIMIT 10


  it limits the number of rows returned from a sql




3) set linesize window


It will make the columns to fit into the window size




4) SET COLINVISIBLE ON


This will allow to show the invisible columns using desc




SQL> create table test_invisible_cols (emp_id number, emp_info char(20),
                                  emp_acc_no number invisible);  2
Table created.



SQL> desc test_invisible_cols
 Name                                                                                                                                    Null?    Type
 --------------------------------------------------------------------------------------------------------------------------------------- -------- -------------------------------------------------------------------------------------------
 EMP_ID                                                                                                                                           NUMBER
 EMP_INFO                                                                                                                                         CHAR(20)


SQL> SET COLINVISIBLE ON


SQL> desc test_invisible_cols
 Name                                                                                                                                    Null?    Type
 --------------------------------------------------------------------------------------------------------------------------------------- -------- -------------------------------------------------------------------------------------------
 EMP_ID                                                                                                                                           NUMBER
 EMP_INFO                                                                                                                                         CHAR(20)
 EMP_ACC_NO (INVISIBLE)                                                                                                                           NUMBER
SQL>




5)  sqlplus -nologintime


from 12.2 sqlplus connection will show the last_login ( dba_users has additional column last_login).


if we want to disable the last login time show, we can use -nologintime




6) set history on - Another important feature is sqlplus history
     - we can list the history
     - we can run the command out of the history list
    - we can edit a command from history
   - we can delete all/a command from history
 ( hist clear/ hist 1 delete )





SQL> set history on



SQL> set history on
SQL> select count(*) from dba_objects;
  COUNT(*)
----------
     60537
SQL> show sga
Total System Global Area 3.9278E+10 bytes
Fixed Size                 30153104 bytes
Variable Size            7381975040 bytes
Database Buffers         3.1675E+10 bytes
Redo Buffers              190418944 bytes

SQL> show parameter recovery
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string      +FLASH01
db_recovery_file_dest_size           big integer 17793M
recovery_parallelism                 integer     0
remote_recovery_file_dest            string

SQL> history
  1  select count(*) from dba_objects;
  2  show sga
  3  show parameter recovery


SQL> history 2 run
Total System Global Area 3.9278E+10 bytes
Fixed Size                 30153104 bytes
Variable Size            7381975040 bytes
Database Buffers         3.1675E+10 bytes
Redo Buffers              190418944 bytes
SQL>





SQL> help history
 HISTORY
 -------
 Stores, lists, executes, edits of the commands
 entered during the current SQL*Plus session.
 HIST[ORY] [N {RUN | EDIT | DEL[ETE]}] | [CLEAR]
 N is the entry number listed in the history list.
 Use this number to recall, edit or delete the command.
 Example:
 HIST 3 RUN - will run the 3rd entry from the list.
 HIST[ORY] without any option will list all entries in the list.

SQL>