GI Patch level mismatch between Nodes - after rolling back failed patch at GI_HOME

Error Messages:


We may get the following error messages while doing ASM operations


ORA-15137: The ASM cluster is in rolling patch state.
The cluster upgrade state is [ROLLING PATCH].




Before implement the solution, Please validate where is the actual problem






1. Check patchlevel


Node1


$ crsctl query crs releasepatch
 Oracle Clusterware release patch level is [1295926169] and the complete list of patches [27678276 ] have been applied on the local node. The release patch string is [18.2.0.0.0].


Node 2

$ crsctl query crs releasepatch
Oracle Clusterware release patch level is [3087104924] and the complete list of patches [27676517 27678271 27678276 ] have been applied on the local node. The release patch string is [18.2.0.0.0].

#  Node 2 is ahead of Node 1



2. Kfod on GI Binary to list the patch levels


Node 1:

 $ $ORACLE_HOME/bin/kfod op=patchlvl
 -------------------
 Current Patch level
 ===================
 1295926169

 $ORACLE_HOME/bin/kfod op=patches
 ---------------
 List of Patches
 ===============
 27678276




Node 2:



$ORACLE_HOME/bin/kfod op=patchlvl
 -------------------
 Current Patch level
 ===================
 3087104924

 $ORACLE_HOME/bin/kfod op=patches
 ---------------
 List of Patches
 ===============
 27676517
 27678271
 27678276


# In Node 1  27676517 &  27678271  are missing.




3.  lsinventory


Node 1
 $ORACLE_HOME/OPatch/opatch lsinventory | grep -i desc
ARU platform description:: Linux x86-64
Patch description:  "OCW RELEASE UPDATE 18.2.0.0.0 (27678276)"
Patch description:  "ACFS RELEASE UPDATE 18.2.0.0.0 (27678271)"
Patch description:  "Database Release Update : 18.2.0.0.180417 (27676517)"

Node 2
$ORACLE_HOME/OPatch/opatch lsinventory | grep -i desc
ARU platform description:: Linux x86-64
Patch description:  "OCW RELEASE UPDATE 18.2.0.0.0 (27678276)"
Patch description:  "ACFS RELEASE UPDATE 18.2.0.0.0 (27678271)"
Patch description:  "Database Release Update : 18.2.0.0.180417 (27676517)"


# But lsinventory shows both patchlevel same


4. Patch log files


Node 1 logs:

Node1_opatch_logs

 Applying /data/oracle/OraclePatches/27681568/27676517 ...
 Patch /data/oracle/OraclePatches/27681568/27676517 applied.

 Applying /data/oracle/OraclePatches/27681568/27678271 ...
 Patch /data/oracle/OraclePatches/27681568/27678271 applied.

 Applying /data/oracle/OraclePatches/27681568/27678276 ...
 Patch /data/oracle/OraclePatches/27681568/27678276 applied. <==== applied successfully





Solutions:


Try the following solutions one by one. Check the releaselevel after each solution implemented, if that particular solution fixed, then you can skip the other steps.


# Sol-1
ALTER SYSTEM STOP ROLLING PATCH

# SOl-2
crsctl stop rollingpatch


#  Sol-3

 As root user,
 $GRID_HOME/crs/install/rootcrs.sh -prepatch
 $GRID_HOME/crs/install/rootcrs.sh -postpatch

 if above does not help, execute following and check status
 $GI_HOME/bin/clscfg -patch


# Sol-4


 Action Plan
 ==============
 Step 1


  $GRID_HOME/crs/install/rootcrs.sh -prepatch


 Step 2

  Add the missing patches to the kfod output[which are available in opatch lsinventory],
  As grid owner,
  $GRID_HOME/bin/patchgen commit -pi 27676517
  $GRID_HOME/bin/patchgen commit -pi 27678271

 Incase if we need to rollback extra patch

  $GRID_HOME/bin/patchgen commit -rb 27678271
  Check the below on both nodes and notify if there is any difference
  kfod op=patchlvl
  kfod op=patches

 Step 3
  If they are same run the below,
  $GRID_HOME/crs/install/rootcrs.sh -postpatch