We may get the following error messages while doing ASM operations
ORA-15137: The ASM cluster is in rolling patch state.
The cluster upgrade state is [ROLLING PATCH].
Before implement the solution, Please validate where is the actual problem
1. Check patchlevel
Node1
$ crsctl query crs releasepatch
Oracle Clusterware release patch level is [1295926169] and the complete list of patches [27678276 ] have been applied on the local node. The release patch string is [18.2.0.0.0].
Node 2
$ crsctl query crs releasepatch
Oracle Clusterware release patch level is [3087104924] and the complete list of patches [27676517 27678271 27678276 ] have been applied on the local node. The release patch string is [18.2.0.0.0].
# Node 2 is ahead of Node 1
2. Kfod on GI Binary to list the patch levels
Node 1:
$ $ORACLE_HOME/bin/kfod op=patchlvl
-------------------
Current Patch level
===================
1295926169
$ORACLE_HOME/bin/kfod op=patches
---------------
List of Patches
===============
27678276
Node 2:
$ORACLE_HOME/bin/kfod op=patchlvl
-------------------
Current Patch level
===================
3087104924
$ORACLE_HOME/bin/kfod op=patches
---------------
List of Patches
===============
27676517
27678271
27678276
# In Node 1 27676517 & 27678271 are missing.
3. lsinventory
Node 1
$ORACLE_HOME/OPatch/opatch lsinventory | grep -i desc
ARU platform description:: Linux x86-64
Patch description: "OCW RELEASE UPDATE 18.2.0.0.0 (27678276)"
Patch description: "ACFS RELEASE UPDATE 18.2.0.0.0 (27678271)"
Patch description: "Database Release Update : 18.2.0.0.180417 (27676517)"
Node 2
$ORACLE_HOME/OPatch/opatch lsinventory | grep -i desc
ARU platform description:: Linux x86-64
Patch description: "OCW RELEASE UPDATE 18.2.0.0.0 (27678276)"
Patch description: "ACFS RELEASE UPDATE 18.2.0.0.0 (27678271)"
Patch description: "Database Release Update : 18.2.0.0.180417 (27676517)"
# But lsinventory shows both patchlevel same
4. Patch log files
Node 1 logs:
Node1_opatch_logs
Applying /data/oracle/OraclePatches/27681568/27676517 ...
Patch /data/oracle/OraclePatches/27681568/27676517 applied.
Applying /data/oracle/OraclePatches/27681568/27678271 ...
Patch /data/oracle/OraclePatches/27681568/27678271 applied.
Applying /data/oracle/OraclePatches/27681568/27678276 ...
Patch /data/oracle/OraclePatches/27681568/27678276 applied. <==== applied successfully
Solutions:
Try the following solutions one by one. Check the releaselevel after each solution implemented, if that particular solution fixed, then you can skip the other steps.
# Sol-1
ALTER SYSTEM STOP ROLLING PATCH
# SOl-2
crsctl stop rollingpatch
# Sol-3
As root user,
$GRID_HOME/crs/install/rootcrs.sh -prepatch
$GRID_HOME/crs/install/rootcrs.sh -postpatch
if above does not help, execute following and check status
$GI_HOME/bin/clscfg -patch
# Sol-4
Action Plan
==============
Step 1
$GRID_HOME/crs/install/rootcrs.sh -prepatch
Step 2
Add the missing patches to the kfod output[which are available in opatch lsinventory],
As grid owner,
$GRID_HOME/bin/patchgen commit -pi 27676517
$GRID_HOME/bin/patchgen commit -pi 27678271
Incase if we need to rollback extra patch
$GRID_HOME/bin/patchgen commit -rb 27678271
Check the below on both nodes and notify if there is any difference
kfod op=patchlvl
kfod op=patches
Step 3
If they are same run the below,
$GRID_HOME/crs/install/rootcrs.sh -postpatch
Check the below on both nodes and notify if there is any difference
kfod op=patchlvl
kfod op=patches
Step 3
If they are same run the below,
$GRID_HOME/crs/install/rootcrs.sh -postpatch
