Troubleshooting /Opt Usages::
Start troubleshooting with below command:
[root@PI13]# du -h --max-depth=6 /opt | grep [0-9]G | sort -k2
to find which directories are consuming /opt space and depending on the results proceed further.
1. Troubleshooting /opt/CSCOlumos usage
Normally it takes ~11GB. If more investigate subdirectories.
a) check /opt/CSCOlumos/logs
[root@PI13gorlos opt]# du -sh /opt/CSCOlumos/logs/
613M /opt/CSCOlumos/logs/
b) check size of IOS images:
[root@PI13gorlos opt]# find /opt/CSCOlumos/ -name *.bin | xargs du -sh
38M /opt/CSCOlumos/tmp/c1841-spservicesk9-mz.151-4.M6.bin
18M /opt/CSCOlumos/tmp/c3750e-ipbasek9-mz.150-1.SE3.bin
3.6M /opt/CSCOlumos/tmp/c2950-i6k2l2q4-mz.121-22.EA10a.bin
total size:
[root@PI13gorlos opt]# find /opt/CSCOlumos/ -name *.bin | xargs du -sh | awk '{total += $1} END {print total}'
59.6
c) check /opt/CSCOlumos/da/cdb directory. Netflow data can fill in the space in opt like Oracle database.
Troubleshooting /opt/CSCOlumos/da/cdb content - TBD
2. Troubleshooting archive logs
PI is running in ARCHIVELOG mode which means that archiving of the redo log is enabled.
There are some down sides of running the database in ARCHIVELOG mode. For example, once an online redo log has been filled, it cannot be reused until it has been archived. If Oracle cannot archive the online redo log (for example, the destination directory for the archived redo logs is filled up), it will switch to the next online redo log and keep working. At the same time, Oracle will continue to try to archive the log file.
Unfortunately, once the database runs out of available online redo logs, we have a problem. If the log files can not be written out, then Oracle would have to overwrite them. This is not good because it means we would lose the data that was in those files since Oracle could not archive the file. As a result of this, in an effort to protect the database, Oracle will not overwrite data in an online redo log file until that log file has been archived. Until the file has been archived, the database will simply stop processing user requests. Once the log file has been archived, the database will be freed, and processing can proceed as normal.
When archived logs have filled assigned disk space you may see following error message in alert_wcs.log:
ORA-19815: WARNING: db_recovery_file_dest_size of 107374182400 bytes is
100.00% used, and has 0 remaining bytes available.
If for any reason you will need to remove archived logs from /opt/oracle/base/fast_recovery_area/WCS/archivelog, here is the process:
a) Stop PI_NCS
# ncs stop
b) Switch to root user
# root
password: xxxxxx
c) Switch to oracle user
# su - oracle
# . ./oracleenv
d) Run oracle commands
# sqlplus / as sysdba
sql> startup mount
sql> exit
# rman
rman> connect target /
rman> crosscheck archivelog all;
rman> delete noprompt archivelog all;
rman> exit
# sqlpus / as sysdba
sql> shutdown abort
sql> exit
e) Leave oracle user
# exit
f) leave root user
# exit
g) Start NCS
# ncs start