The server encountered an internal error

When we log in into OID DAS if we see some error message like this ( OK is Highlighted in Error Message )


OK

The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, you@your.address and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

I tried to find out what may be the issue and started searching on Metalink and got hold of this note

739676.1
The Server Encountered An Internal Error Or Misconfiguration And Was Unable To Complete Your Request

This note suggest that we should restart opmn processes using the following commands

To implement the solution, please execute the following steps:

1. Stop and restart the mid tier by doing the following:
cd ora_home/opmn/bin
2. Run opmnctl stopall then opmnctl startall

When I gave shutdown command it worked fine

oim$ opmnctl shutdown
opmnctl: shutting down opmn and all managed processes...

but on startup it gave number of errors

oim$ opmnctl startall

opmnctl: starting opmn and all managed processes...
================================================================================
opmn id=oimdb:6200
1 of 4 processes started.

ias-instance id=OAS_Infrastructure.oimdb.sbo.hampton.k12.va.us
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ias-component/process-type/process-set:
OC4J/oca/default_island

Error
--> Process (pid=14283)
failed to start a managed process after the maximum retry limit
Log:
/d01/oracle/oasoim/opmn/logs/OC4J~oca~default_island~1

--------------------------------------------------------------------------------
ias-component/process-type/process-set:
OC4J/OC4J_SECURITY/default_island

Error
--> Process (pid=14272)
failed to start a managed process after the maximum retry limit
Log:
/d01/oracle/oasoim/opmn/logs/OC4J~OC4J_SECURITY~default_island~1

--------------------------------------------------------------------------------
ias-component/process-type/process-set:
HTTP_Server/HTTP_Server/HTTP_Server

Error
--> Process (pid=14256)
failed to start a managed process after the maximum retry limit
Log:
/d01/oracle/oasoim/opmn/logs/HTTP_Server~1

[oimdb:log] oimapp$ opmnctl status

Processes in Instance: OAS_Infrastructure

On checking these log files the what i found was very basic check i should have done earlier

Space issue :(

These log files say that clearly

/d01/oracle/oasoim/opmn/logs/OC4J~oca~default_island~1


09/04/28 05:33:51 Fatal Error: Transaction log file (/d01/oracle/oasoim/j2ee/oca/persistence/oca_default_island_1/transaction.state) did not exist and was not possible to create, the most common reason for this is an invalid path or Orion lacking security to write to that path: No space left on device

/d01/oracle/oasoim/opmn/logs/HTTP_Server~1


/d01/oracle/oasoim/Apache/Apache/bin/apachectl start: execing httpd
Ouch! ap_mm_create(1048576, "/d01/oracle/oasoim/Apache/Apache/logs/mm.14250") failed
Error: MM: mm:core: failed to truncate memory file (No space left on device): OS: No such file or directory

--------
09/04/28 05:33:45 Start process
--------
/d01/oracle/oasoim/Apache/Apache/bin/apachectl start: execing httpd
Ouch! ap_mm_create(1048576, "/d01/oracle/oasoim/Apache/Apache/logs/mm.14256") failed
[oimdb:log] oimapp$



After checking the space on the server I got the confirmation of this as well.

fd 0 0 0 0% /dev/fd
/dev/dsk/c0t0d0s1 8072501 124726 7867050 2% /var
swap 5541816 1461736 4080080 27% /tmp
swap 4080136 56 4080080 1% /var/run
/dev/dsk/c0t0d0s7 99799 1041 88779 2% /state
/dev/dsk/c0t1d0s0 70584033 42845587 27032606 100% /d01
fas-dc-1:/vol/Software
157286400 132940788 24345612 85% /software


My /d01 mount point is 100% full so navigating to the log location on /d01/oracle/oasoim/ldap/odi/log

I removed the log files and Trace files created by Oracle HR Agent and IPlanet Export profile

oimapp$ rm -rf *trc* *aud*

Now again I tried to start opmn and it started successfully and the error was corrected.

Cheers
Aashish