Weblogic Stuck threads

One way to check if you have stuck threads is through Weblogic Admin console.

1. Login to weblogic admin console with admin credentials.
2. Expand Environment and click on Servers in the left menu
3. Click on the Admin server or any managed server you want to check the stuck threads
4. Click on the Monitoring tab in the server properties
5. click on Threads tab.
6. Check the Health column in Self-Tuning Thread Pool table. If it says OK there are no stuck threads.
7. If Health column value shows "Warning" there are stuck threads in JVM.
8. Next thing is that you will have to find the root cause of the stuck threads. For that JVM thread dump needs to be taken.
9. There is a tool called jstack starting from JDK 1.6 version.
10. Find out the process ID of the weblogic service and execute "jstack <<PID>> > threaddump.log"command.
11. Above command generates threaddump.log file with JVM thread dump.
12. Check the STUCK threads and fix the issue. It might be in application code or connection to external services.

Happy debugging.

Thanks

Siva Pokuri.

This server is being started in managed server independence mode in the absence of the admin server

Error:-

<Mar 14, 2018 9:05:45 AM IST> <Emergency> <Management> <BEA-141151> <The admin server could not be reached at http://192.168.65.153:7001.>

<Mar 14, 2018 9:05:45 AM IST> <Info> <Configuration Management> <BEA-150018> <This server is being started in managed server independence mode in the absence of the admin server.>

<Mar 14, 2018 9:05:45 AM IST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>

<Mar 14, 2018 9:05:45 AM IST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>

<Mar 14, 2018 9:05:45 AM IST> <Notice> <Log Management> <BEA-170019> <The server log file /u03/user_projects/domains/oimdomain/servers/oim_server1/logs/oim_server1.log is opened. All server side log events will be written to this file.>

<Mar 14, 2018 9:05:53 AM oracle.security.jps.internal.idstore.util.LibOvdUtil pushLdapNamesToLibOvd

INFO: Pushed ldap name and types info to libOvd. Ldaps : DefaultAuthenticator:idstore.ldap.provideridstore.ldap.

Mar 14, 2018 9:05:53 AM oracle.security.jps.az.internal.runtime.pd.register.PDPRegister run

INFO: PDP registration succeeded.

Mar 14, 2018 9:05:54 AM oracle.iam.platform.auth.providers.wls.OIMAuthenticationProvider initialize

INFO: Authentication module initialized

<Mar 14, 2018 9:05:56 AM IST> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>

<Mar 14, 2018 9:05:56 AM IST> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: java.lang.SecurityException: Method 'getAdministrationURL' cannot be invoked without administrator access

java.lang.SecurityException: Method 'getAdministrationURL' cannot be invoked without administrator acces at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:237)at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:223) at weblogic.server.channels.RemoteChannelServiceImpl_1036_WLStub.getAdministrationURL(Unknown Source)  at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:184)at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:147)Truncated. see log file for complete stacktrace

Caused By: java.lang.SecurityException: Method 'getAdministrationURL' cannot be invoked without administrator access at weblogic.rmi.internal.AdminAccessOnlyServerRef.getWorkManager (AdminAccessOnlyServerRef.java:29)at weblogic.rmi.internal.BasicServerRef.getWorkManager(BasicServerRef.java:442) at weblogic.rmi.internal.BasicServerRef.dispatch(BasicServerRef.java:358 at weblogic.rmi.internal.BasicServerRef.dispatch(BasicServerRef.java:1022)at weblogic.rjvm.RJVMImpl.dispatchRequest(RJVMImpl.java:1173)

        Truncated. see log file for complete stacktrace

<Mar 14, 2018 9:05:56 AM IST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>

<Mar 14, 2018 9:05:56 AM IST> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>

<Mar 14, 2018 9:05:56 AM IST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>


Resolution:-

In my case, we are troubleshooting issues in DR OIM 11.1.2.3 and server went to hung state
after the server was rebooted Weblogic Amin URL is pointing to a different IP address.

Note:-We need to point Weblogic Admin URL to correct IPAddress ie:-192.168.65.153:7001.
            Then take backup and remove tmp and cache and start Admin and Managed servers.




Thanks,
Aditya.

Weblogic samples vulnerability in 10g and 12c

Weblogic vulnerability fix provided by Oracle in the latest Oracle Security Alert for CVE-2017-9805

Click below link to check bug details.

http://www.oracle.com/technetwork/security-advisory/alert-cve-2017-9805-verbose-3889406.html#FMW

Thanks

Siva Pokuri.

TIP: Enable HTTP debug in Weblogic server

Steps:

1. Stop all the weblogic admin servers & managed servers

2. Add below list to JAVA_OPTIONS in setDomainEnv.sh file in the weblogic domain.

-Dweblogic.log.StdoutSeverity=Debug
-Dweblogic.log.LogSeverity=Debug
-Dweblogic.log.LoggerSeverity=Debug
-Dweblogic.debug.DebugHttpSessions=true
-Dweblogic.debug.DebugReplication=true
-Dweblogic.debug.DebugHttp=true
-Dweblogic.debug.DebugHttpLogging=true

3. If you want to enable HTTP debug for any specific admin or managed server follow below steps.

a. Login to Weblogic Admin console

b. Click on “Servers” link

c. Click on the Admin server/ managed server and click on debug

d. Expand weblogic

e. Expand servlet

f. Select “DebugHTTP”

g. Select “DebugHTTPSessions”

h. Select “DebugHTTPLogging”

i. Navigate to weblogic>>core>>cluster

j. Select “DebugReplication”

k. Click on “Enable” button

l. Click on “Activate Changes”

4. Navigate to Environment >>Servers in the left panel

5. Click on the server name

6. Click on Logging tab

7. Expand Advanced tab

8. Change the severity level from “Trace” to “Debug”

9. Click on “Save” button

10. Click on “Activate Changes”

11. Start Weblogic admin & managed servers in the cluster

12. Check the server log files

-- Siva Pokuri.

How to Installation OID 11g (11.1.1.7.0)

Note: Assuming Weblogic Server 11g (10.3.6) & Oracle Database 11g installed

Steps:

1) Download Oracle Internet Directory 11g RCU & Installer from OTN
   
     URL: http://www.oracle.com/technetwork/middleware/downloads/oid-11g-161194.html

2) Follow below steps to create OID database schema

3) Install Oracle Internet Directory using IDM 11g suite

-- Siva Pokuri.

How to integrate OAM & OAAM 11g R2 PS2 (11.1.2.2.0)

Oracle Access Manager & Oracle Adaptive Access Manager 11g R2 PS2(11.1.2.2.0) Integration

Environment:

-- Oracle Access Manager 11g R2 PS2(11.1.2.2.0)

-- Oracle Adaptive Access Manager 11g R2 PS2(11.1.2.2.0)

-- Oracle Database 11g

-- Oracle WebLogic Server 10.3.6

-- Oracle Enterprise Linux 64-bit

-- Oracle Repository Creation Utility 11g R2 PS2(11.1.2.2.0)

-- Oracle HTTP Server 11g R1

-- OAM WebGate for OHS webserver 11.1.1.7.0

-- OAM user store is Weblogic embedded LDAP

Steps:

-- Login to Weblogic Administration console and create "oaamadmin" user and assign all the OAAM admin privileges.

-- Login to OAAM Admin console and add property as shown in the below screen shot

-- Follow below screen shots to create TAP key Store.

-- Assign a password for IAMSuiteAgent in OAM Admin console and update the same in Weblogic Administration console Security Realm>> IAMSuite Agent provider

-- Restart all OAM and OAAM admin & managed servers.

-- Update TAP Scheme challenge parameters as shown in the below screen shots.

-- Setup OAM Integration with OAAM.

-- Update the OAM, OAAM details in oaam_cli.properties file as shown in the below screen shot.

-- Set ORACLE_MW_HOME environment variable and execute setupOAMTapIntegration.sh as shown in the below screen shots

-- Enter "oaamadmin" user(which we created in the first step) and enter the password followed by OAAM database credentials and TAP keystore password. 

-- Verify oaam.uio.security.mode property value set to 1 in OAAM Admin console properties.

Testing:

Change any existing protected resource Authentication Schema from "LDAP Scheme" to "TAPScheme" and try to access the protected resource and try login with a valid user in the directory server.

-- Successfully logged in with OAAM security profile setup.

Hope this helps.

Thanks

Siva Pokuri.