Enterprise Manager performance very slow after upgrading WLS/SOA Suite to 11.1.1.7

After upgrading the SOA Suite 11.1.1.4 cluster server to 11.1.1.7 (PS6), it is observed that the em Enterprise Manager Console (em console) is loading very slow and sometimes takes more than 120 seconds. Sometimes, trying to access any composite link will time out and leads to em home page again. In short, Fusion Middleware Control UI performance goes unpredictable in upgraded environment.

Log Files may show following statement:

ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN emd.targets setContents.101 - Label Servlet/JSP By Server - Request Processing Time (since startup) (ms) very long. Has been truncated.

< Error > < oracle.adfinternal.view.faces.config.rich.RegistrationConfigurator > < BEA-000000 > < ADF_FACES-60096:Server Exception during PPR, #2
java.net.SocketException: Socket closed

< BEA-001153 >

Cause: 
As the number of fmwc monitored targets increases, the login times on em console goes up as well. fmwc discovery is always performed as part of login.  Time delay or Time out is expected because EM discovery MBeans need to be invoked for every target.

Solution:

Oracle has provided solution to this problem under Oracle Support RaV Note "How to Enable Discovery Cache To Avoid Long Delay During Login To Fusion Middleware Control (Doc ID 1423893.1)".

Solution is to cache the discovery results in the servlet context and use it for subsequent logins. This discovery result will be shared by all the fmwc users. This will still require the entire discovery to be done at least once. If the initial login is slow due to discovery of large number of targets, changes in this note will not reduce the initial login delay.

• If the caching is enabled, fmwc will use the cached discovery results.
• The default setting is "not use the cached results"

However, this solution is applicable only to version 11.1.1.2.0 to 11.1.1.6.0.  If you are on pre-11.1.1.6 environment, please install the patch 13251077. Then follow the steps below.

If you are on 11.1.1.6 or on 11.1.1.7,  Patch installation is not required as it is already part of 11.1.1.6+ version.

1.  Open Enterprise Manager Console. On the left panel, navigate to Weblogic Domain --> . Right click and from the context Menu select the option "System MBean Browser". 

2. Access the MBean Propery in the tree at Application Defined MBeans --> emosms.props --> Server: Your_Admin_Server_Name_Here --> Application:em --> properties --> emoms.properties. Then on top of theright hand panel expand the option Show MBean Information", it should look something like below:

emoms.props:Location=AdminServer,name=emoms.properties,type=Properties,Application=em

3.  On Right Panel, Select tab Operations, click on setProperty

4.  Enter following three key-value pair one by one and Invoke the setProeprty() operation.

# Enable caching of  Discovery data and use it for other subsequent users.
# Values=true/false Default=false
Key:    oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_USE_CACHED_RESULTS

Value: true
# If caching of discovery data is true, this parameter indicates how long the discovery data
# from cache should be used before requiring a fresh discovery.
# Time value is in milliseconds. Default is 7200000 milliseconds.
Key:    oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_MAX_CACHE_AGE

Value: 7200000 
# If caching of discovery data is true, a user logs in and a discovery session is in progress,
# this parameter indicates how long the user can wait for current discovery to complete.
# After this wait time is elapsed and discovery is still not finished: If there is already data
# in cache it will be used, else the user will launch a new discovery session.
# Time value is in milliseconds. Default is 10000 milliseconds.
Key:    oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_MAX_WAIT_TIME

Value: 10000 

We can go to Attributes Tab and verify that the new properties are available under Properties option.
5.   Once Values are set,  Refresh the Farm. You can now observe the difference in em page performance. If newly added components or their changed values are not getting reflected on em console, then also we should use the option to Refresh the Farm.

Now, login again to the em console and performance improvement should be observed while accessing components on the home page.

Issue with above solution:

As caching is enabled and Cache Age is set to 7200000 , it is possible that 

1. When one of the managed server nodes are restarted, the status of the recently started managed server and its components are not displayed with the correct status till cache age is expired. This happens when you are working on the same browser screen. One can avoid this by opening em console in a different browser. 
2. New Application deployment made through wlst/ant scripts or using JDev are not reflected on the em console till cache age expires.

In such cases, solution is to reset the values of the caching properties as required for the specific environment. It is possible that we may decrease the Cache Age value but may keep the Max Wait Time values somewhere >6500. Also, there is one more property "oracle.sysman.core.model.host.cacheAliveTime" exists for the same MBean. We can tweek its values to improve em screen perofrmance. You might also want to increase the value for "oracle.sysman.emRep.dbConn.statementCacheSize" in case of bigger cluster environments.

The solution steps can also be executed with the WLST scripting. More information on wlst commands, please go through the Oracle Support Doc ID 1423893.1. Please note that you cannot Refresh Farm using WLST script and manual refresh is required. 

Hope this helps

Cheers

Nirav