The successStatusType must be of success type and not equal to null - SOA Suite 11.1.1.7 Upgrade

problem:

After upgrading current 11.1.1.4 environment to 11.1.1.7, observed below errors in SOA server Logs. This error comes every time while starting of each of the SOA managed servers.


 <Error> <oracle.sdp.messaging.engine> <SDP-25088> <Unable to refresh the driver locator cache, due to the following error:
Exception Description: The method [setSuccessStatusType] on the object is throwing an exception.
Argument: [null]
Internal Exception: java.lang.reflect.InvocationTargetException
Target Invocation Exception: java.lang.IllegalArgumentException: The successStatusType must be of success type and not equal to null
Mapping: oracle.toplink.mappings.TransformationMapping[successStatusType]
Descriptor: RelationalDescriptor(oracle.sdpinternal.messaging.AddressImpl --> [DatabaseTable(ADDRESS)])>
 <Warning> <oracle.toplink.default> <BEA-000000> <
Exception [TOPLINK-106] (Oracle TopLink - 11g Release 1 (11.1.1.6.0) (Build 111018)): oracle.toplink.exceptions.DescriptorException
Exception Description: The method [setSuccessStatusType] on the object is throwing an exception.
Argument: [null]
Internal Exception: java.lang.reflect.InvocationTargetException
Target Invocation Exception: java.lang.IllegalArgumentException: The successStatusType must be of success type and not equal to null
Mapping: oracle.toplink.mappings.TransformationMapping[successStatusType]
Descriptor: RelationalDescriptor(oracle.sdpinternal.messaging.AddressImpl --> [DatabaseTable(ADDRESS)])


Similar errors are observed when User Messaging Service option is accessed.

a. Login to the Enterprise Manager.
b. From the left hand tree expand 'User Messaging Service' and select 'usermessagingserver (ServerName)'.
c. In the right hand pane select 'User Messaging Service' drop down and then 'Message Status'.
d. We get the same error on the em console and in the logs.

Cause:

In ORASDPM database, we can find that for all the rows, the success_status column is null.

select count(*) from address where success_status is null

Solution:

Execute following SQL Commands in your ORASDPM database.

SQL> conn _ORASDPM/:   -- Connect ORASDPM database of your environment
SQL> update address set success_status='DELIVERY_TO_GATEWAY_SUCCESS' where success_status is null;
SQL> commit;


Restart managed SOA Servers and this error shoud no longer exist.


API Reference @ http://docs.oracle.com/cd/E29542_01/apirefs.1111/e14011/index.html?oracle/sdp/messaging/Address.html. Look for Enum StatusType; apparently, it cannot have the null value.


Hope this helps

Cheers
Nirav

Enterprise Manager performance very slow after upgrading WLS/SOA Suite to 11.1.1.7

After upgrading the SOA Suite 11.1.1.4 cluster server to 11.1.1.7 (PS6), it is observed that the em Enterprise Manager Console (em console) is loading very slow and sometimes takes more than 120 seconds. Sometimes, trying to access any composite link will time out and leads to em home page again. In short, Fusion Middleware Control UI performance goes unpredictable in upgraded environment.

Log Files may show following statement:

ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN emd.targets setContents.101 - Label Servlet/JSP By Server - Request Processing Time (since startup) (ms) very long. Has been truncated.

< Error > < oracle.adfinternal.view.faces.config.rich.RegistrationConfigurator > < BEA-000000 > < ADF_FACES-60096:Server Exception during PPR, #2
java.net.SocketException: Socket closed

< BEA-001153 >

Cause: 
As the number of fmwc monitored targets increases, the login times on em console goes up as well. fmwc discovery is always performed as part of login.  Time delay or Time out is expected because EM discovery MBeans need to be invoked for every target.

Solution:

Oracle has provided solution to this problem under Oracle Support RaV Note "How to Enable Discovery Cache To Avoid Long Delay During Login To Fusion Middleware Control (Doc ID 1423893.1)".

Solution is to cache the discovery results in the servlet context and use it for subsequent logins. This discovery result will be shared by all the fmwc users. This will still require the entire discovery to be done at least once. If the initial login is slow due to discovery of large number of targets, changes in this note will not reduce the initial login delay.

• If the caching is enabled, fmwc will use the cached discovery results.
• The default setting is "not use the cached results"

However, this solution is applicable only to version 11.1.1.2.0 to 11.1.1.6.0.  If you are on pre-11.1.1.6 environment, please install the patch 13251077. Then follow the steps below.

If you are on 11.1.1.6 or on 11.1.1.7,  Patch installation is not required as it is already part of 11.1.1.6+ version.

1.  Open Enterprise Manager Console. On the left panel, navigate to Weblogic Domain --> . Right click and from the context Menu select the option "System MBean Browser". 

2. Access the MBean Propery in the tree at Application Defined MBeans --> emosms.props --> Server: Your_Admin_Server_Name_Here --> Application:em --> properties --> emoms.properties. Then on top of theright hand panel expand the option Show MBean Information", it should look something like below:

emoms.props:Location=AdminServer,name=emoms.properties,type=Properties,Application=em

3.  On Right Panel, Select tab Operations, click on setProperty

4.  Enter following three key-value pair one by one and Invoke the setProeprty() operation.

# Enable caching of  Discovery data and use it for other subsequent users.
# Values=true/false Default=false
Key:    oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_USE_CACHED_RESULTS

Value: true
# If caching of discovery data is true, this parameter indicates how long the discovery data
# from cache should be used before requiring a fresh discovery.
# Time value is in milliseconds. Default is 7200000 milliseconds.
Key:    oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_MAX_CACHE_AGE

Value: 7200000 
# If caching of discovery data is true, a user logs in and a discovery session is in progress,
# this parameter indicates how long the user can wait for current discovery to complete.
# After this wait time is elapsed and discovery is still not finished: If there is already data
# in cache it will be used, else the user will launch a new discovery session.
# Time value is in milliseconds. Default is 10000 milliseconds.
Key:    oracle.sysman.emas.discovery.wls.FMW_DISCOVERY_MAX_WAIT_TIME

Value: 10000 

We can go to Attributes Tab and verify that the new properties are available under Properties option.
5.   Once Values are set,  Refresh the Farm. You can now observe the difference in em page performance. If newly added components or their changed values are not getting reflected on em console, then also we should use the option to Refresh the Farm.

Now, login again to the em console and performance improvement should be observed while accessing components on the home page.

Issue with above solution:

As caching is enabled and Cache Age is set to 7200000 , it is possible that 

1. When one of the managed server nodes are restarted, the status of the recently started managed server and its components are not displayed with the correct status till cache age is expired. This happens when you are working on the same browser screen. One can avoid this by opening em console in a different browser. 
2. New Application deployment made through wlst/ant scripts or using JDev are not reflected on the em console till cache age expires.

In such cases, solution is to reset the values of the caching properties as required for the specific environment. It is possible that we may decrease the Cache Age value but may keep the Max Wait Time values somewhere >6500. Also, there is one more property "oracle.sysman.core.model.host.cacheAliveTime" exists for the same MBean. We can tweek its values to improve em screen perofrmance. You might also want to increase the value for "oracle.sysman.emRep.dbConn.statementCacheSize" in case of bigger cluster environments.

The solution steps can also be executed with the WLST scripting. More information on wlst commands, please go through the Oracle Support Doc ID 1423893.1. Please note that you cannot Refresh Farm using WLST script and manual refresh is required. 

Hope this helps

Cheers

Nirav

How to serch composites for BPEL process defined index values in soa suite 11g

Problem:

I have migrated my 10g BPEL processes to 11g. Few BPEL processes, based on input values they receive, sets the 1 to 6 index values. For a process where instance volume is higher/large, it is easy to search the instance using the index value. In 10g, I used to search on BPEL Console --> Instances tab, using "Index/Custom Key" text box. But as soon as I went to 11g, I found there is no option on em(Enterprise Manager) console to search BPEL instance using index.

After research, I found out that in 11g Oracle has not provided option to serach BPEL instance using those indexes. You can search for composites using composite indexes, but not the BPEL instances using BPEL process defined indexes. However, the values are still being stored in CI_INDEXS table under soainfra database schema.

Some simple but very useful 10g functionality is omitted even after the release of 11.1.1.4 release.

solution:

Simplest way, fire a query and get the instance ID from the ci_indexes table. But wait, it is not that easy. Let us see what exactly happens there.

I have a composite TestCompsite1. My bpel process TestHelloWorld is part of the composite TestComposite1. During execution, the bpel process setting values in ci_indexes table, say index_1 = "OracleWorld".

You can search the ci_indexes table using below query

select * from ci_indexes where index_1 like 'OracleWorld' (Add more OR conditions as you may want to search on other index keys)

You will get bpel instance number as cikey column value. But you cannot search using that output on em console as it only allows you to search based on the compsoite instance number of composite instance index values. The solution is below query, join ci_indexes with cube_instance

select a.cmpst_id cube_composite_id, a.cikey cube_cikey, b.cikey index_cikey

from cube_instance a, ci_indexes b

where a.cikey = b.cikey

and index_1 like 'OracleWorld'

the column value of cube_composite_id is what exactly you will search on em console for. This will open composite. If you have multiple bpel execution, then enable "show Instance ID" option. Use the index_cikey column value to reach the correct bpel instance.

I thought this would be easy when I saw values in ci_indexes table, but it required to look further in soainfra db. For simplicity, one may like to wrap this query in report to make the life easier.

Hope this will help

Cheers

Nirav