SOA 11g File Adapter Pre & Post Processing Technique

File Adapter is one of the most feature rich & powerful technology adapter that is available with Oracle SOA Suite. In this article, I will be explaining how to perform pre-processing of a file when it is being read by the File Adapter in SOA suite with a use-case.

In an enterprise scenario a file process operation is not as simple as we would like it to be. Often there are situations where files of different formats that might be required to be processed selectively. Also, the files could be packaged & encrypted in one common format. How do we handle this scenario?

In Oracle SOA Suite, the File Adapter based on JCA 1.5 architecture provides feasibility to perform pre/post processing of file that are being processed (read or write). This feature is known as the "Pipeline & Valve" concept of the File Adapter. As you can see from the diagram below, the file adapter polls a specific folder from the file system - The file stream then passes through the pipeline (which holds a sequence of custom defined valves). A pipeline can have 'n' number of valves which will be processed in the order specified. The valves are nothing but content processors where a Java call be made.

The scenario that we are going to consider in this post is a toned down version of a real-time business use-case but sufficient enough to drive home the pre/post processing capabilities of the file adapter.

Use-Case:

A bunch of applications publish files of various different formats to a common folder which should be processed. However, only files of certain formats must be picked up for processing and the rest left to be processed by a different system.

This is a classic case where the SOA suite file adapter's pre-processing technique could be put to use in determining the file format before processing a file.

Step 1: Building the custom valve to determine mime type of the file

Let us first start by creating the valve (Java class) which will determine the file format. All valves must implement either the Valve interface or StagedValve interface. Optionally, to make it easier, the custom valve class can also extend AbstractValve or AbstractStagedValve classes.

In this case, I have created a MimeValidationValve Java class which extends AbstractValve. Only files which have the following mime types are allowed.

text/plain
application/pdf
application/msword

This is configurable in the properties file. I have made use of Apache Tika open source toolkit which is very powerful in extracting metadata information of the files polled. Read more on Apache Tika here.

One important point to note here is that you will need to add the bpm-infra.jar library to the Java project in order to compile the valve class. This library is available under  <Middleware_Home>/Oracle_soa1/soa/modules/oracle.soa.fabric_11.1.1

Project Properties -> Libraries & Classpath -> Add Jar/Directory

You can download the valve Java project here

Step 2: Building the SOA application to process files

You can download the SOA project here. This project has a 'Read' file adapter to poll for files from the file system, a 'Write' file adapter to write files back to file system and a BPEL process connecting these adapters.

Step 3: Creating the Pipeline XML file

Recall that a pipeline can have one or more valves which will be processed in the order specified. The pipeline XML file will hold references to the various different valves that must be invoked by the file adapter before the translation phase or handing it over to the SOA runtime.

Let us now go back to the SOA project and create the pipeline XML file. Let us review the FilePipeLine.xml under the DGFileProcessor SOA application.


<?xml version="1.0" encoding="windows-1252" ?>
<pipeline xmlns="http://www.oracle.com/adapter/pipeline">
  <valves>
    <valve>mimetype.MimeValidationValve</valve>
  </valves>
</pipeline>

Note that any number of valves can be added to the pipeline by just adding the <valve> elements.

Step 4: Plumbing the Pipeline with the appropriate file adapter

Now that we have the valves created and the pipeline/valve configurations done, the final piece that remains to be done is to let the file adapter trigger pipeline processing. To do this, open the corresponding file adapter's jca file and add the following property.

<property name="PipelineFile" value="FilePipeLine.xml"/>

That completes the solution for the use-case.

WebCenter Spaces OBIEE Integration

This post is more of a self-reference material for the future. Thought it might help others looking for it as well.

Wouldn't it be a great idea for enterprises which already have WebCenter Spaces & OBIEE technologies to leverage the integration feature? Yes, think about showing BI analytics to your business users / stakeholders through the WebCenter spaces portal solution.

There are a set of configurations that needs to be done at both OBIEE and the WebLogic server level to enable single sign-on for seamless integration. A very good detailed step-by-step instruction on achieving the integration is documented here.

http://www.oracle.com/technetwork/middleware/webcenter/portal/learnmore/wc-biappsconfig-396418.pdf

SOA 11g MDS - Share Common Artifacts

As promised on my previous post, I am detailing the step-by-step instructions which allows for sharing of common artifacts (schema files/WSDLs) among SOA projects and more importantly helps in loose coupling of service bindings & remove service dependencies.

A typical use case is where the abstract WSDL definition of an external service can be stored in MDS and referenced from the SOA composite which will loosely couple the services and also guard against 'SOA Composite Invalidation' issues.

SOA 11g provides us with MDS repository (both file based & DB based) where common artifacts can be stored and referenced by all projects in the SOA infrastructure.

File based MDS is used during design/deploy time where as DB based MDS is used at run-time by the SOA infra. A configuration file named adf-config.xml that is part of every SOA project manages this partition mapping to be used by JDeveloper during design/deploy time.

All SOA projects that are deployed on the SOA infrastructure are stored in the MDS repository. If you create a connection from the JDeveloper 'Resource Palette' to the SOA MDS, there will be a 'deployed composites' folder which will contain all deployed artifacts.

Connection Information for the MDS connection:
username: XXX_MDS (usually DEV_MDS)
password: xxxxxxxxx
MDS Partition: soa-infra

The file based MDS repository is generally the 'seed' partition under a typical JDeveloper install;

<JDev_Home>\integration\seed


Under this partition there is a root folder (soa) available by default which contains all the configuration files, schemas & WSDLs required by various different SOA components during design time. These files are also pre-deployed on the database MDS repository.

We are now free to create custom folders under the 'seed' partition to hold artifacts that are common references in a SOA environment. Generally, we create an 'apps' folder under the 'seed' partition which in turn holds various folders to hold various artifact types;

 Once the artifacts that needs to be shared/referenced are placed under the appropriate namespace (in this case 'apps'), we can let the SOA project in JDeveloper to point to MDS for these artifacts from a common location.

adf-config.xml configuration:

As noted earlier, adf-config.xml is an important configuration file which holds the details regarding the MDS (both file based & DB based). By default, this file declares the 'seed' partition & the metadata store for 'soa' namespace.

This file can be found under <YourApplication>\.adf\META-INF.
Alternatively, in JDeveloper, expand the 'Application Resources' panel; drill through Descriptors -> ADF META-INF to find this file

Step 1: Just add the metadata store namespace for '/apps' in additions to default '/soa/shared' namespace - This is to let the JDeveloper reference the artifacts under the '/apps' during the design time from the file system metadata store.

<adf-mds-config xmlns="http://xmlns.oracle.com/adf/mds/config">
    <mds-config xmlns="http://xmlns.oracle.com/mds/config">
      <persistence-config>
        <metadata-namespaces>
          <namespace path="/soa/shared" metadata-store-usage="mstore-usage_1"/>
          <namespace path="/apps" metadata-store-usage="mstore-usage_1"/>
        </metadata-namespaces>
        <metadata-store-usages>
          <metadata-store-usage id="mstore-usage_1">
            <metadata-store class-name="oracle.mds.persistence.stores.file.FileMetadataStore">
              <property name="metadata-path"
                        value="${oracle.home}/integration"/>
              <property name="partition-name" value="seed"/>
            </metadata-store>
          </metadata-store-usage>
        </metadata-store-usages>
      </persistence-config>
    </mds-config>
  </adf-mds-config> 

Step 2: In the SOA composite, reference the common artifacts from MDS as shown below;

 <import namespace="http://xmlns.oracle.com/bpmn/bpmnProcess/CreditCardProcess"
          location="oramds:/apps/orderprocess/common/CreditServiceProcess.wsdl"
          importType="wsdl"/>

Now that we have successfully referenced the SOA project to refer the file based MDS it is barely enough only during design time. For deploy time & run time, these artifacts must be made available in the DB based MDS.

There are two methods available for us to deploy the artifacts in SOA DB based MDS;

Approach 1: Ant based script which allows to deploy the artifacts into SOA MDS. I am not going to get to the details of this as this has been blogged in detail by Edwin Biemond; Reference below;

http://biemond.blogspot.com/2009/11/soa-suite-11g-mds-deploy-and-removal.html

Approach 2: For a simple JDeveloper based SOA deployment follow the steps below;

Step 1: Create a new application (Generic Application)

Step 2: Name the application (eg., OrderProcessMDSApp) and click 'Next'

 Step 3: Provide a name to the project (eg., OrderProcessMDSProj) and click 'Finish'

 Step 4: Now, right click on the OrderProcessMDSProj and choose 'Project Properties'. In the 'Project Properties' dialog, choose 'Deployment' and create a new deployment profile by clicking on 'New...' button

 Step 5: In the 'Create Deployment Profile' dialog, leave the default 'Archive Type' to 'JAR File' and name the deployment profile (eg., soamdsarchive) and click 'OK'

 Step 6: This will open up the 'Edit JAR Deployment Profile Properties' dialog. Click on the 'Contributors' section under 'File Groups -> Project Output'.

Step 7: Click on the 'Add..' button and choose the folder where the 'content to be deployed' is present. Here the '/apps' namespace under the file based MDS can be chosen where the content is already placed & available.

Step 8: Now, click on the 'Filters' section and choose the artifacts under the '/apps' that you would like to deploy or upload to MDS database repository

Step 9: Click on OK to complete the deployment profile creation process. Now, we have created a project JAR deployment profile which contains the artifacts to be deployed on to the SOA MDS. Now, we will have to create a application deployment profile of 'soa bundle' type which will be deployed to the soa-infra.

Step 10: From the 'Application Menu', choose 'Deploy' -> 'New Deployment Profile..' option

Step 11: Under the 'New Gallery' go to 'Deployment Profiles' choose 'SOA Bundle' and click OK

Step 11: Under the 'Create Deployment Profile', provide a name to the application deployment profile (eg., soabundle1) and click OK.

Step 12: In the 'SOA Bundle Deployment Profile Properties' go to 'Dependencies' and choose the JAR archive that we created earlier eg., soamdsarchive and click OK

Finally, deploy the application to the soa server. This will ensure that the artifacts under the '/apps' metadata namespace is deployed on to the SOA MDS database repository.

You can verify the same from the SOA MDS connection created under the 'Resource Palette'. This should now reflect the 'apps' folder at the same level as 'soa' metadata store.

Now, if you try to deploy the SOA project which references MDS you may hit a deployment error complaining that the resource referenced is not found. This is because the JDeveloper consults adf-config.xml configuration file during design & deploy time (refer back to the initial architecture diagram). Since there is no information about the DB based MDS in the adf-config the deployer complains.

To fix this, go back to the adf-config.xml file and add the DB metadata store information so that the deployer can read & locate the referenced resources from MDS.

<adf-mds-config xmlns="http://xmlns.oracle.com/adf/mds/config">
    <mds-config xmlns="http://xmlns.oracle.com/mds/config">
      <persistence-config>
        <metadata-namespaces>
          <namespace path="/soa/shared" metadata-store-usage="mstore-usage_2"/>
          <namespace path="/apps" metadata-store-usage="mstore-usage_2"/>
        </metadata-namespaces>
        <metadata-store-usages>
          <metadata-store-usage id="mstore-usage_2">
            <metadata-store class-name="oracle.mds.persistence.stores.db.DBMetadataStore">
              <property value="DEV_MDS" name="jdbc-userid"/>
              <property value="xxxxxxx" name="jdbc-password"/>
              <property value="jdbc:oracle:thin:@hostname:1521:orcl"
                        name="jdbc-url"/>
              <property value="soa-infra" name="partition-name"/>
            </metadata-store>
          </metadata-store-usage>
        </metadata-store-usages>
      </persistence-config>
    </mds-config>
  </adf-mds-config>

That completes our configuration and eventually this elaborate post. Now, the deployment will be smooth as we have referenced the DB MDS in the adf-config.xml file.

Hope this post helped you understand & implement the SOA MDS related activities. Let me know your feedback through comments.

Resolving Invalid State of SOA Composite

Ever wondered why a SOA composite goes to invalid state when the SOA server starts up? No matter what you try - restarting the composite from within the fusion middleware control (EM console) or undeploying - nothing would work. Only a server restart "Might" rid you of the issue.

I am carefully using "Might" here because, the SOA composite under consideration might refer to a service which could be down when the composite was started. This happens because;

The SOA server doesn't know of the service dependencies that a SOA composite has.

For instance let us consider a SOA composite A referencing external services B, C & D deployed on the same server instance. Ideally, we would like to have a service startup order where the services B, C & D are started ahead of composite A.

But that is not the reality as there is no way to set the service startup priority on the SOA server. It wouldn't be a nice idea to weave this setting at the server level as well because, consider a cyclic dependency issue where composite A references composite B and vice versa which would be too complex & cause instability.

This is a classic case for any SOA environment and the issue here is because the concrete WSDL references are used in the SOA composite to connect with the external services.

Now the question is how do we resolve these service dependency issues?

Answer lies in leveraging the SOA infrastructure capability. If you look closely into any SOA composite (source code), there will be a set of common services that will be referenced from the metadata store (MDS);


<import namespace="http://xmlns.oracle.com/bpel/workflow/taskService"
          location="oramds:/soa/shared/workflow/TaskServiceInterface.wsdl"
          importType="wsdl"/>


These service definitions are stored in the MDS repository which is available all the time for the SOA services to reference provided the underlying DB is up & running. You can check this for yourself by establishing a SOA MDS repository connection from the JDeveloper resource palette.

Similarly, as a best practice you can store all the abstract WSDL definitions of the referenced services in the MDS repository & reference them in your SOA composite. This way whenever the server tries to start the services, irrespective of the service startup order, the SOA composite starts up successfully as it will always have the abstract WSDL reference - After all, at design/compile time, the only thing that a SOA composite requires is the abstract WSDL definition. Only at runtime does the composite would require a concrete WSDL to invoke the right service.

Now, how do we let the design & runtime environments know what service reference (abstract/concrete) to use?

After storing the referenced abstract WSDLs & XSDs in the MDS repository, make the following changes to effect this;

1. Update the import statements in your composite XML to refer the WSDL/XSDs from concrete references to abstract ones;

<import namespace="http://www.oracle.com/orderprocess"
location="oramds:/apps/orderprocess/common/orderprocess_submitorder.wsdl" importType="wsdl"/>

2. Update the service wsdl location to MDS abstract WSDL reference as the soa engine would only look at this element to look up & start the services;

<reference name="SubmitOrder_OrderProcess" ui:wsdlLocation="oramds:/apps/orderprocess/common/orderprocess_submitorder.wsdl">

Leave the WSDL URL in the binding layer intact (to refer to concrete WSDL) because at runtime, the bindings will be used to invoke the actual web service.

To read more about abstract & concrete WSDLs, I recommend you to read my previous blog post here

How to store these artifacts in MDS repository? Ways to do it along with step-by-step instructions in my next post. Here it is!

BPM 11g Subprocess (Default, Loop, Multi-Instance & Event based) Exemplified

With some common use-case references, I will be making an effort today to explain how, when & where various different sub-processes are used in BPM 11g. A subprocess is nothing but an embedded/nested subprocess within the parent process.

A subprocess - in a very simple use-case can be used to organize & modularize certain set of activities based on its function for better readability.

One important point to note here is that a subprocess must have an incoming & outgoing sequence flow akin to other BPM activities with the only exception of an event-based subprocess.

Oracle BPM 11g offers two variants of subprocess activities;

1. SubProcess (Default): This can be implemented in 3 different fashions based on its loop characteristics;

a) None: Primitive form of a subprocess to just organize & modularize activities within a certain business function as depicted in the diagram. For example, in an order process BPM process, there may be certain activities such as getting credit score of the customer from a credit bureau agency (external), checking inventory of the products in the order from the inventory management system (example JD Edwards) which constitutes a business function namely "Pre-Process Order" which can be grouped under a subprocess.

b) Loop: Think of this option as synonymous to a regular programming language's for/while/do-while construct. When this option is chosen, a loop condition needs to be specified which will let the subprocess know the number of times it would loop itself before the process control flow comes out of it. The loop condition can be specified either as a simple or XPath expression. Additionally, "Loop Maximum" value can be specified which will act as a "break" condition to come out of the loop.

If you had noticed, I referenced for/while & do-while constructs. Interesting to note here that a do-while would process the loop at least once even if the condition evaluates to false where as a while construct would loop only if the condition evaluates to true. This is exactly what the "Evaluate Before" option lets us do. By default this option is checked which means that the loop construct would act as a while and if unchecked would behave like a do-while.

Extending the order process use-case which has multiple orders in a process (bulk order processing), where the individual orders need to be processed sequentially (You might ideally would like to have them processed in parallel. But for whatever reasons - may be human resource crunch, the process analyst decides to have the orders processed serially :). Let us keep the use case going!).

c) MultiInstance: This is very similar to a loop subprocess except that this option would enable parallelism. This allows multiple instances for each item in a collection either based on the cardinality or collection data. Cardinality allows loop condition to be specified where as collection accepts the collection data. For instance, if we have a collection of orders that needs to be processed in parallel rather than sequentially, we would opt in for a MultiInstance subprocess which creates a task for the Finance user for every order simultaneously.

2. Event SubProcess: This is the next variant of subprocess which can exist independently in a main BPM process which means that it doesn't need an input & output sequence flow. Whenever, an event is triggered this subprocess can listen to those events and get invoked. Similar to event based gateway, catch & throw events, the event-based subprocess can be triggered with a message event, timer event, signal or on error. Also, the same can be configured to be "Interrupting" which will break the main process flow on event.

Let us now add some flavor to the order process use-case by adding fraud handling capability. We will now build a event-based subprocess which will handle fraud and will be triggered based on a message. The finance reviewer can trigger a fraud event which will be sent to the fraud analysis team for further diagnosis.

That completes the BPM 11g subprocesses usage. Watch out for more on individual subprocesses & their intricacies in my next posts.

Abstract WSDL vs Concrete WSDL

In recent times, the abstract WSDL terminology is used more than ever and I am seeing more & more coffee-table discussions on this topic. In this post I will make an effort to provide an illustrated explanation in regards to the differences & usage of concrete and abstract WSDL.

First up, abstract WSDL is not a new technology. A typical WSDL (Web Services Description Language) consists of the following sections; types, messages, portType, binding & service elements. A quick look at what each of these sections define will ease the task of clarification of the differences;

Types - Defines the data type definitions for messages that will be exchanged by the web service. Generally defined from a schema (XSD)

Message - Defines the set of actual messages that will be exchanged. A message can consist of one or more parts which is a logical separation which together constitutes the message.

PortType - Defines the abstract operations provided/available and abstract messages involved. Operation refers to the messages involved in the transaction.

Bindings - Defines the message format and protocol information for operations defined by the portType.

Service - Defines the endpoint where the webservice will be exposed

Based on these definitions, we can figure out that the Types, Message & PortType sections of a WSDL definition is abstract (since it has no information pertaining to the communication protocol & how or where the service is exposed) where as the Bindings & Service section is concrete.

Hence, if we have a WSDL which just has these elements (excluding the Bindings & Service sections), then the WSDL is referred to as "Abstract WSDL". This WSDL definition cannot be used to connect to the webservice due to the 'missing information' on where & how the service is exposed. But there are certain advantages with abstract wsdl which we will see in a bit.

Definition of concrete WSDL now becomes a no-brainer. If a WSDL has all the sections - including the concrete parts such as Bindings & Service then it is referred to as a concrete WSDL. This WSDL defnition can be used to connect to the actual service since it has the protocol and service endpoint information.

Now, where exactly do we use an abstract WSDL?

Coming to the world of SOA, I am taking the example of Oracle SOA composites referencing external services. Ever wondered why some SOA composites are in "Invalid State" even though all the services that it references are up & available? This is because of the usage of concrete WSDL reference where the referenced service is not available yet when the caller service composite starts up. There is no way for the server to know about the service dependencies when it starts. By leveraging the concept of the abstract WSDL, we would be able to effectively resolve dependencies both at design time as well as runtime.

More on resolving service dependency issues in SOA on my next post here

Enabling Business-Driven Development Through Oracle BPM

Recently I was talking to the VP of Business Operations of a leading financial services company. The excerpt from the discussion below;

"How do I empower my business analysts/line of business users to take full control of our business processes?"

"After all, the people in our organization who best understands the business processes are our business analysts. But what we traditionally see is that the Information Technology trying to overwhelm and drive business which is a tragedy. I feel it is imperative for my business analysts to take control of our key business processes which are now driven by our IT whims & fancies."

"But the question that I would like to ask you is how to do it? and more importantly what solution can you offer?" 

The first thing that we need to understand here is that;

"There are really no IT projects in an enterprise. It is only the business processes that drives business which require IT to move it forward"

More often than not, this is forgotten. Honestly speaking the IT wing of a business knows very little about the business and business has very little control on the IT. Increasingly this is the same deficiency or insufficiency more and more businesses are facing. In trying to be agile on their business fronts they are unable to reflect at the same speed on their IT which has now become a critical part.

How to bridge this gap?

The answer lies in ensuring a seamless integration between the business and IT which we term as "Business-Driven Development".

Top three things to enable and sustain this approach are;

We need a technology that will

Equip your business analysts to design & model the business processes who is better placed to understand them

Enable them to share their business process models with the IT (Developers) to get them implemented. What we are trying to enable here is a top-down development approach driven purely by business

Empower stakeholders and LOB users to monitor the key performance indicators (KPIs) on the business processes in real-time so as to take immediate actions to resolve them

Oracle Business Process Management Suite (BPM) along with Oracle Business Activity Monitoring (BAM) - part of the Oracle Fusion Middleware stack can help you achieve all of the above and much more.

Lets delve into these for a moment and look at the components which will help us in achieving the goal.

Oracle Business Process Composer: This is purely a web-based modelling tool which will enable the business analysts of your organization to take control of the core business processes. The composer equips business analysts to work in groups, perform iterative model design/development, configure business rules, share the models with IT for further development, deploy them on server and much more. The important thing to note here is that a business process model can be developed from scratch without any IDE installation - all from within your web browser. I am sure the business guys are going to love this.

Oracle Business Process Workspace: Again a web-based workspace portal which allows all business users under various organization  roles to login and perform actions on their tasks.

Oracle Business Activity Monitoring: A web-based process monitoring dashboard which enables operational visibility into the business processes real-time. Again the BAM dashboards can be built by business users from within the browser based on the KPIs.

Is this not cool technology to take your business forward. Worrying less about IT and concentrating more on business development & growth has always been a dream. Now, take advantage of the latest & greatest technology from Oracle to move your business FORWARD!

Please feel free to write back to me if you need more information on this topic.