SOA 11g File Adapter Pre & Post Processing Technique

File Adapter is one of the most feature rich & powerful technology adapter that is available with Oracle SOA Suite. In this article, I will be explaining how to perform pre-processing of a file when it is being read by the File Adapter in SOA suite with a use-case.

In an enterprise scenario a file process operation is not as simple as we would like it to be. Often there are situations where files of different formats that might be required to be processed selectively. Also, the files could be packaged & encrypted in one common format. How do we handle this scenario?

In Oracle SOA Suite, the File Adapter based on JCA 1.5 architecture provides feasibility to perform pre/post processing of file that are being processed (read or write). This feature is known as the "Pipeline & Valve" concept of the File Adapter. As you can see from the diagram below, the file adapter polls a specific folder from the file system - The file stream then passes through the pipeline (which holds a sequence of custom defined valves). A pipeline can have 'n' number of valves which will be processed in the order specified. The valves are nothing but content processors where a Java call be made.

The scenario that we are going to consider in this post is a toned down version of a real-time business use-case but sufficient enough to drive home the pre/post processing capabilities of the file adapter.

Use-Case:

A bunch of applications publish files of various different formats to a common folder which should be processed. However, only files of certain formats must be picked up for processing and the rest left to be processed by a different system.

This is a classic case where the SOA suite file adapter's pre-processing technique could be put to use in determining the file format before processing a file.

Step 1: Building the custom valve to determine mime type of the file

Let us first start by creating the valve (Java class) which will determine the file format. All valves must implement either the Valve interface or StagedValve interface. Optionally, to make it easier, the custom valve class can also extend AbstractValve or AbstractStagedValve classes.

In this case, I have created a MimeValidationValve Java class which extends AbstractValve. Only files which have the following mime types are allowed.

text/plain
application/pdf
application/msword

This is configurable in the properties file. I have made use of Apache Tika open source toolkit which is very powerful in extracting metadata information of the files polled. Read more on Apache Tika here.

One important point to note here is that you will need to add the bpm-infra.jar library to the Java project in order to compile the valve class. This library is available under  <Middleware_Home>/Oracle_soa1/soa/modules/oracle.soa.fabric_11.1.1

Project Properties -> Libraries & Classpath -> Add Jar/Directory

You can download the valve Java project here

Step 2: Building the SOA application to process files

You can download the SOA project here. This project has a 'Read' file adapter to poll for files from the file system, a 'Write' file adapter to write files back to file system and a BPEL process connecting these adapters.

Step 3: Creating the Pipeline XML file

Recall that a pipeline can have one or more valves which will be processed in the order specified. The pipeline XML file will hold references to the various different valves that must be invoked by the file adapter before the translation phase or handing it over to the SOA runtime.

Let us now go back to the SOA project and create the pipeline XML file. Let us review the FilePipeLine.xml under the DGFileProcessor SOA application.


<?xml version="1.0" encoding="windows-1252" ?>
<pipeline xmlns="http://www.oracle.com/adapter/pipeline">
  <valves>
    <valve>mimetype.MimeValidationValve</valve>
  </valves>
</pipeline>

Note that any number of valves can be added to the pipeline by just adding the <valve> elements.

Step 4: Plumbing the Pipeline with the appropriate file adapter

Now that we have the valves created and the pipeline/valve configurations done, the final piece that remains to be done is to let the file adapter trigger pipeline processing. To do this, open the corresponding file adapter's jca file and add the following property.

<property name="PipelineFile" value="FilePipeLine.xml"/>

That completes the solution for the use-case.

WebCenter Spaces OBIEE Integration

This post is more of a self-reference material for the future. Thought it might help others looking for it as well.

Wouldn't it be a great idea for enterprises which already have WebCenter Spaces & OBIEE technologies to leverage the integration feature? Yes, think about showing BI analytics to your business users / stakeholders through the WebCenter spaces portal solution.

There are a set of configurations that needs to be done at both OBIEE and the WebLogic server level to enable single sign-on for seamless integration. A very good detailed step-by-step instruction on achieving the integration is documented here.

http://www.oracle.com/technetwork/middleware/webcenter/portal/learnmore/wc-biappsconfig-396418.pdf

SOA 11g MDS - Share Common Artifacts

As promised on my previous post, I am detailing the step-by-step instructions which allows for sharing of common artifacts (schema files/WSDLs) among SOA projects and more importantly helps in loose coupling of service bindings & remove service dependencies.

A typical use case is where the abstract WSDL definition of an external service can be stored in MDS and referenced from the SOA composite which will loosely couple the services and also guard against 'SOA Composite Invalidation' issues.

SOA 11g provides us with MDS repository (both file based & DB based) where common artifacts can be stored and referenced by all projects in the SOA infrastructure.

File based MDS is used during design/deploy time where as DB based MDS is used at run-time by the SOA infra. A configuration file named adf-config.xml that is part of every SOA project manages this partition mapping to be used by JDeveloper during design/deploy time.

All SOA projects that are deployed on the SOA infrastructure are stored in the MDS repository. If you create a connection from the JDeveloper 'Resource Palette' to the SOA MDS, there will be a 'deployed composites' folder which will contain all deployed artifacts.

Connection Information for the MDS connection:
username: XXX_MDS (usually DEV_MDS)
password: xxxxxxxxx
MDS Partition: soa-infra

The file based MDS repository is generally the 'seed' partition under a typical JDeveloper install;

<JDev_Home>\integration\seed


Under this partition there is a root folder (soa) available by default which contains all the configuration files, schemas & WSDLs required by various different SOA components during design time. These files are also pre-deployed on the database MDS repository.

We are now free to create custom folders under the 'seed' partition to hold artifacts that are common references in a SOA environment. Generally, we create an 'apps' folder under the 'seed' partition which in turn holds various folders to hold various artifact types;

 Once the artifacts that needs to be shared/referenced are placed under the appropriate namespace (in this case 'apps'), we can let the SOA project in JDeveloper to point to MDS for these artifacts from a common location.

adf-config.xml configuration:

As noted earlier, adf-config.xml is an important configuration file which holds the details regarding the MDS (both file based & DB based). By default, this file declares the 'seed' partition & the metadata store for 'soa' namespace.

This file can be found under <YourApplication>\.adf\META-INF.
Alternatively, in JDeveloper, expand the 'Application Resources' panel; drill through Descriptors -> ADF META-INF to find this file

Step 1: Just add the metadata store namespace for '/apps' in additions to default '/soa/shared' namespace - This is to let the JDeveloper reference the artifacts under the '/apps' during the design time from the file system metadata store.

<adf-mds-config xmlns="http://xmlns.oracle.com/adf/mds/config">
    <mds-config xmlns="http://xmlns.oracle.com/mds/config">
      <persistence-config>
        <metadata-namespaces>
          <namespace path="/soa/shared" metadata-store-usage="mstore-usage_1"/>
          <namespace path="/apps" metadata-store-usage="mstore-usage_1"/>
        </metadata-namespaces>
        <metadata-store-usages>
          <metadata-store-usage id="mstore-usage_1">
            <metadata-store class-name="oracle.mds.persistence.stores.file.FileMetadataStore">
              <property name="metadata-path"
                        value="${oracle.home}/integration"/>
              <property name="partition-name" value="seed"/>
            </metadata-store>
          </metadata-store-usage>
        </metadata-store-usages>
      </persistence-config>
    </mds-config>
  </adf-mds-config> 

Step 2: In the SOA composite, reference the common artifacts from MDS as shown below;

 <import namespace="http://xmlns.oracle.com/bpmn/bpmnProcess/CreditCardProcess"
          location="oramds:/apps/orderprocess/common/CreditServiceProcess.wsdl"
          importType="wsdl"/>

Now that we have successfully referenced the SOA project to refer the file based MDS it is barely enough only during design time. For deploy time & run time, these artifacts must be made available in the DB based MDS.

There are two methods available for us to deploy the artifacts in SOA DB based MDS;

Approach 1: Ant based script which allows to deploy the artifacts into SOA MDS. I am not going to get to the details of this as this has been blogged in detail by Edwin Biemond; Reference below;

http://biemond.blogspot.com/2009/11/soa-suite-11g-mds-deploy-and-removal.html

Approach 2: For a simple JDeveloper based SOA deployment follow the steps below;

Step 1: Create a new application (Generic Application)

Step 2: Name the application (eg., OrderProcessMDSApp) and click 'Next'

 Step 3: Provide a name to the project (eg., OrderProcessMDSProj) and click 'Finish'

 Step 4: Now, right click on the OrderProcessMDSProj and choose 'Project Properties'. In the 'Project Properties' dialog, choose 'Deployment' and create a new deployment profile by clicking on 'New...' button

 Step 5: In the 'Create Deployment Profile' dialog, leave the default 'Archive Type' to 'JAR File' and name the deployment profile (eg., soamdsarchive) and click 'OK'

 Step 6: This will open up the 'Edit JAR Deployment Profile Properties' dialog. Click on the 'Contributors' section under 'File Groups -> Project Output'.

Step 7: Click on the 'Add..' button and choose the folder where the 'content to be deployed' is present. Here the '/apps' namespace under the file based MDS can be chosen where the content is already placed & available.

Step 8: Now, click on the 'Filters' section and choose the artifacts under the '/apps' that you would like to deploy or upload to MDS database repository

Step 9: Click on OK to complete the deployment profile creation process. Now, we have created a project JAR deployment profile which contains the artifacts to be deployed on to the SOA MDS. Now, we will have to create a application deployment profile of 'soa bundle' type which will be deployed to the soa-infra.

Step 10: From the 'Application Menu', choose 'Deploy' -> 'New Deployment Profile..' option

Step 11: Under the 'New Gallery' go to 'Deployment Profiles' choose 'SOA Bundle' and click OK

Step 11: Under the 'Create Deployment Profile', provide a name to the application deployment profile (eg., soabundle1) and click OK.

Step 12: In the 'SOA Bundle Deployment Profile Properties' go to 'Dependencies' and choose the JAR archive that we created earlier eg., soamdsarchive and click OK

Finally, deploy the application to the soa server. This will ensure that the artifacts under the '/apps' metadata namespace is deployed on to the SOA MDS database repository.

You can verify the same from the SOA MDS connection created under the 'Resource Palette'. This should now reflect the 'apps' folder at the same level as 'soa' metadata store.

Now, if you try to deploy the SOA project which references MDS you may hit a deployment error complaining that the resource referenced is not found. This is because the JDeveloper consults adf-config.xml configuration file during design & deploy time (refer back to the initial architecture diagram). Since there is no information about the DB based MDS in the adf-config the deployer complains.

To fix this, go back to the adf-config.xml file and add the DB metadata store information so that the deployer can read & locate the referenced resources from MDS.

<adf-mds-config xmlns="http://xmlns.oracle.com/adf/mds/config">
    <mds-config xmlns="http://xmlns.oracle.com/mds/config">
      <persistence-config>
        <metadata-namespaces>
          <namespace path="/soa/shared" metadata-store-usage="mstore-usage_2"/>
          <namespace path="/apps" metadata-store-usage="mstore-usage_2"/>
        </metadata-namespaces>
        <metadata-store-usages>
          <metadata-store-usage id="mstore-usage_2">
            <metadata-store class-name="oracle.mds.persistence.stores.db.DBMetadataStore">
              <property value="DEV_MDS" name="jdbc-userid"/>
              <property value="xxxxxxx" name="jdbc-password"/>
              <property value="jdbc:oracle:thin:@hostname:1521:orcl"
                        name="jdbc-url"/>
              <property value="soa-infra" name="partition-name"/>
            </metadata-store>
          </metadata-store-usage>
        </metadata-store-usages>
      </persistence-config>
    </mds-config>
  </adf-mds-config>

That completes our configuration and eventually this elaborate post. Now, the deployment will be smooth as we have referenced the DB MDS in the adf-config.xml file.

Hope this post helped you understand & implement the SOA MDS related activities. Let me know your feedback through comments.