SOA 11g File Adapter Pre & Post Processing Technique

File Adapter is one of the most feature rich & powerful technology adapter that is available with Oracle SOA Suite. In this article, I will be explaining how to perform pre-processing of a file when it is being read by the File Adapter in SOA suite with a use-case.

In an enterprise scenario a file process operation is not as simple as we would like it to be. Often there are situations where files of different formats that might be required to be processed selectively. Also, the files could be packaged & encrypted in one common format. How do we handle this scenario?

In Oracle SOA Suite, the File Adapter based on JCA 1.5 architecture provides feasibility to perform pre/post processing of file that are being processed (read or write). This feature is known as the "Pipeline & Valve" concept of the File Adapter. As you can see from the diagram below, the file adapter polls a specific folder from the file system - The file stream then passes through the pipeline (which holds a sequence of custom defined valves). A pipeline can have 'n' number of valves which will be processed in the order specified. The valves are nothing but content processors where a Java call be made.

The scenario that we are going to consider in this post is a toned down version of a real-time business use-case but sufficient enough to drive home the pre/post processing capabilities of the file adapter.

Use-Case:

A bunch of applications publish files of various different formats to a common folder which should be processed. However, only files of certain formats must be picked up for processing and the rest left to be processed by a different system.

This is a classic case where the SOA suite file adapter's pre-processing technique could be put to use in determining the file format before processing a file.

Step 1: Building the custom valve to determine mime type of the file

Let us first start by creating the valve (Java class) which will determine the file format. All valves must implement either the Valve interface or StagedValve interface. Optionally, to make it easier, the custom valve class can also extend AbstractValve or AbstractStagedValve classes.

In this case, I have created a MimeValidationValve Java class which extends AbstractValve. Only files which have the following mime types are allowed.

text/plain
application/pdf
application/msword

This is configurable in the properties file. I have made use of Apache Tika open source toolkit which is very powerful in extracting metadata information of the files polled. Read more on Apache Tika here.

One important point to note here is that you will need to add the bpm-infra.jar library to the Java project in order to compile the valve class. This library is available under  <Middleware_Home>/Oracle_soa1/soa/modules/oracle.soa.fabric_11.1.1

Project Properties -> Libraries & Classpath -> Add Jar/Directory

You can download the valve Java project here

Step 2: Building the SOA application to process files

You can download the SOA project here. This project has a 'Read' file adapter to poll for files from the file system, a 'Write' file adapter to write files back to file system and a BPEL process connecting these adapters.

Step 3: Creating the Pipeline XML file

Recall that a pipeline can have one or more valves which will be processed in the order specified. The pipeline XML file will hold references to the various different valves that must be invoked by the file adapter before the translation phase or handing it over to the SOA runtime.

Let us now go back to the SOA project and create the pipeline XML file. Let us review the FilePipeLine.xml under the DGFileProcessor SOA application.


<?xml version="1.0" encoding="windows-1252" ?>
<pipeline xmlns="http://www.oracle.com/adapter/pipeline">
  <valves>
    <valve>mimetype.MimeValidationValve</valve>
  </valves>
</pipeline>

Note that any number of valves can be added to the pipeline by just adding the <valve> elements.

Step 4: Plumbing the Pipeline with the appropriate file adapter

Now that we have the valves created and the pipeline/valve configurations done, the final piece that remains to be done is to let the file adapter trigger pipeline processing. To do this, open the corresponding file adapter's jca file and add the following property.

<property name="PipelineFile" value="FilePipeLine.xml"/>

That completes the solution for the use-case.

Apache Tika - File Content Analyser

If you are fiddling around with files and documents day-in day-out and look to detect their content type for whatsoever reason - Apache Tika can save your day.

A valid large enterprise use-case I can think of.....

Require a system/solution to read through few hundreds of thousands of documents of a few hundred distinct MIME types every day - validate - process - and push to subscribed interfaces via various protocols.

From the architecture/design perspective, I would like my solution to have extensibility rolled in - This would ensure that I can have a single file reader which can handle/read multiple file formats so that I don't have to spend time & effort on IT for incorporating a new file format in near future. Also prudent would be to have the file formats validated before consuming/reading them by way of pre-processors which would be responsible for detecting the content-types before reading. This would really save a lot of time and gain performance.

Consider a scenario where a huge video file is morphed into a PDF by just altering the extension. It would not be very wise for the IT solution to just read the file extension and presume the validity of file format.

Apache Tika does exactly that - It has the intelligence to detect the exact MIME type of the file just by reading the meta data and remember it is not very invasive (It doesn't read the entire content of a 5 GB file to understand the actual content type). In its own words this open source project can be described as - "detects and extracts metadata and structured text content from various documents using existing parser libraries" Downside - It is a bit heavy (~25 MB library). But the benefit is that it can read, detect and parse a wide variety of file contents and provide with a specific MIME type which apparently is not the case with many of the java APIs out there.

More information on APIs and Documentation here


I happen to use this open source library in one of my implementation and found really helpful. Thought it might help someone out there looking for such requirements :)