File Adapter is one of the most feature rich & powerful technology adapter that is available with Oracle SOA Suite. In this article, I will be explaining how to perform pre-processing of a file when it is being read by the File Adapter in SOA suite with a use-case.
In an enterprise scenario a file process operation is not as simple as we would like it to be. Often there are situations where files of different formats that might be required to be processed selectively. Also, the files could be packaged & encrypted in one common format. How do we handle this scenario?
In Oracle SOA Suite, the File Adapter based on JCA 1.5 architecture provides feasibility to perform pre/post processing of file that are being processed (read or write). This feature is known as the "Pipeline & Valve" concept of the File Adapter. As you can see from the diagram below, the file adapter polls a specific folder from the file system - The file stream then passes through the pipeline (which holds a sequence of custom defined valves). A pipeline can have 'n' number of valves which will be processed in the order specified. The valves are nothing but content processors where a Java call be made.
The scenario that we are going to consider in this post is a toned down version of a real-time business use-case but sufficient enough to drive home the pre/post processing capabilities of the file adapter.
Use-Case:
A bunch of applications publish files of various different formats to a common folder which should be processed. However, only files of certain formats must be picked up for processing and the rest left to be processed by a different system.
This is a classic case where the SOA suite file adapter's pre-processing technique could be put to use in determining the file format before processing a file.
Step 1: Building the custom valve to determine mime type of the file
Let us first start by creating the valve (Java class) which will determine the file format. All valves must implement either the Valve interface or StagedValve interface. Optionally, to make it easier, the custom valve class can also extend AbstractValve or AbstractStagedValve classes.
In this case, I have created a MimeValidationValve Java class which extends AbstractValve. Only files which have the following mime types are allowed.
text/plain
application/pdf
application/msword
This is configurable in the properties file. I have made use of Apache Tika open source toolkit which is very powerful in extracting metadata information of the files polled. Read more on Apache Tika here.
You can download the valve Java project here
Step 2: Building the SOA application to process files
You can download the SOA project here. This project has a 'Read' file adapter to poll for files from the file system, a 'Write' file adapter to write files back to file system and a BPEL process connecting these adapters.
Step 3: Creating the Pipeline XML file
Recall that a pipeline can have one or more valves which will be processed in the order specified. The pipeline XML file will hold references to the various different valves that must be invoked by the file adapter before the translation phase or handing it over to the SOA runtime.
Let us now go back to the SOA project and create the pipeline XML file. Let us review the FilePipeLine.xml under the DGFileProcessor SOA application.
<?xml version="1.0" encoding="windows-1252" ?>
<pipeline xmlns="http://www.oracle.com/adapter/pipeline">
<valves>
<valve>mimetype.MimeValidationValve</valve>
</valves>
</pipeline>
Note that any number of valves can be added to the pipeline by just adding the <valve> elements.
Step 4: Plumbing the Pipeline with the appropriate file adapter
Now that we have the valves created and the pipeline/valve configurations done, the final piece that remains to be done is to let the file adapter trigger pipeline processing. To do this, open the corresponding file adapter's jca file and add the following property.
<property name="PipelineFile" value="FilePipeLine.xml"/>
That completes the solution for the use-case.
In an enterprise scenario a file process operation is not as simple as we would like it to be. Often there are situations where files of different formats that might be required to be processed selectively. Also, the files could be packaged & encrypted in one common format. How do we handle this scenario?
In Oracle SOA Suite, the File Adapter based on JCA 1.5 architecture provides feasibility to perform pre/post processing of file that are being processed (read or write). This feature is known as the "Pipeline & Valve" concept of the File Adapter. As you can see from the diagram below, the file adapter polls a specific folder from the file system - The file stream then passes through the pipeline (which holds a sequence of custom defined valves). A pipeline can have 'n' number of valves which will be processed in the order specified. The valves are nothing but content processors where a Java call be made.
The scenario that we are going to consider in this post is a toned down version of a real-time business use-case but sufficient enough to drive home the pre/post processing capabilities of the file adapter.
Use-Case:
A bunch of applications publish files of various different formats to a common folder which should be processed. However, only files of certain formats must be picked up for processing and the rest left to be processed by a different system.
This is a classic case where the SOA suite file adapter's pre-processing technique could be put to use in determining the file format before processing a file.
Step 1: Building the custom valve to determine mime type of the file
Let us first start by creating the valve (Java class) which will determine the file format. All valves must implement either the Valve interface or StagedValve interface. Optionally, to make it easier, the custom valve class can also extend AbstractValve or AbstractStagedValve classes.
In this case, I have created a MimeValidationValve Java class which extends AbstractValve. Only files which have the following mime types are allowed.
text/plain
application/pdf
application/msword
This is configurable in the properties file. I have made use of Apache Tika open source toolkit which is very powerful in extracting metadata information of the files polled. Read more on Apache Tika here.
One important point to note here is that you will need to add the bpm-infra.jar library to the Java project in order to compile the valve class. This library is available under <Middleware_Home>/Oracle_soa1/soa/modules/oracle.soa.fabric_11.1.1
Project Properties -> Libraries & Classpath -> Add Jar/Directory
You can download the valve Java project here
Step 2: Building the SOA application to process files
You can download the SOA project here. This project has a 'Read' file adapter to poll for files from the file system, a 'Write' file adapter to write files back to file system and a BPEL process connecting these adapters.
Step 3: Creating the Pipeline XML file
Recall that a pipeline can have one or more valves which will be processed in the order specified. The pipeline XML file will hold references to the various different valves that must be invoked by the file adapter before the translation phase or handing it over to the SOA runtime.
Let us now go back to the SOA project and create the pipeline XML file. Let us review the FilePipeLine.xml under the DGFileProcessor SOA application.
<?xml version="1.0" encoding="windows-1252" ?>
<pipeline xmlns="http://www.oracle.com/adapter/pipeline">
<valves>
<valve>mimetype.MimeValidationValve</valve>
</valves>
</pipeline>
Note that any number of valves can be added to the pipeline by just adding the <valve> elements.
Step 4: Plumbing the Pipeline with the appropriate file adapter
Now that we have the valves created and the pipeline/valve configurations done, the final piece that remains to be done is to let the file adapter trigger pipeline processing. To do this, open the corresponding file adapter's jca file and add the following property.
<property name="PipelineFile" value="FilePipeLine.xml"/>
That completes the solution for the use-case.
Hello, i m new to soa 11g.n i m currently using jdev11.1.1.3.0..i have problem with file adapter..in this i create two files F1,F2.i want to read the file F1 and write it into F2.but i m not giving correct path details of F2.then at that time the file was read but not write..its giving error..so my question is where can i get that disappeared file?
ReplyDeleteDo you mean recovering the already 'processed' file from the SOA instance? - I don't think this is possible. But from a design stand point, it is always prudent to archive the processed files in a separate location. If space is a constraint, a scheduler process can be created to delete archive files periodically (may be delete file which are > 7 days old).
DeleteHi Sathya ...
ReplyDeletei want to try the pre processing of files in OSB..is it possible ?
Yes you can. Just create your file adapter in a SOA composite - implement your pre/post processing techniques and consume them within OSB. You have facility to import any JCA adapter resource within OSB.
Deletewill oracle give support if any issue happens since we are modifying the file adapter .jar file
ReplyDeleteI don't think they would. But honestly I don't have a 100% correct answer. You should write to Oracle Support to get the right answer. Just out of curiosity, what is your use-case where you are trying to modify the adapter jar?
DeleteHi Sathya,
ReplyDeletewe are receiving file with some secured information like username and password. we want these fields to be masked while storing in archive and in BPEL we should receive the correct values(Unmasked values) .Will this be possible through Valves.
Hi Niranjan,
DeleteI don't think this requirement can be satisfied by the adapter out-of-box. One workaround (not-so-straight-forward) I can think of is to encrypt the archived file in the BPEL process once it is consumed/worked-upon. I will get back to you via this blog if I can achieve it in a more sophisticated fashion.
Hi Sathyam
ReplyDeleteI have not yet gone through the sample code uploaded with this post, but, was just wondering if it contains how to generate a PDF file.
I need to create a pdf file from an XML that has base64binary element. I decoded the base64binary into string and then tried generating PDF, but, to no avail.
I would appreciate if you can guide me about how can I generate a pdf using File adapter.
Thanks in advance
Hi Abhinav,
DeleteYour requirement seems very simple. You just have to create a File Adapter in your SOA composite with "Opaque Schema" (Choose "Native Format Translation Not Required" option during adapter creation). Then map the base64 encoded XML element from your input to the "opaqueElement" of FileAdapter. Whatever file attachment format is sent as base64 binary encoded can be decoded with this approach.
You may also want to take a look at the following MTOM sample here: https://java.net/downloads/oraclesoasuite11g/Binary%20Content%20and%20Large%20Documents/MTOM_SOASuite.zip
Hi Sathya
DeleteYes, I figured that out eventually. I was informed that the file is encoded and zipped which need to be decoded and unzipped before writing the same out.
Well, I found out that wasn't the case and got it done the plain simple way(as described by you).
But, anyways, thank you very much for your reply.
Regards
Abhinav
Great! Sounds good. Thanks for your update.
DeleteHi satya,
ReplyDeleteCan you please tell me that how to linked that java valves with SOA Project.
As described in the blog, you must make a jar archive and attach it as library/dependency to your SOA project.
DeleteHi Sathya,
ReplyDeleteI just want to encrypt the file in soa 11g using FIle adapter before processing to Target Path. We receive data like xml and that data make into file with encryption .txt or any format.
How it could be over come.
I am not sure I understand your question correctly Harish.
DeleteHi Satya,
ReplyDeleteMy soa composite is exposed as a webservice which has a payload ( has base64binary format - using to send pdf file as attachment ) and is invoked to transfer pdf as attachment. I am able to transfer successfully to destination server. But I receive file as corrupted error when I am trying to open the pdf at destination server . Is there something that I can do from Soa to get rid of this error and transfer , open the pdf file successfully.
Thanks in advance
Hi Satya,
ReplyDeleteGreat blog on pipelines and valves. When extracting a zip file which has numerous files, how do I pass the individual file names to the bpel. I set the filenames in the valve java code using setMessageKey(), but not sure how to pick it up in the bpel. Can you please help me figure this.