This is one of the most widely used integration patterns and also the most asked interview question.
In this pattern a huge message (also called batch file) is debatched and sent to multiple or single recipient which process these messages and sends a response or acknowledgement back. Later all these responses are gathered and collated into a single message in Parent Orchestration. So basically this complete process can be divided into two sub processes- Splitter Pattern and Gatherer pattern.
You can read more about the Scatter and Gather Pattern here.
There can be many ways to implement this pattern in BizTalk but below implementation is asynchronous and loosely coupled.
a) For debatching the huge message (batch file) I have called/executed Receive Pipeline with XML Disassembler component from within the Parent Orchestration. This acts as Splitter.
b) Sent out the debatched messages using Start Orchestration shape, this will allow asynchronous processing of all the child messages(debatched messages)
c) For gathering the responses in parent orchestration I have used Self Correlation, acts as Gatherer
1) Create a parent orchestration to receive input message (batch file) with “Transaction Type = Long Running”. This scope should be long running because it will contain an Atomic Scope for debatching.
2) Do some processing/transformation if required.
3) Create an Atomic Scope for debatching the big message. Here the scope should be Atomic because we will be calling/executing Pipeline within it and the pipeline is of non- serializable type.
4) Execute Receive Pipeline (having XML Disassembler component) in Expression Shape for debatching.
Here “varReceivePipelineOutput” is a variable of type- “Microsoft.XLANGs.Pipeline.ReceivePipelineOutputMessages” and should be defined within the Atomic Scope created for debatching. This class extends IEnumerator interface so implements GetCurrent and MoveNext methods.
You will have to add reference to “Microsoft.XLANGs.Pipeline” dll present at below location – <Install dir>\Microsoft BizTalk Server 2013\Microsoft.XLANGs.Pipeline.dll
Read more about debatching XML messages in orchestration here.
5) Loop through all the debatched messages using “varReceivePipelineOutput.MoveNext()”. MoveNext function indicates whether the pipeline should move to the next message.
6) Construct Child Message using Message Assignment shape.
msgChild=null; varReceivePipelineOutput.GetCurrent(msgChild); ->This will retrieves the current XLANG message.
7) Increase the count variable.
intCount = intCount + 1;
8) Start Child Orchestration for further processing by passing Child Message and PortType (Created to receive response from Child Orchestration at step 10).
Start Orchestration Configuration Image
- You will have to create Child orchestration before this step. Developing Child orchestration will be explained below later in this page.
- Create a Configured Port to Receive Port to receive messages from Child Orchestration
a) Right Click on the Port Surface -> New configured Port- > Give PortName and PortType
b) Select Port Direction as “I’ll always be receiving message on this port” and Port Binding as “Direct” and type as “Self Correlating” as shown below.
Self Correlated Port Configuration Image
9) Create a Loop to receive all the messages sent out. Here we loop till the value of intcount becomes zero.
intCount > 0
10) Receive Processed message from the Child Orchestration, this receive shape is connected to the Self Correlated Receive port created in step 8-b-2
11) Reduce count variable
intCount= intCount -1;
Full Parent Orchestration Image
1) Create a Child Orchestration which receives debatched messages from Parent Orchestration and processes it further.
As first step create Orchestration Parameters
a) Port_SelfCorr -> Port Parameter to receive PortType
- Right Click Orchestration Parameters -> New Configured Port Parameter -> Give a desired Name
- “Use Existing Port Type” and select Self Correlated Port created in Parent Orch as shown below
- Click Next and give Port Direction as “I’ll always be sending message on this port”
Configured Port Parameter In Child Orchestration Image
3. Do some processing (I have done some transformation).
4. Send the Processed Message back to Parent Orchestration. This send port is connected to the port created while creating Port Parameter
Full Child Orchestration Image
You can find the complete solution here.
@Gmail, @Facebook , @Twitter, @LinkedIn , @MSDNTechnet, @My Personal Blog
12 thoughts on “Implementing Scatter Gather Pattern in BizTalk using Self Correlation”
Nicely done article
Thanks Pushpendra 🙂
Really Nice way to Describe,
I like your answer on Biztalk Forums,
LikeLiked by 2 people
Thanks a lot Saurin, your comments/suggestions really matter a lot to me 🙂
Very well explained Prashant. Appreciated!
LikeLiked by 2 people
Really a nice and complete explanation,,
Thanks a lot Venu 🙂
Good explanation, In Receive Location which Pipeline I need to use?
Please note I am debatching flat file, Not xml.
I have 10 records(10 lines) , I am debatching it and I need to aggregate while sending, Please suggest.
BizTalk only understands XML so if you want to receive a flat file you will have to use FlatFileDisassembler to convert in into XML.
However, one solution could be –
Receive incoming message as XMLDocument(untyped msg) in orchestration and later use FlatFileDisassembler for debatching. The flow will remain same. In the end you will have to FlatFileAssembler.
Let me know if you face any issue. You can drop an email or contact me on Facebook.
We can receive flat files also directly into orchestrations, just set the receive location pipeline as passthrough
True, you can receive flat files in orch by using passthrough pipeline and untyped messages. But for any processing you will have to convert it to xml right?