Implementing Scatter Gather Pattern in BizTalk using Self Correlation

This is one of the most widely used integration patterns and also the most asked interview question.

In this pattern a huge message (also called batch file) is debatched and sent to multiple or single recipient which process these messages and sends a response or acknowledgement back. Later all these responses are gathered and collated into a single message in Parent Orchestration. So basically this complete process can be divided into two sub processes- Splitter Pattern and Gatherer pattern.

You can read more about the Scatter and Gather Pattern here.

There can be many ways to implement this pattern in BizTalk but below implementation is asynchronous and loosely coupled.

Basics:-

a) For debatching the huge message (batch file) I have called/executed Receive Pipeline with XML Disassembler component from within the Parent Orchestration. This acts as Splitter.

b) Sent out the debatched messages using Start Orchestration shape, this will allow asynchronous processing of all the child messages(debatched messages)

c) For gathering the responses in parent orchestration I have used Self Correlation, acts as Gatherer

Detailed Steps:-

Parent Orchestration

1) Create a parent orchestration to receive input message (batch file) with “Transaction Type = Long Running”. This scope should be long running because it will contain an Atomic Scope for debatching.

2) Do some processing/transformation if required.

3) Create an Atomic Scope for debatching the big message. Here the scope should be Atomic because we will be calling/executing Pipeline within it and the pipeline is of non- serializable type.

4) Execute Receive Pipeline (having XML Disassembler component) in Expression Shape for debatching.

    varReceivePipelineOutput= Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteReceivePipeline(typeof(TestDebatchedProcessing.ReceivePipeline_DebatchInput),msgFormatted);

Here “varReceivePipelineOutput” is a variable of type- “Microsoft.XLANGs.Pipeline.ReceivePipelineOutputMessages” and should be defined within the Atomic Scope created for debatching. This class extends IEnumerator interface so implements GetCurrent and MoveNext methods.

Note: –

You will have to add reference to “Microsoft.XLANGs.Pipeline” dll present at below location – <Install dir>\Microsoft BizTalk Server 2013\Microsoft.XLANGs.Pipeline.dll

Read more about debatching XML messages in orchestration here.

5) Loop through all the debatched messages using “varReceivePipelineOutput.MoveNext()”. MoveNext function indicates whether the pipeline should move to the next message.

6) Construct Child Message using Message Assignment shape.

    msgChild=null; varReceivePipelineOutput.GetCurrent(msgChild); ->This will retrieves the current XLANG message.

7) Increase the count variable.

    intCount = intCount + 1;

8) Start Child Orchestration for further processing by passing Child Message and PortType (Created to receive response from Child Orchestration at step 10).

Start Orchestration Configuration Image

Note: –

• You will have to create Child orchestration before this step. Developing Child orchestration will be explained below later in this page.
• Create a Configured Port to Receive Port to receive messages from Child Orchestration

     Steps :-

a) Right Click on the Port Surface -> New configured Port- > Give PortName and PortType

b) Select Port Direction as “I’ll always be receiving message on this port” and Port Binding as “Direct” and type as “Self    Correlating” as shown below.

Self Correlated Port Configuration Image

9) Create a Loop to receive all the messages sent out. Here we loop till the value of intcount becomes zero.

     intCount > 0

10) Receive Processed message from the Child Orchestration, this receive shape is connected to the Self Correlated Receive port created in step 8-b-2

11) Reduce count variable

      intCount= intCount -1;

 Full Parent Orchestration Image

Child Orchestration

1) Create a Child Orchestration which receives debatched messages from Parent Orchestration and processes it further.

As first step create Orchestration Parameters

a) Port_SelfCorr -> Port Parameter to receive PortType

b) msgChild

1. Right Click Orchestration Parameters -> New Configured Port Parameter -> Give a desired Name
2. “Use Existing Port Type” and select Self Correlated Port created in Parent Orch as shown below
3. Click Next and give Port Direction as “I’ll always be sending message on this port”

Configured Port Parameter In Child Orchestration Image

3. Do some processing (I have done some transformation).

4. Send the Processed Message back to Parent Orchestration. This send port is connected to the port created while creating Port Parameter

 Full Child Orchestration Image

You can find the complete solution here.

Contact Me:- 

@Gmail, @Facebook , @Twitter, @LinkedIn , @MSDNTechnet, @My Personal Blog