Implementing Scatter Gather Pattern in BizTalk using Self Correlation

This is one of the most widely used integration patterns and also the most asked interview question.

In this pattern a huge message (also called batch file) is debatched and sent to multiple or single recipient which process these messages and sends a response or acknowledgement back. Later all these responses are gathered and collated into a single message in Parent Orchestration. So basically this complete process can be divided into two sub processes- Splitter Pattern and Gatherer pattern.

You can read more about the Scatter and Gather Pattern here.

There can be many ways to implement this pattern in BizTalk but below implementation is asynchronous and loosely coupled.


a) For debatching the huge message (batch file) I have called/executed Receive Pipeline with XML Disassembler component from within the Parent Orchestration. This acts as Splitter.

b) Sent out the debatched messages using Start Orchestration shape, this will allow asynchronous processing of all the child messages(debatched messages)

c) For gathering the responses in parent orchestration I have used Self Correlation, acts as Gatherer

Detailed Steps:-

Parent Orchestration

1) Create a parent orchestration to receive input message (batch file) with “Transaction Type = Long Running”. This scope should be long running because it will contain an Atomic Scope for debatching.

2) Do some processing/transformation if required.

3) Create an Atomic Scope for debatching the big message. Here the scope should be Atomic because we will be calling/executing Pipeline within it and the pipeline is of non- serializable type.

4) Execute Receive Pipeline (having XML Disassembler component) in Expression Shape for debatching.

    varReceivePipelineOutput= Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteReceivePipeline(typeof(TestDebatchedProcessing.ReceivePipeline_DebatchInput),msgFormatted);

Here “varReceivePipelineOutput” is a variable of type- “Microsoft.XLANGs.Pipeline.ReceivePipelineOutputMessages” and should be defined within the Atomic Scope created for debatching. This class extends IEnumerator interface so implements GetCurrent and MoveNext methods.

Note: –

You will have to add reference to “Microsoft.XLANGs.Pipeline” dll present at below location – <Install dir>\Microsoft BizTalk Server 2013\Microsoft.XLANGs.Pipeline.dll

Read more about debatching XML messages in orchestration here.

5) Loop through all the debatched messages using “varReceivePipelineOutput.MoveNext()”. MoveNext function indicates whether the pipeline should move to the next message.

6) Construct Child Message using Message Assignment shape.

    msgChild=null; varReceivePipelineOutput.GetCurrent(msgChild); ->This will retrieves the current XLANG message.

7) Increase the count variable.

    intCount = intCount + 1;

8) Start Child Orchestration for further processing by passing Child Message and PortType (Created to receive response from Child Orchestration at step 10).

Start Orchestration Configuration Image

Start Orchestration Configuration Image

Note: –

  • You will have to create Child orchestration before this step. Developing Child orchestration will be explained below later in this page.
  • Create a Configured Port to Receive Port to receive messages from Child Orchestration

     Steps :-

a) Right Click on the Port Surface -> New configured Port- > Give PortName and PortType

b) Select Port Direction as “I’ll always be receiving message on this port” and Port Binding as “Direct” and type as “Self    Correlating” as shown below.

Self Correlated Port Configuration Image

Self Correlated Port Configuration Image

9) Create a Loop to receive all the messages sent out. Here we loop till the value of intcount becomes zero.

     intCount > 0

10) Receive Processed message from the Child Orchestration, this receive shape is connected to the Self Correlated Receive port created in step 8-b-2

11) Reduce count variable

      intCount= intCount -1;

Parent Orchestration Full Image

 Full Parent Orchestration Image

Child Orchestration

1) Create a Child Orchestration which receives debatched messages from Parent Orchestration and processes it further.

As first step create Orchestration Parameters

a) Port_SelfCorr -> Port Parameter to receive PortType

b) msgChild

  1. Right Click Orchestration Parameters -> New Configured Port Parameter -> Give a desired Name
  2. “Use Existing Port Type” and select Self Correlated Port created in Parent Orch as shown below
  3. Click Next and give Port Direction as “I’ll always be sending message on this port”

Orchestration Parameters -Configured Port Parameter Image

Configured Port Parameter In Child Orchestration Image

3. Do some processing (I have done some transformation).

4. Send the Processed Message back to Parent Orchestration. This send port is connected to the port created while creating Port Parameter

Full Child Orchestration Image

 Full Child Orchestration Image

You can find the complete solution here.

Contact Me:- 

@Gmail@Facebook , @Twitter, @LinkedIn @MSDNTechnet, @My Personal Blog 


10 thoughts on “Implementing Scatter Gather Pattern in BizTalk using Self Correlation”

  1. Hi Prashanth,
    Good explanation, In Receive Location which Pipeline I need to use?
    Please note I am debatching flat file, Not xml.
    I have 10 records(10 lines) , I am debatching it and I need to aggregate while sending, Please suggest.


    1. Hi Eliyas,
      BizTalk only understands XML so if you want to receive a flat file you will have to use FlatFileDisassembler to convert in into XML.
      However, one solution could be –
      Receive incoming message as XMLDocument(untyped msg) in orchestration and later use FlatFileDisassembler for debatching. The flow will remain same. In the end you will have to FlatFileAssembler.
      Let me know if you face any issue. You can drop an email or contact me on Facebook.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s