Writing effective adapters for Biztalk Server

Introduction
Many people, both within Microsoft and in third-party companies, have written successful adapters for Microsoft® BizTalk® Server. This task can be difficult, and this paper is intended to present some of the "tricks of the trade" that these developers have learned, in the hope that these will help others avoid common problems.

This document is structured as a set of issues that programmers face when writing adapters, and provides guidelines for resolving these issues. These guidelines can be read in any order, although you may find that reading the whole document first will make the individual guidelines easier to understand.

Effective Writing for Adapters
Much of the complexity in writing adapters for BizTalk Server is centered on the problems of batched operations and that is what we will look at first.
Operations, Messages, Batching, and Transactions
Adapters commonly support sending messages to and receiving messages from BizTalk Server:
· Send side operations occur when information is being sent by BizTalk Server to the protocols supported by the adapter.
· Receive Side operations occur when the adapter receives information from the outside world and passes it into BizTalk Server.
Adapters in BizTalk Server perform a number of fundamental operations when receiving and sending messages. These operations are expressed as methods on an IBTTransportBatch batch object. With one exception, the methods take a message as a parameter. These operations are listed as follows:

Receive side operations:
· One-Way Submit: void SubmitMessage(IBaseMessage msg). The adapter can submit a message to BizTalk Server.
· Suspend: void MoveToSuspendQ(IBaseMessage msg). The adapter can suspend or move a message to the suspend queue when certain failures occur after submission.
· Submit Request: void SubmitRequestMessage(IBaseMessage requestMsg, string correlationToken, [MarshalAs(UnmanagedType.Bool)]bool firstResponseOnly, DateTime expirationTime, IBTTransmitter responseCallback). This operation allows a receive adapter to submit a message to BizTalk Server in a request-response pair.

Send side operations:
· Resubmit: void Resubmit(IBaseMessage msg, DateTime timeStamp). The adapter can resubmit a message after transmission failures occur. The resubmit operation requests that BizTalk Server schedules the message for transmission at a later time.
· Move to Next Transport: void MoveToNextTransport(IBaseMessage msg). The adapter can move the message to the next transport after all retries have been exhausted. This operation requests that a message is transmitted using the backup transport for that message.
· Suspend: void MoveToSuspendQ(IBaseMessage msg). The adapter can move the message to the suspend queue when no backup transport is present
· Delete: void DeleteMessage(IBaseMessage msg) The adapter can delete the message after a successful transmission. Deleting a message in BizTalk Server is the way that a send adapter tells BizTalk Server that the adapter is done with the message. Generally the SubmitResponse operation is done in the same batch as the Delete operation.
· Submit Response: void SubmitResponseMessage(IBaseMessage solicitMsgSent, IBaseMessage responseMsgToSubmit). The adapter can submit an immediate response back to BizTalk Server. This operation includes the original message in the call along with the response so that BizTalk Server can correlate them. The response message does not need to be the true external system response message, which might arrive later.
Messages
A BizTalk Server message is a packet of data that arrives in BizTalk Server or is sent from BizTalk Server. Everything going into and coming out of BizTalk Server is a message. Examples include HTTP posts, FTP files, e-mail, MQSeries messages, and SQL result sets. These are all made into messages in the BizTalk Server world.
A message is made up of a header section and a body section:
· Headers are simple name-value pairs. The names are all drawn from various property schemas. The system provides some core schemas and BizTalk Server applications can add their own. Headers are copied into memory when a message is processed.
· The body of a message has one or more parts. Each part is a stream of bytes. The body of the message is never copied into memory at one time.
The basic data model for a message is inspired by the Internet e-mail protocol MIME.
Batches
When your adapter has a series of operations to perform at one time, you should batch these operations to optimize performance.
Programmatically, message batches are collections of operations with associated messages.
Transactions
Batches are associated with transactions. For every batch, there is a single transaction. The adapter may create the transaction and associate it with the batch (in which case we call it a transactional batch) or it can leave it to BizTalk Server to manage internally by providing a NULL transaction object with the batch (in which case we call it a normal batch). BizTalk Server itself creates an internal batch object in this case, but this batch object will not be accessible to the adapter.
Note The idea of a "transactional Batch" can be somewhat confusing, because all batches, inside of BizTalk Server, are considered transactions.
Transactional Batches
If the adapter wants to involve the external system in the transaction in any way (for example, SQL Server in the case of the SQL Adaptor) then it must specifically create a transaction object. A transactional batch is a batch for which the adapter provides the transaction object that BizTalk Server will use.
For the purposes of this paper, a transaction is a true Microsoft Distributed Transaction Coordinator (MSDTC) COM+ transaction.
Note For more information on MSDTC, see the documentation for the Distributed Transaction Coordinator at: http://go.microsoft.com/fwlink/?LinkId=44297
Transactional adapter
A transactional adapter is an adaptor that makes use of transactional batches. A "Non-transactional adapter" uses normal batches.
Using Batches
Use batching to take advantage of BizTalk Server optimizations

BizTalk Server uses batching to:
· Amortize the cost of the transaction across many messages.
· Increase speed by reducing the internal number of database round trips.
· Make more efficient use of the BizTalk Server thread pool by using the BizTalk Server Asynchronous API.
Batches are atomic
A batch is a unit of work that is atomic; that is, it either succeeds or fails. If one operation in a batch succeeds but another operation fails, all the operations that make up a batch are invalidated and must be repeated.
This means that an adapter must do two things in response to a failed batch:
· Determine what to do with the failed message
· Re-submit the successful messages
An adapter may provide the transaction object for a batch
There are two kinds of batches defined previously in this paper: transactional batches and normal batches. These types are used in the following ways:
Normal batches
Not all adapters need to manage transactions externally. The FILE adapter is an example of an adapter that does not require access to the transaction, because the external file operations it manages are not transactional. In this case, the adapter does not provide a transaction object to BizTalk Server. If the adapter does not provide a transaction object to BizTalk Server as part of a batch of messages, BizTalk Server creates an internal transaction for it automatically. This can be thought of as a "normal" batch.

Transactional batches
Other adapters (such as the SQL adapter supplied with BizTalk Server) must coordinate an external SQL Server transaction with an internal BizTalk Server transaction. To do this, the adapter needs access to the BizTalk Server transaction object. A transaction object is created and and associated with the batch before the batch is submitted to BizTalk Server. This is a transactional batch. By supplying your own transaction object, you can achieve the "guaranteed, once and once only", delivery of data into and out of BizTalk Server.
Transactional database adapters like the SQL adapter that shipped with BizTalk Server 2004 have the potential for deadlocks in the external database because of the single transaction used for the batch. This is why the SQL adapter hard-coded its batch size to one.
Note An adapter provides BizTalk Server with a transaction to ensure that either BizTalk Server or the external system has a record of the data. Because only one or the other has this record, once and once only delivery is assured.

Handling Batch Errors and Confirming Batch Success
Error processing should be associated with an individual operation and not with the batch that contains the operation
Performance issues aside, the batching of messages should be invisible to the user of the adapter. This means that the failure of one operation in a batch should not affect any other operation in any way. However, batches are atomic, so the failure of one message results in an error for the batch, and no operations are processed.
Your write the code that is responsible for handling the error, re-submitting the successful messages, and suspending the unsuccessful ones. Fortunately, BizTalk Server provides a detailed error structure that enables your adapter to determine the specific operation that failed, and to construct further batches in which the successful operations are resubmitted and the unsuccessful ones suspended.
The final durable state of the operation should not be affected by the batching that happened to go on within the adapter.

Use SetErrorInfo to report failure to BizTalk Server
If you are suspending a message, you must provide failure information to BizTalk Server from the previous message context. BizTalk Server provides error reporting capabilities using the SetErrorInfo method on both the IBaseMessage interface as well as on ITransportProxy:
· When a failure occurs while processing a message, set the exception using SetErrorInfo(exception) on the message to be suspended. This way, the engine preserves this error with the message for later diagnosis, and logs it to the event log to alert the administrator.
· If you encounter an error during initialization or internal bookkeeping (not during message processing) you should call SetErrorInfo(exception) on the ITransportProxy pointer passed to you during initialization. If your adapter is based on a BaseAdapter implementation, you should always have access to this pointer. Otherwise, you should be certain that you cache it.
Reporting an error with either of these methods results in the error getting to the event log. It is important that you associate the error with the related message if you are able to do so.
Transactional adapters must call DTCommitConfirm()
When you write an adapter that creates a transaction object and hands it to BizTalk Server, you are accepting responsibility for writing code that does several things:
· Decides the final outcome of the batch operation: to either commit or end the transaction.
· Informs BizTalk Server of the final outcome by calling the method, DTCConfirmCommit()
Using DTCConfirmCommit()
The adapter must inform BizTalk Server about the final outcome of the transaction in order to maintain its internal tracking data. The adapter informs BizTalk Server of the outcome by calling DTCConfirmCommit. If the adapter does not do this, a significant memory leak occurs.

A possible race condition
The two tasks listed above (resolve errors and decide the final outcome) seem simple enough, but in fact they rely on information from different threads:
· The adapter processes errors based on information passed by BizTalk Server to the BatchComplete callback in the adapter. This callback is on the adapter's thread.
· DTCConfirmCommit is a method on the IBTDTCCommitConfirm object. An instance of the IBTDTCCommitConfirm object is returned by the batch IBTTransportBatch::Done() call. This instance is on the same thread as the IBTTransportBatch::Done() call, which is different from the adapter's thread.
· For every call that adapter makes to IBTTransportBatch::Done() there is a corresponding callback BatchComplete() that is called by the messaging engine in a separate thread to report the result of the batch submission. In BatchComplete() the adapter needs to commit or roll back the transaction based on whether the batch passed or failed. In either case, the adapter should then call DTCConfirmCommit() to report the status of the transaction.
A possible race exists because the adapter’s implementation of BatchComplete can assume that the IBTDTCCommitConfirm object returned by IBTTransportBatch::Done() is always available when BatchComplete executes. However, BatchComplete() can be called in a separate messaging engine thread, even before IBTTransportBatch::Done() returns. It is possible that when the adapter tries to access IBTDTCCommitConfirm object as a part of the BatchComplete implementation, there is an access violation. Use the following solution to avoid this condition.
Use events to avoid a race condition
The problem is solved with an event in the following example. Here the interface pointer is accessed through a property that uses the event. The get always waits for the set.
protected IBTDTCCommitConfirm CommitConfirm
{
set
{
this.commitConfirm = value;
this.commitConfirmEvent.Set();
}
get
{
this.commitConfirmEvent.WaitOne();
return this.commitConfirm;
}
}
protected IBTDTCCommitConfirm commitConfirm = null;
private ManualResetEvent commitConfirmEvent = new ManualResetEvent(false);

Now assign the return value from IBTTransportBatch::Done() to this property and use it in the BatchComplete call.
Handling Receive-Specific Batch Errors
Handling receive failures
BizTalk Server does not handle all failures; the adapter must be configured to handle errors like "no subscription."
When an adapter submits an operation (or batch of operations) to BizTalk Server there can be various reasons for failure. The two most significant are:
· The receive pipeline failed.
· A routing failure occurred while publishing a message.
The messaging engine automatically tries to suspend the message when it gets a receive pipeline failure. The suspend operation may not always be successful due to the following issues.
If the messaging engine hits a routing failure while publishing a message, then the messaging engine will not even try to suspend the message.
It is always possible that a message will fail. In such a situation, the adapter should explicitly call the MoveToSuspendQ() API and should try to suspend the message. When an adapter tries to suspend a message, one of the following should be true:
· The same message object that it had submitted (recommended) should be suspended.
· If the adapter has to create a new message, then it should set the message context of the new message with the pointer to the message context of the message that was originally submitted. The reason for this is that the message context of a message has a lot of valuable information about the message and the failure. Tthis information is required to debug the failed message.
Note If the adapter creates a new message object and suspends it, the adapter should copy the error information from the old message object to the new message object.
Some adapters, such as the HTTP adapter provided with BizTalk Server, do not require that the message be suspended. These adapters can just return an error back to their client.
Causes of failure
A simple cause of failure are the errors that can occur as the batch is constructed or when IBTTransportBatch::Done() is called.
· Submit failure: The Submit call can fail for a limited number of reasons, and all of them are fatal. These reasons include:
· Out-of-memory.
· The schema assembly has been dropped from the deployment. In this case, the Submit fails with a cryptic error. In the MQSeries adapter, the generic failure exception from BizTalk Server is caught, and an extended error message is written in the system event log. This message suggests that one of the possible causes of the error is that the schema assembly has somehow been dropped from the deployment.
In general, if submit fails you should try to suspend the message using the same transaction.
· IBTTransportBatch::Done() failure: The IBTTransportBatch::Done() call can fail for one of several reasons. In general, you should always attempt one suspend operation and only abort if that fails. One of the error codes you might receive from the failure of IBTTransportBatch::Done() is that BizTalk Server is trying to shut down. In this case, you should just end the transaction and leave it because the Terminate call is probably happening concurrently. Other scenarios occur when you have successfully constructed the batch and successfully executed IBTTransportBatch::Done(). In these cases, the errors are returned in BatchComplete and the adapter must work out what to do with them. The rest of the topic "Handling receive specific batch errors" deals with this case.

Processing BatchComplete errors
BatchCompelete() is a callback provided by the adapter that is invoked by BizTalk Server to indicate the completion status of a batch operation.
The most important parameter passed to BatchComplete is the batch status hResult. This indicates success or failure for the batch. If the batch failed, it means that none of the operations in the batch have succeeded. (See the topic, "Batches are Atomic.") The adapter goes through the batch status structure and determines which messages failed (this is known as filtering the batch).

Comments

Lompo said…
Welcome to this blog Ben
We are expecting great writings from you

Popular posts from this blog