Integration architecture can be an intimidating topic to dive into. There is a plethora of different technologies, processes, platforms and patterns advertised to meet your every integration need. So, how do we break out of analysis paralysis and begin on our rewarding journey of integration architecture so we can give the devs something useful to do?
Believe it or not – and feel free not to, I won’t take it personally – there are just three super-patterns, which can model any integration scenario you can think of. Further, these three super-patterns each break down into two sub-patterns, giving us – if my math is correct – six concrete integration patterns.
With these patterns, it becomes a much simpler task of identification and modelling. Each sub-task maps naturally to technologies that fit the pattern.
Data vs Metadata
Before we enumerate the different super-patterns and sub-patterns, we need to talk about data and metadata. The flow of data, metadata or a combination of both, compose each pattern outlined in this article, so it is important to understand the distinction between the two.
Put simply, data is content, metadata is context. The term “meta” implies self-reference. For example, consider an image file. The data is the image itself, the bits and bytes which a photo program will translate to generate a picture. The metadata in this example is data about the data – information about the image such as the size, resolution, encoding format and colour model.
Introducing Data and Metadata Messaging Integration Terminology (DaMMIT)
Until today, there was no accepted standard for differentiating between data and metadata in architectural diagrams. This article proposes a new notation to do just this.
- A solid circle at the tail of an arrow identifies the flow of data.
- A hollow circle at the tail of an arrow identifies the flow of metadata.
This notation is compatible with other standards, which deal with the head or shaft of the arrow. For example, we draw an asynchronous flow of data as:
Or a response of metadata as:
Pull Based Integration
The first – and arguably the most prevalent – integration super pattern is that of Pull Based Integration. We use the pull-based interaction patterns when a consumer is requesting a response from a single provider. Data or metadata is retrieved without creating a residual obligation.
This super-pattern tends to map to synchronous request-response technologies and processes such as APIs or Web Services.
Pull-based integration is composed of two sub-patterns: query-response and request-confirm.
The query-response pattern is a query of data held by the provider using metadata to identify or filter the returned data.
The use-cases for this pattern are many and varied. Any GET verb on a RESTful API probably follows this pattern. For example, consider a “Get Account Details” call by a mobile application. The app sends metadata of the account (e.g. identification information) to the provider which returns the account data (name, address, etc.).
The request-confirm pattern is a query of the state of the data held by the provider. The provider returns metadata on the supplied data.
The use-cases for the request-confirm pattern are more nuanced and rarer than that of the query-response pattern. As an example, consider a form-validation service. The consumer would send the completed form to the provider which would validate the form and return metadata describing its acceptability.
Push Based Integration
Pushed-based integration is when the provider sends data or metadata to one or more external consumers. The two patterns for providing push-based interaction are notification and information distribution.
Push-based patterns map best to asynchronous technologies such as event streaming.
A notification is a fire-and-forget pattern that is used to perform notifications about data changes. This pattern should be used when a provider is notifying an interested consumer that an event has taken place on their data.
An example use-case for the notification pattern is a delivery status update of a courier company. Metadata on a package (e.g. its delivery status) is sent to consumers to notify them of a state change of the data.
The information distribution is the second variation of the push-based integration super pattern. It is an informal information exchange between parties to perform data synchronisation. We should use this pattern when a provider has changed data that a subscriber has registered an interest to keep synchronised.
An example use-case for this pattern is the distribution of a schedule. A large conference may publish a schedule of lectures available to conference goers through an application. Schedule changes can trigger a push of the data from the provider (conference organiser) to the consumers (conference attendees).
The Information Exchange super pattern is commonly associated with more complex integrations. The request-response and commercial transaction sub-patterns form the information exchange super-pattern.
Enterprise Application Integration, or Enterprise Service Bus topologies typically implement the information exchange patterns.
The request-response pattern is a set of related data exchanges between systems for a single workflow, with data flowing in both directions.
This pattern is common in the UBL service interactions. For example, the quotation process involves the customer sending data pertaining to the quotation to the supplier, who responds with data of the quotation itself.
This pattern implements a transaction which requires a form of non-repudiation. The term non-repudiation refers to an assertion which cannot be disputed. There are two main types of non-repudiation in integration architecture – non-repudiation of receipt, and non-repudiation of origin.
Non-repudiation of receipt means that a system sending a message has a guarantee that the destination system has in fact received it; and has concrete evidence to refute any contrasting claim (“I never got that message!”) made by the receiving system. Typically, this is achieved by requiring a receipt on delivery sent by the receiving system back to the sender, which the sender then retains to prove delivery was successful.
Non-repudiation of origin means that the receiving system can be sure that the message originated where they expected and that it has not been altered in transit. This is achieved by payload encryption and digital signatures and refutes any contrasting “I never sent that!” or “That’s not what I sent!” claims made by the sender.
The use-cases for this pattern are somewhat tautological – commercial transactions. Payments or other legally binding transactions between systems will require this pattern of integration.
Integration architecture can be intimidating. However, if we keep in mind these simple building-blocks of super-patterns and sub-patterns, and if we simplify our interactions as the flow of data and metadata between systems, then we can model even large or complex applications consistently and understandably.