Vaughn Butt
Senior Integration Developer
Why Dumb Pipes are the Best Integration Design
The Enterprise Service Bus (ESB) and the descendant, the API Gateway (API-G), were originally created to apply routing, mediation, and transformation to integrated systems.
This was good.
Application developers did not need to implement the same logic in every app and could leave these plumbing issues to the plumbers.
Over time, each vendor added more and more features to their ESB, slowly creating an (almost) general-purpose programming language in their platform.
This evolution is like Skynet acquiring capabilities, but instead of gaining self-awareness the ESB or API Gateway just becomes another application platform . This turns the platform from a pipe with many features into a lousy application platform with numerous limitations; essentially a platform to create smart pipes.
Well, I say “Stop it!”.
Those who deliver solutions with your platform and know what they are doing do not want to write applications in your shiny new language.
Application developers want to write business applications using application languages such as Java, Go, Python etc. And they don’t want to worry about protocol mediation, message routing, or data format transformation. On the other hand, Integration developers just want to build interoperability between applications, while addressing cross-cutting concerns.
I do think that the list of features integration developers want to use seems to be ever-growing, and this is one of the reasons platform vendors keep adding more features.
Unfortunately, we have seen this pattern before. For example, enterprise databases can be used to build and host web services. The result is an instance of the ‘Inner Platform Effect’ antipattern.
“The Inner-Platform Effect is a result of designing a system to be so customisable that it ends up becoming a poor replica of the platform it was designed with. This “customisation” of this dynamic inner-platform becomes so complicated that only a programmer, (and not the end user,) is able to modify it.”
Here is another example.
“The idea was that business rules, such as the conditions required to cancel an order, do not need to be coded in things like C#. They could be placed in some external store, such as an XML file or a database table, and a programmer would never need to modify the actual “code” to change the rule. He would, instead, modify the XML file or database table.
Unfortunately, such wonderful ideas often come face-to-face with the ultimate idea-wrecker we call ‘reality’. As it turns out, the ultimate rules engine that never involves modifying the core codebase looks something like ‘Microsoft Access’. Many technology companies that try to build this end up with some horrible system that’s halfway between that and a maintainable application.”
Zawinski’s Law of Software Envelopment, also known as Zawinski’s Law, states: “Every programme attempts to expand until it can read mail. Those programmes which cannot so expand are replaced by ones which can.”
My assertion is that “smart pipes” being business applications written using integration tools, are the equivalent of the Web service in the Database or the Editor that can read email.
I said we have seen this pattern before! But also, fortunately, we have seen this pattern before. We now know what to do to avoid it and how to avoid the consequences when we can’t.
Note, this is not a rant about Databases or an Enterprise Service Bus or API Gateways with too many features. Or even the fact that all those tools eventually seem to succumb to the second-system effect.
This is advice to be aware of, the consequences when you use those features.
The Advice Bit
While, under some circumstances, it might make sense to implement business functions in an integration platform (that is, build or use a smart pipe) – that is OK – just be careful, and:
- Consider the consequences; business functionality is required to be implemented on a platform that it is not suited for.
- Consciously delineate responsibilities between processes running on application platforms, and processes running on integration platforms.
The power is in your hands – use it wisely. Avoid building or using Inner Platforms.
The best designs use dumb pipes.
+Follow us on LinkedIn to receive further integration industry updates from around the globe.