When you are the lucky owner of a 30 year-old application, the amount of data you are staying on is huge. The database is composed of hundreds of tables, both static and dynamic data. Some might have tens of thousands of records each month. Let's say that we deal with a distributed application with a classic web application container.
Obviously you try to keep up with the emerging technologies and you migrate to new frameworks over and over again with the same goal in mind: "...next time, it will be easier to implement a, b, c..". Then you reach a point where your app is composed of very smart modules such that even Robert C. Martin would buy you a drink for the SRP implementation.
In a business2business collaboration, you get the chance to extend your business model by extracting data from your system in a smart way. I will emphasize how that can happen.
When you need collaboration between components, you get into a debate on whether you should use a Service Bus or not. This article has offers a supporting argument for EAI, ESB and a bit of SOA. There are quite a few, very good articles emphasizing that care should be taken when an ESB is envisioned.
If you are the owner of those modules that need to orchestrate everything is in your house, the components listen to you. You invent the collaboration model. At this point, the ESB could add an overhead if it's not done properly.
Let's say you reached a critical point where you serve many customers and many businesses and some of them want custom functionalities, impossible to be sold to other clients. You do the math and it's highly unlikely that you will build that yourself.
Now you have a problem. If you don't move fast enough, the customer will search for alternatives: build software on its own or look at your competitors.
Example: When you want to export financial data from your system to other parties, you have to be very careful which data and what format you will use. You are not in the financial expertise domaine, so you might need some help. Also, you will not build financial modules yourself.
So, how would you approach this problem? The customer needs very small pieces of your data in a new functionality, but you can't build that yourself.
The simplified infrastructure looks similar to this picture. This would be multiplied by a factor of tens or even hundreds. For simplicity, we kept just 3 stations belonging to 3 different customers here. There could be hundreds or even more.
And here's the moment when an ESB solution opens up a new set of possibilities.
There are two options for a new business model.
A. Let it ask for information from your system when it wants/is able to.
B. Send events to it when something happens with the thing it wants to control
For both situations, you can define a service-level agreement (SLA and create a payment plan based on usage.
Items to solve from the technical perspective
If you want to pursue a similar strategy, a few questions need answers.
Q: How do you send events from Station A1 … An to Client A headquarters?
A: Create a Service Bus layer in between your servers and clients. Thus, this ESB is meant to facilitate the collaboration between your servers and your clients, and not between the components your architecture has. Still, there's nothing wrong in making components collaborate via a similar BUS, but which is a different topic for discussion. You could establish multiple ESB instances (with/without load balancers) just to avoid the single point of failure problem.
Q: How do you enable Client A to ask for information from specific servers?
A: Each request can be parsed and forwarded to specific targets. The logic of doing this can be placed directly in the ESB with XQueries. Parse the request, decide on which endpoint to go and then forward it and wait for a response. This is a bird eye view of the routing capabilities of an ESB. The routing strategies could be combined with database endpoint extractions based on request header contents. Each server must have a communication channel with the ESB.
Q: How do you build services specific to one client only, but with the capacity of being extended just in case the sales team is able to sell something?
A: Consider separating the concerns such as: internal service and internal definition, and external service and external definition. Use the 'transforming' capability an ESB has. To do the actual 'extension' or to 'reuse' the service you might need to do some basic touches in the ESB instance itself and add new endpoints and routings. Reusability here means that you will create new external definitions and services, while all the internal ones remain unchanged.
Q: How do you build services specific to multiple clients?
A: You have multiple options here. You should prefer the tricks from the previous question. Still, if you really need that specific thing, just name the service accordingly and treat it as a regular service. E.g.: FinanceForClientXService_V1.
Q: How can you make the service independent of the client side changes regarding the message that it is sent/received?
A: This is where the transformations help you. An ESB is capable of performing message transformations, from internal definitions to external definitions. That usually happens in a dedicated language, like XQuery, but you could opt for sending, through the ESB directly, a 'modified' message.
Q: How do you cope with subscriptions for services shared between multiple clients?
A: You could build up an entire business out of the data extraction from your old data farm. The easy way would be to create a database with subscriptions (client name, service, expiration period, etc) and 'consult' the database on each request.
Q: How do you handle multiple versions of a service? Client A may want more out of your service, while client B does not want to pay a cent for anything more.
A: Here's when you need smart versioning. ServiceX_v1, ServiceX_v2 could be the exact same service from the functional perspective, with just one or more different fields in the message delivered by both. Still, from the ESB perspective they can be treated as different services. Yet, it is possible to define an internal proxy and an external proxy service for the same service. The external proxy behaves as if it knows only the 'external definition' while the internal proxy knows only the 'internal definition'.
Q: Aren't those items similar to what SOA proposes?
A: In pure SOA, the services have a dedicated "source of data". This is not the case in many situations. It is difficult to migrate systems to pure SOA architectures. SOA can make use of a service, bus but we just move the intersection point between services from app server to the ESB.
Q: How does a client connect to the ESB? How does the application server connect to the ESB?
A: For instance, via HTTP. Everything is a collaboration among web services. Application server -> ESB -> Client. All parties must have accessible service endpoints. This adds some static configuration to the endpoints. The endpoints don't change that much, so it won't become a pain maintaining the endpoints. All parties will have to define their message definitions with names and types independently. There's also the problem with security, but I won't go into any detail since that's not ESB specific. We use the HTTP protocol thus, ssl, https and certificates are the answer. They are easily configurable in the ESB solutions.
The beautiful part of the ESB is that you can add transformations. It does not matter how the clients name a field from a message. It could be anything. Even a different type. However, you need to know exactly how the interface looks. Both client side interfaces and your interfaces have to be known by the ESB. The interface from your side will be similar to
<envelope>
<someObject>
<objectName>some value
</objectName>
<someObject>
</envelope>And for Client A it will be transformed to
<envelope>
<someObject>
<myObjectName>some value
</myObjectName>
<someObject>
</envelope>Similarly, for Client B it will be
<envelope>
<someObject>
<nameOfObject>some value
</nameOfObject >
<someObject>
</envelope>The problem with backwards compatibility and multiple versions of the same service occurring at the same time is easily solvable by treating each service as a new service, something like ServiceForFinance_V1 and ServiceForFinance_V2 and so on. Each has its own message deffinitions. One client could make use of both versions at the same time. It is not neccessary to build V2 to do everything that V1 does + some additional stuff. They can have completely different functionalities.
And now the fun part. I described two possiblities here: to let the client ask for information or to inform the client when something changed.
For the second option, when you inform the clients about changes, this is where the fun starts. You can never rely on the client's machines, infrastructure, capability of responding to your messages, storing them and process them if that client has a lot of stations in the field. Various clients have various processing speeds. You need a way to configure the event notifications per client capability.
Let's say that Client A is capable of receiving 100 messages per second, while Client B can do only 50, and Client C can do only 2 per second.
At this point, the ESB solution you choose becomes very, very important. You need to queue messages and throttle them.
Many ESB engines have throttling capabilities. I have seen "in house" throttlers that work very well.
And then, even more fun. You add logging to your services and these will be capable of creating nice dashboards with solutions like Splunk (log farm indexer), and see who uses what and when. Those tools are affordable and some require more development then others, but the real fun part is the big things that you can build with them. You still have to be very technical to use them, so nothing happens by magic. There are a lot more fun features of the ESB which will be covered in a future article.
Send me your thoughts vlad dot vesa at tss-yonder.com