Modular architectures

[Versión en castellano]This link opens in a popup window

Background

When we design a system architecture, we should be extremely cautious about its complexity and how the system could be maintained in the future. Keeping in mind that maintenance is not only related to understand how the system works, providing help to end users, but mainly about adding new functionalities which could complement or modify previously designed behaviour.

Hence, we should focus on different approaches which could help us to achieve this goal. Being the main one how to tackle modularity and subsystems intercommunication. Even though, systems should be built in an iterative way, we cannot despise a proper architecture definition, which is obviously based on our knowledge and previous experience.

In this post I’m going to focus on one example that we have been recently working on, and how we have implemented a modular architecture approach, having in mind from the very beginning what we do wanted to achieve and, give or take, how we wanted to build it.

The problem

In this case we had to tackle the development of an internal management application, with some specific considerations, related mainly to how end-users where going to interact with it. Since, in fact, application was not only intended for internal users, but also it was targeted to direct end-users’ interaction.

Moreover, this interaction had to consider different input and output channels, e-mail, WhatsApp, Telegram, Web, or APP to name but a few. These requirements are quite diffuse, you could create a simple monolith which could fulfil, theoretically, expectations. But provided you analyse it deep inside; you could face some issues:

  • Security ones… should I give access to my application to all users?
  • Interaction-type ones… if there is a new channel in the future, how could it be added to the system?
  • End-user identification… should I create end-users in my system? Is that not a layer of extraordinary complexity?

Thus, answering to all these questions requires designing the system to provide answers to all of them, which means:

  • We should create an API responsible of end-user interaction.
  • We are not responsible of end-user identification, that’s another system responsibility.
  • We should create components as independently and reusable as we could.
  • We must provide different notification mechanisms bot for internal and end-users.

Besides, a deeper analysis could give us another vision, not functional, but technical in which we could also provide answers to other questions as having independent development teams, as to end as soon as possible, but maintaining a common approach to the problem, avoiding repeating the wheel.

In the following scheme, I show you the full architectural vision of the system, a very schematic one indeed, but, perhaps, enough for having a better understanding.

00_esquema_general00_esquema_general

As you can see, I’ve depicted different modules, or subsystems, which are intended for different tasks, providing you a glimpse of how we have achieved a global functional architecture and proper technical one.

  • We have considered a common REST API which is used from the different interaction systems, I’ve called them agents.
  • But, as you can see, I’ve tried to isolate functionality, so not only do we have a Telegram or a WhatsApp agent, but they are sharing a common functionality core. Same applies to Web and APP agent.
  • There are also some insightful elements related to modularity as the hot-deployment PNP engine, micro-frontends for the frontend management application and an event-based delivered mechanism.

I’ll try to give you a more detailed explanation in the following points.

The common API subsystem

The idea behind of this common API is providing different agents an input/ouput mechanism, which concern is also agent security.

01_common_rest01_common_rest


As you can view this system is responsible of providing the input/output, but also is accountable of global system security on a per-agent basis.

  • We can create new agents.
  • These agents share a secret with the system, which is used to send and receive information. This secret can be changed in any moment.
  • System can exchange with the agents some concerns related to their configuration.
  • We can limit the number of requests per agent, that’s important just as another layer of security.
  • In fact, having each agent as an independent service (note that I’m not using the word micro-service but service) allows us to treat them independently and stop one of them if its behaviour is erratic or if it is being attacked, without compromising global system reliability and operation.

The web and APP subsystem

Although it could seem pretty obvious, neither the web client nor the app can store security tokens to exchange information with the server, security considerations are very important here, never do that. That means that we have to create a middleware which is accountable not only of storing these tokens, but also of identifying users.

02_webs_apps02_webs_apps


Besides whom has the responsibility of storing tokens, there are also some issues to have into consideration,

  • We share a common codebase for app and web. Actually, I’m quite disappointed with those frameworks that promise the write once, run everywhere. It’s not as easy as it seems.
  • Middleware is accountable of user identification and security. Not only do I talk about the different authentication mechanisms but, mostly, about the task of ensuring who can access which information, taking into account that core system has no knowledge about end-user.
  • We use device capabilities, service workers, notifications and geolocation.

BOTS: WhatsApp & Telegram

Do not reinvent the wheel. Design patterns are crucial when planning a proper architecture. Speaking about BOTS, you don’t have two different ones, you do have one common and shared subsystem and two – at the present moment – different implementations.

03_bots03_bots

Moreover, as you can see in the previous picture, the involved systems try to generalize as most as possible,

  • The common BOT API is a conversational system, a state machine.
  • It’s based on actions (what the user can do) and answers (what I, as a bot, can reply)
  • It’s the only responsible of backend interaction.
  • Specific details of WhatsApp and Telegram are, therefore, restricted to network communication, and bot specific details, namely maximum message size and button disposition.

As you can notice, I’m talking about typical software engineering patterns. Decoupling, isolation, single responsibility.

Events and queues

It’s difficult to imagine a system that has no need of events. Your system triggers events when something happens, and there are other systems (subsystems) which are subscribed to events to perform something. There is neither magic involved nor you need a complex Kafka queue to support events. It’s just a simple producer-subscriber pattern.

04_queues04_queues

Despite simplicity, if you want to create a truly modular system you need to add support for some "complex" problems as:

  • I do want to add new listeners, they could be hot discovered.
  • My listeners could have their own libraries, I don’t want to end with an humongous monolith.
  • That means… using advanced features as custom classloaders and a ci configuration that supports all of these. Neither too complex, nor too easy.

Hence, when you have to develop a system you should probably consider that you are going to need events and that doesn’t mean that you need a Kafka Queue, those are implementations details… a proper architecture should allow to replace your custom implementation for an alternative one if that’s needed in the future.

The management application

As I’ve stated in previous posts, perhaps one of the main problem when you develop a complex application is that a typical development workflow is not enough. You cannot end up with a monolith in the frontend, that is complex to maintain, complex to develop and complex to test. My proposal, divide your frontend in different micro-frontends, each of them should have its own micro-backend layer, as shown in the following pictures.

05_microfrontends05_microfrontends

As I stated when I talked about subscribers’ discovery, supporting microfronteds requires, again, hot discovery, custom classloaders, and a pretty complex configuration for bundling micro-backed and micro-frontend in a single deployable file. Fortunately you have to create the configuration only once, applying a proper Gradle multi-project configuration.

Besides the file bundling, there are some other matters to consider,

  • Supporting translation on a service basis
  • Supporting stylesheets on a service basis
  • Service orchestration… we are using REACT for frontend and we are orchestrating them server-driven with a WebComponent approach.

In summary

You don’t need to create utterly complex architectures, use only what you need. But keep in mind that you don’t have to seek simplicity for the sake of it. Some things are complex by nature, and your mission as an engineer is to make the complexity operative.