Search for a command to run...
I'm Chris Bath, a technical lead specialising in full-stack development, solving complex technical problems, nurturing high-performing teams, and consulting companies to overcome their scaling needs.
No comments yet. Be the first to comment.
Building solid functional software and high-performance teams requires a treasure trove of information and insights, enabling informed decisions to be made promptly. Considering the many options available to achieve these desired outcomes in the curr...
This post explores how to customise JSON output using several Jackson features and Spring Boot . Even when you can't modify class definitions, learn valuable techniques like mixin and delegates . Use case We have the following class definition: publi...
I'm always hearing of organisations that are looking to embark on the journey of decomposing their monolith into microservices.
This post explores near real-time data migration challenges from monolithic to microservices architecture.
Several strategies, such as shared databases, APIs, and event-driven approaches, are explored, including a real-world example using change data capture (CDC) and event-driven architecture.
Many factors influence the decision to break down a monolith into microservices.
While moving to microservices comes with its challenges. The examples listed below are common to a monolith architecture in the average company, but a change of architecture alone won't solve them all.
As a monolith grows, the complexity of the moving parts increases the cognitive load on the engineers, increasing the time to realise value for the end customers.
One team adds 100 new tests that slow the pipeline by 1 minute, or someone breaks the build. It impacts everyone and causes some real problems.
When a company reaches a specific size, it's not uncommon to have multiple squads that own their particular areas (onboarding squad, checkout squad, etc.).
With a monolith, as the number of people working on it increases, the risk of stepping on each other toes rises. You start to see more merge conflicts, a need for clear ownership (Who should update this?), and security concerns (Am I the right person to change this?).
As a monolith application grows, its resource demand tends to increase, another scheduled job here, another high throughput API there. When performance issues arise, workarounds become common by throwing more compute resources or a larger db at the problem.
While no architecture offers a silver bullet in this space, monoliths can be very challenging to resolve technical debt, and it's not uncommon for a monolith to have business logic tangled across multiple domain boundaries. Each change comes with the risk of a large blast radius, which could impact millions of lines of code, and refactoring is frequently a long and slow process.
Monoliths tend to have many years of tenure. The tech stack that was modern at the time is now considered old and out of fashion. Dependencies are long out of support, attracting talent is becoming more challenging and adding new functionality is often slow and risky.
The key stakeholders are convinced and have given the green light to proceed, and now the difficult work starts. The How...
Each monolith to microservices journey is different. You may already have a mature golden path for building microservices, or you can pause new feature development. There are so many factors to consider for your journey.
Two things are for sure: we want this migration to succeed (how will you define and measure?), and we want it timely, e.g., sooner rather than later.
Today, we will focus on one of these challenges: dealing with the data currently residing within a monolithic application.
Data comes in many shapes and sizes, and we have many options regarding access and storage.
We will focus on relational SQL databases (MySQL, Postgres, MSSQL, etc.), as these are some of the most common data storage used in monolith applications in my experience.
We will look at approaches to allow our microservices to utilise this data.
One approach is to use the same database as the monolith, and I can already sense some ears burning at the thought of this (database per service comes to mind).
One upside of this approach is that we can defer data migrations and focus on extracting business logic.
I have seen this approach work well in the short term when a team is unfamiliar with what data truly represents their domain model or domains yet to be extracted.
With modern database platforms, you can limit what data is accessible and database operations performed, ensuring guardrails are in place.
This approach has some downsides that come to mind:
This approach is a stepping stone, and it can help to form a clearer understanding of a domain and provide quick access to data that has yet to migrate.
But you need to ensure good access controls and that this is a stage of migration, not the target end state.
Rather than directly accessing the monolith database, we can expose APIs in the monolith, allowing access to the required data.
Doing so allows us to provide a clear contract while hiding the internals of the database. These APIs can be migrated directly to microservices in the future.
A common deciding factor for using an API approach is when data consistency is crucial. For example, the customer's latest account balance is checked and reserved before allowing a payment.
This approach works well when the monolith release cycle does not become a bottleneck. If you consider the future state of the domain and apply this to the APIs, these can form the foundation of future microservice APIs.
We can look at an asynchronous approach to solving the data problem by emitting events using a message broker.
We can integrate the monolith to publish events directly to a message broker when things of interest occur.
In doing so, the monolith will proactively inform the platform when something interesting has happened.
With the monolith emitting the event directly to the message broker, no additional infrastructure is required compared to other approaches.
We can use CDC (change data capture) to inform us of all changes made within the monolith database.
Using a CDC approach allows us to reduce coupling between the monolith and microservices, as neither system needs to be directly aware of each other.
The CDC process emits change of a given table, and you have a few options which can be mixed and matched depending on your requirements.
The first approach is to emit changes from the table as is, e.g. any changes made in the customer table are sent using the same structure.
insert into customer(id, name) values(1, 'Alexandra')
A change capture process would allow us to emit an event that looked like this:
{
"before": null,
"after": {
"id": "1",
"name": "Alexandra"
},
...
}
For more complex data, you can format a specific payload and add it to a particular table, e.g., ' outbox`.
One way is to join tables together and save their outcome.
insert into outbox(message) values('{"name": "Alexandra", "address": { "postcode": "A postcode"} }')
A change capture process would allow us to emit an event that looked like this:
{
"before": null,
"after": {
"message": {
"name": "Alexandra",
"address": {
"postcode": "A postcode"
}
}
},
...
}
Publishing events representing the domain allows a clean approach for decoupled data sharing between the monolith and your microservices.
If done well, the emitting of these domain events will, over time, be migrated away from the monolith, with the relevant microservices taking over ownership of emitting these events.
One project I've worked on required us to solve the problem of accessing the data within the monolith.
We had several constraints and considerations:
The business was moving to a new CRM, and when we had new customer sign-ups or profile updates, this data needed to be in the CRM in near real-time for the customer support agents.
The CRM didn't support a pull model (e.g. it couldn't fetch data from our platform).
With these constraints and requirements, we opted for an event-driven approach using CDC.
Our monolith application required no changes, and was unaware of any of the new components.
The database only needed minor configuration updates, such as the bin retention period.
We needed a tool to perform the CDC process, and two good options stood out.
Debezium and AWS Database Migration Service (AWS DMS) we felt that Debezium was the more feature-rich offering and would provide a lot of flexibility, however, we ended up settling on using AWS DMS.
Our primary reason was that we were a small team with limited time and resources. We needed to keep operational overheads as low as possible. AWS DMS is a fully managed offering, allowing us to set up a working solution with minimal effort.
We used Terraform to manage the DMS infrastructure and replication task.
Kafka was the planned backbone of the microservice estate, and we were using the Amazon Managed Streaming for Apache Kafka (AWS MSK).
DMS has out-the-box support for Kafka as a target destination, Terraform managed these resources.
This microservice was responsible for reading the raw CDC events from Kafka and shaping them into domain events.
We used the Kafka Streams library to manipulate the CDC data into the desired format.
We had several different operations to apply to form the domain events we wanted, such as filtering, joining, restructuring and aggregation.
For many reasons, we did this in a new microservice rather than the monolith. A few noticeable ones are:
As the domain the microservices owned expanded, they took over ownership of emitting these domain events. It was transparent to other consumers, enabling a smooth transition from the monolith application.
With the enriched CDC events that formed the target domain events, we would put these back into Kafka, and these domain events were now accessible by any microservice that wished to subscribe to them.
This service was responsible for taking the data from the domain event topics, in this case from the topic domain.namespace.customer and converting these into API requests that the CRM would accept.
The micro-service is only aware of the outcome of the data pipeline, and it uses a simple Kafka consumer to consume the domain topic.
The CRM is the end sink for the data, a popular cloud-hosted CRM solution which exposes REST APIs.
The first five steps above were all reusable components forming part of our data pipeline from the monolith application.
It allowed us to move data out of the monolith in near real-time in a safe and controlled manner.
Adding new domain events was as simple as
And that was it. The domain events are now ready for anyone who needs it in the rest of the platform.
Less than a couple of hours of work and new events would be available in production with no changes required to the monolith application.
If I were to change anything with the implementation above, I would consider using Apache Flink rather than Kafka streams. When you need to support an architecture that doesn't revolve only around Kafka and needs to support a wide range of different data sources, streaming/batch data modes, and advanced data manipulation, Apache Flink is a very appealing solution.
We learned many lessons with running a data pipeline and loads of details I have skimmed over, but I will save that for a future post where we explore the developer experience.
We have covered a few different approaches to unlocking the data that lives within the monolith.
While it's not uncommon to deploy any combination of the strategies mentioned today throughout the lifetime of a migration project, these strategies form only one part of the giant puzzle. Understanding the trade-offs and when to apply them within the context of your project is the hard part.
What are your thoughts? What approaches have you used? Do let us know!
Thank you for reading.