I recently read Sam Newman’s book on Monolithic to Microservices architecture. I had read his previous book on Building Microservices on related topic that focused more on design and implementation of microservices but there is some overlap of topics in these books.
Chapter 1 – Just Enough Microservices
The first chapter defines how microservices can be designed to be deployed independently by modeling around a business domain.
Benefits
The major benefits of microservices include:
Independent Deployability
Modeled Around a Business Domain
The author explains one of common reason for three-tier architecture with UI/Business-Logic/Database is so common is due to Conway’s law that predicates that system design mimics organization’s communication structure.
Own Their Own Data
In order to keep reduce coupling, author recommends against sharing data with microservices.
Problems
Though, microservices provide you isolation and flexibility but they also add complexity that comes with network communication such as higher latencies, distributed transactions, and handling network failures. Other problems include:
User Interface
The author also cautions against focusing only on the server side and leaving UI as monolithic.
Technology
The author also cautions against chasing shiny new technologies instead of leveraging what you already know.
Size
The size of a microservice depends on the context but just having a small-size should not be the primary concern.
Ownership
The microservices architecture allows strong ownership but it requires that they are designed around the business domain and product lines.
Monolith
The author explains monolithic apps where all code is packaged into a single process.
Modular Monolith
In modular monolith, the code can be broken into different modules and is for deployment.
Distributed Monolith
If boundaries of microservices are not loosely coupled, they can result in distributed monolith that has disadvantages of monolithic and microservices.
Challenges and Benefits of Monolith
The author explains that monolithic design is not necessarily a legacy but a choice that depends on the context.
Cohesion and Coupling
He uses cohesion and coupling when defining microservices where stable systems encourage high cohesion and low coupling that provides independent deployment and minimize chatty services.
Implementation Coupling
The implementation coupling may be caused by sharing domain or database.
Temporal Coupling
The temporal coupling using synchronous APIs to perform an operation.
Deployment Coupling
The deployment coupling adds risk of adding unchanged modules to the deployment.
Domain Coupling
The domain coupling is caused by sharing full domain object instead of events or reducing unrelated information.
Domain-Driven Design
The author reviews domain-driven design concepts such as aggregate and bounded context.
Aggregate
In DDD, an aggregate uses a state machine to manage lifecycle of a grouped object that can be used to design a microservice so that it handles the lifecycle and storage of those aggregates.
Bounded Context
Bounded context represents a boundary of business domain that may contain one or more aggregates. These concepts can be used to define service boundaries so that each service is cohesive with a well-defined interface.
Chapter 2 – Planning a Migration
The chapter two defines a migration path for micro-services by defining goals for the migration and why you should adopt a microservice architecture.
Why Might You Choose Microservices
Common reasons for such architecture includes:
- improving autonomy
- reduce time to market
- scaling independently
- improving robustness
- scaling the number of developers while minimizing coordination
- embracing new technologies
When Might Microservices Be a Bad Idea?
The author also describes scenarios when a microservice architecture is a bad idea such as:
- when business domain is unclear
- early adopting microservices in startups
- customer-installed software.
Changing Organizations
The author describes some of ways organizations can be persuaded to adopt this architecture using Dr. John Kottler’s eight-step process:
- establishing a sense of urgency
- creating the guided coalition
- developing a vision and strategy
- communicating the change vision
- empowering employees
- generating short-term wins.
Importance of Incremental Migration
Incremental migration for microservice architecture is important so that you can release these services to the production and learn from the actual use.
Cost of Change
The author explains cost of change in terms of reversible and irreversible decisions commonly used at Amazon.
Domain Driven Design
The author goes over domain-driven design again and shows how bounded context can define boundaries of the microservices. You can use event storming to define a shared domain model where participants define first domain events and then group those domain events. You can then pick a context that has few in-bound dependencies to start with and using strangler fig pattern for incremental migration. The author also shows two-axis model for service decomposition by comparing benefit vs ease of decomposition.
Reorganizing Teams
The chapter then reviews team restructure so that you can reassign responsibilities towards cross-functional model who can fully own and operate specific microservices.
How Will You Know if the Transition is Working?
You can determine if the transition is working by:
- having regular checkpoints
- quantitative measures
- Qualitative measures
- Avoiding the sunk cost fallacy.
Chapter 3 – Splitting the Monolith
The chapter three describes how to split the monolith in small steps.
To change the Monolith or Not?
You will have to decide whether to move existing code as is or reimplement.
Refactoring the Monolith
A key blocker in breaking the monolith might be tightly coupled design that requires some refactoring before the migration. The next step in this process might be a modular monolith where you have a single unit of deployment but with statically linked modules.
Pattern: Stranger Fig Application
The Strangler Fig Application incrementally migrates existing code by moving modules to external microservices. In some cases those microservices may need to invoke other common behavior in the monolith.
HTTP Proxy
If the monolith is using an HTTP reverse proxy to intercept incoming calls, it can be extended to redirect requests for the new service. If the new service chooses to implement a new protocol, it may require other changes to the proxy layer that could add risk and goes against general recommendation of “Keep the pipes dumb, the endpoints smart.” One way to remediate is to create a layer for converting protocol from the legacy to the new format such as REST to gRPC.
Service Mesh
Instead of a centralized proxy, you can use service meshes such as Envoy that is deployed as a control-plane along with each service that acts as a proxy for communicating with the service.
Message Interception and Content-based Routing
If a monolith is using messaging, you can intercept messages and use a content-based router to send messages to the new service
Pattern: UI Composition
The UI composition looks at how user interface can be migrated from monolithic backend to microservice architecture.
Page Composition
The page-composition migrates one page at a time based on product verticals to ensure that old page links are replaced with the new URLs when they are changed. You can choose a low-risk vertical for UI migration to reduce the risk of breaking functionality.
Widget Composition
The widget composition reduces the UI migration risk by just replacing a single widget at a time. For example, you may use Edge-Side Includes (ESI) to define a template in the web page and a web server splices in this content. Though, this technique is less common these days due to browser can make requests to populate a widget. This technique was used by Orbitz to render UI modules from a central orchestration service but due to its monolithic design, it became a bottleneck for coordinating changes. The central orchestration service was then migrated to microservices incrementally.
Mobile Applications
These UI composition changes can also be applied to mobile apps, however mobile app is a monolith and whole application needs to be deployed. Some organizations such as Spotify allow dynamic changes from the server side.
Micro Frontends
Modern web browsers and standards such as Web Component specification to help build single page apps and micro frontends by using widget-based composition.
The UI composition is highly effective when migrating vertical slices and you have the ability to change the existing user interface.
Pattern: Branch by Abstraction
The “Branch by Abstraction” can be used with incremental migration using Strangler Fig when the functionality is deeply nested and other developers may be making changes to the same codebase. In order to prevent merge conflicts from large changes and to keep minimal disruption for developers, you create an abstraction for the functionality to be replaced. This abstraction can then be implemented by both existing code and new implementation. You can then switch over the abstraction to new implementation once you are done and clean up old implementation.
Step1: Create abstraction
Create an abstraction using “Extract Interface” refactoring.
Step2: Use abstraction
Refactor the existing clients to use the new abstraction point.
Step3: Create new implementation
Implement the abstraction to call the external service inside the monolith. You may simply return “Not Implemented” errors in the new implementation and deploy code into production as this new service isn’t actually being called.
Step4: Switch implementation
Once the new implementation is done, you can switch the abstraction to point to the new implementation. You may also use feature flags to toggle back and forth quickly.
Step5: Clean up
Once the new implementation is fully working, the old implementation can be removed and you may also remove the abstraction if needed.
Fallback Mechanism
A variation of the branch by abstraction pattern called verify branch by abstraction can be used to implement a live verification where if the new implementation fails, then the old implementation could be used instead.
Overall, branch by abstraction is a fairly general-purpose pattern and useful in most cases where you can change the existing code.
Pattern: Parallel Run
As the strangler fig pattern and branch by abstraction pattern allow both old and new implementations to coexist in production, you can use parallel runs to call both implementations and compare results. Typically, the old implementation is considered the source of truth until the new implementation can be verified (Examples: new pricing system, FTP upload).
N-Version Programming
Critical control systems such as air flight systems use redundant subsystems to interpret signals and then use quorum to find the results.
Verification Techniques
In addition to simply comparing results, you may also need to compare nonfunctional aspects such as latency and failure rate.
Using Spies
A spy is used with unit testing to stub a piece of functionality such as communication with an external service and verify the results (Examples: sending an email or remote notification). Spy is generally run as external process and you may use record/play to replay the events for testing. GitHub’s Scientist is a notable library for this pattern.
Dark Launching and Canary Releasing
The parallel run can be combined with canary release to test the new implementation before releasing to all users. Similarly, dark launching allows enabling the new implementation to only selected users (A/B testing).
Parallel run requires a lot of effort so care must be taken when it’s used.
Pattern: Decorating Collaborator
If you need to trigger some behavior inside the monolith, you can use decorating collaborator pattern to attach new functionality (Example: Loyalty Program – earn points on orders). You may use proxy to intercept the request and add new functionality. On the downside, this may require additional data, which adds more complexity and latency.
This is a simple and elegant approach but it works best the required information can be extracted from the request.
Pattern: Change Data Capture
This pattern allows reacting to changes made in a datastore instead of intercepting the calls. This underlying capture system is coupled with the monolithic datastore (Example: Issue Loyalty Cards – trigger on insert that calls Loyalty Card Printing service).
Implementing Change Data Capture
Database triggers
For example, defining triggers on relational database that calls a service when a record is inserted.
Transaction log pollers
The transactional logs from transactional databases can be inspected by external tools to capture data changes and then pass this data to message brokers or other services.
Batch delta copier
This simply scans the database on a regular schedule for the data that has changed and copies this data to the destination.
The change data capture has a lot of implementation challenges so its use must be kept to minimal.
Chapter 4 – Decomposing the Database
This chapter reviews patterns for managing a single shared database:
Pattern: The Shared Database
The implementation coupling is generally caused by the shared database because the ownership or usage of the database schema is not clear. This weakens the cohesion of business logic because the behavior is spread across multiple services. The author points that the only two situations where you may share the database are when using database for read-only static reference data or when a service is directly exposing a database to handle multiple consumers (Database as a service interface). Also, in some cases the work involved with splitting the database might be too large for incremental migration where you may use some coping patterns to manage the shared database.
Pattern: Database View
When sharing a database, you may define database views for different services to limit the data that is visible to the service.
The Database as a Public Contract
When sharing a database, it might be difficult who is reading or writing the data especially when different applications use the same credentials. This also prevent changing the database schema because some of the applications might not be actively maintained.
Views to Present
One way to change schema without breaking existing application is to define views that looks like old schema. The database view may also just project limited information to implement a form of information hiding. In some cases you may need a materialized view to improve performance. You should use this pattern only when existing monolithic schema can’t be decomposed.
Pattern: Database Wrapping Service
The database wrapping service hides the database behind a service. This can be used when the underlying schema can’t be broken down. This provides better support for customized projection and read/write than the database views.
Pattern: Database-as-a-Service Interface
In cases when you just need to query the database, you may create a dedicated reporting database that can be exposed as a read-only endpoint. A mapping engine can listen for changes in the internal database and then persist them in the reporting database for query purpose by internal/external consumers.
Implementing a Mapping Engine
You may use a change data capture system (Debezium), a batch process to copy the data or a message broker to listen for data events. This allows presenting data in different database technology and provides more flexibility than the database views.
Pattern: Aggregate Exposing Monolith
When a microservice needs a data inside the monolith database, you can expose a service endpoint to provide the access to the domain aggregate. This pattern allows exposing the information in a well defined interface and is safer than exposing the shared database despite additional work.
Pattern: Change Data Ownership
If the monolith needs to access the data in a shared database that should belong to the new microservice, then monolith can be updated to call the new service and treat it as a source of truth.
Database Synchronization
As a strangler fig pattern allows switching back to monolith if we find an issue in the microservice but it requires that the data between the monolith and the new service remains in sync. You may use database view and a shared database for short term until the migration is successfully completed. Alternatively, you may sync both databases via code but it requires some careful thoughts.
Pattern: Synchronize Data in Application
Migrating data from one database to another requires performing synchronization between two data sources.
Step 1: Bulk Synchronize Data
You may take a snapshot of the source data and import it into the new data source while the existing system is kept running. As the source data might be changed while the import process is running, a change data capture process can be implemented to import the changes since the import. You can then deploy new version of the application after this process.
Step 2: Synchronize on Write, Read from Old Schema
Once both databases are in sync, the new application writes all data to both databases while reading from the old database.
Step 3: Synchronize on Write, Read from New Schema
Once, you verify the reads from new database work, you can switch the application to switch the new database as a source of truth.
Pattern: Trace Write
This is a variation of the synchronize data in application pattern where the source of truth is moved incrementally and both sources are considered sources of truth during migration. For example, you may just migrate one domain aggregate at a time and other services may use either data source depending on the information they need.
Data Synchronization
If data is duplicated inconsistency, you may need to apply following options:
- Write to one source – data to the other source of truths is synchronized after the write.
- Send writes to both sources – The client makes a call to both sources or use an intermediary to broadcast the request.
- Send writes to either source – the data is synchronized behind the scene.
The last option should be avoided as it requires two way synchronization. In other cases, there will be some delay in the data being consistent (eventual consistency).
Splitting Apart the Database
Physical versus Logical Database Separation
A physical database can host multiple logically separated schemas so migration to separate physical databases can be planned later to improve robustness and throughput/latency. A single physical database can become a single point of failure but it can be remedied by using multi-primary database modes.
Splitting the Database First, or the Code?
Split the Database First
This may cause multiple database calls instead of one action such as SELECT statement or break transactional integrity so you can detect performance problems earlier. However, it won’t yield much short-term benefits.
Pattern: Repository per bounded context
Breaking down the repositories along the boundaries of bounded context help understand dependencies and coupling between tables.
Pattern: Database per bounded context
This allows you to decompose database around the lines of bounded context so that each bounded context uses a distinct schema. This pattern can be applied in startups when the requirements may change drastically so you can keep multiple schemas while using monolithic architecture.
Split the Code First
This allows understanding data dependencies in the new service and benefit of independent deployment thus offering the short-term improvements.
Pattern: Monolith as data access layer
This allows creating an API in the monolith to provide access to the data.
Pattern: Multi-schema storage
When adding new tables while migrating from the monolith, add new tables to its own schema.
Split Database and Code Together
This split the code and data at once but it takes more effort.
Pattern: Split Table
This splits a single shared table into multiple tables based on boundaries of bounded contexts. However, this may break database transactions.
Pattern: Move Foreign-Key Relationship to Code
Moving the Join
Instead of using database join but in the new service, you will need to query the data from another service.
Database Consistency
As you can’t rely on the referential integrity enforced by the database with multiple schemas, you may need to enforce it in the application such as:
- check before deletion or existence but it can be error prone.
- handle deletion gracefully – such as not showing missing information and services may also subscribe to add/delete events for related data (recommended).
- don’t allow deletion
Shared Static Data
Duplicate static reference data
This will result in some data inconsistencies.
Pattern: Dedicated reference data schema
However, you may need to share the physical database.
Pattern: Static reference data library
This may not be suitable when using different programming languages and you will have to support multiple versions of the library.
Pattern: Static reference data service
This will add another dependency but it can be cached at the client side with update events to sync the local cache.
Transactions
ACID Transactions
This will be hard with multiple schemas but you may use state such as PENDING/VERIFIED to manage atomicity.
Two-Phase Commits
This breaks transaction into voting and commit phases and rollback message is sent if any worker doesn’t vote for commit.
Distributed Transactions – just say No
Sagas
A saga or long-lived transactions use an algorithm that can coordinate multiple changes in state but avoid locking resources. It breaks down LLT into a sequence of transactions that can occur independently as a short-lived.
Saga Failure Modes
Saga provides backward and forward recovery where backward recovery reverts the failure by using compensating transactions. The forward recovery allows continuation from the failure by retrying it.
Note: The rollback will undo all previously executed transactions. You can move forward the steps that are most likely to fail to avoid triggering compensating transactions on large number of steps.
Implementing Sagas
- orchestrated sags – You may use multiple orchestration to break down the saga using BPM or other tools.
- Choreographed sagas – This broadcasts events using a message broker. However, the scope of saga transaction may not be apparently visible.
Chapter 5 – Growing Pains
More Services, More Pain
Ownership at Scale
- Strong code ownership – large scale microservices
- Weak code ownership
- Collective code ownership
Breaking Changes
A change in a microservice may break backward compatibility or other changes for consumers. You can ensure that you don’t break contracts when making changes to the services by using explicit schemas and maintaining semantics. You may also support multiple versions of the service if you need to break backward compatibility and allow existing clients to migrate to the new version.
Reporting
A monolithic database simplifies reporting but with different schemas and databases, you may need to build a reporting database to aggregate data from different services.
Monitoring and Troubleshooting
A monolithic app is easier to monitor but with multiple microservices you will need to use log aggregation and define a correlation id with tracing tools to see a transaction events from different services.
Test in Production
You may use synthetic transactions to perform end-to-end testing in production.
Observability
You can gather information about the system by tracing, logs and other system events.
Local Developer Experience
When setting up a local environment, you may need to setup a large number of services that can slow down development process. You may need to stub out services or use tools such as Telepresence proxy to call remote services.
Running Too Many Things
You may use Kubernetes to manage the deployment and troubleshooting the microservices.
End-to-End Testing
A microservice architecture increases the scope of end-to-end testing and you have to debug false negative due to environmental issues. You can use following approaches for end-to-end testing:
- Limit scope of functional automated tests
- Use consumer-driven contracts
- Use automated release remediation and progressive delivery
- Continually refine your quality feedback cycles
Global versus Local Optimization
You may have to solve same problem with multiple services such as service deployment. You may need to evaluate problems as irreversible or reversible decisions and adopt a broader discussion for irreversible decisions.
Robustness and Resiliency
Distributed systems exhibit a variety of failures so you may need to run redundant services, use asynchronous communication or apply other patterns such as circuit breakers, retries, etc.
Orphaned Services
The orphaned services don’t have clear owners so you can’t immediately fix it if the service stops working. You may need a service registry or other tools to track the services, however some services may predate these tools.