Network architecture
See: Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, 1st Edition, Publisher : O’Reilly Media; 1st edition (April 18, 2017)
See: Patterns of Enterprise Application Architecture (Addison-Wesley Signature Series (Fowler)) 1st Edition
See: Fundamentals of Software Architecture: An Engineering Approach 1st Edition, January 2020
See also: Event-driven architecture (EDA), Microservices, API, API Bibliography – Application Programming Interface Books and References
Building Event-Driven Microservices: Leveraging Organizational Data at Scale, 1st Edition, by Adam Bellemare, 2020, B08C9V1FC9 (BEDM)
Fair Use Source: B08C9V1FC9 (BEDM)
Organizations today often struggle to balance business requirements with ever-increasing volumes of data. Additionally, the demand for leveraging large-scale, real-time data is growing rapidly among the most competitive digital industries. Conventional system architectures may not be up to the task. With this practical guide, you’ll learn how to leverage large-scale data usage across the business units in your organization using the principles of event-driven microservices.
Author Adam Bellemare takes you through the process of building an event-driven microservice-powered organization. You’ll reconsider how data is produced, accessed, and propagated across your organization. Learn powerful yet simple patterns for unlocking the value of this data. Incorporate event-driven design and architectural principles into your own systems. And completely rethink how your organization delivers value by unlocking near-real-time access to data at scale.
You’ll learn:
Adam Bellemare is a Staff Engineer, Data Platform at Flipp. He’s held this position since 2017. He joined Flipp in 2014 as a senior developer at Flipp. Prior to that, he held positions in embedded software development and quality assurance. His expertise includes: Devops (Kafka, Spark, Mesos, Zookeeper Clusters. Programmatic Building, scaling, destroying); Technical Leadership (Bringing Avro formatting to our data end-to-end, championing Kafka as the event-driven microservice bus, prototyping JRuby, Scala and Java Kafka clients and focusing on removing technical impediments to allow for product delivery); Software Development (Building microservices in Java and Scala using Spark and Kafka libraries); and Data Engineering (Reshaping the way that behavioral data is collected from user devices and shared with our Machine Learning, Billing and Analytics teams).
Preface
Conventions Used in This Book
O’Reilly Online Learning
How to Contact Us
Acknowledgments
1. Why Event-Driven Microservices
What Are Event-Driven Microservices?
Introduction to Domain-Driven Design and Bounded Contexts
Leveraging Domain Models and Bounded Contexts
Aligning Bounded Contexts with Business Requirements
Communication Structures
Business Communication Structures
Implementation Communication Structures
Data Communication Structures
Conway’s Law and Communication Structures
Communication Structures in Traditional Computing
Option 1: Make a New Service
Option 2: Add It to the Existing Service
Pros and Cons of Each Option
The Team Scenario, Continued
Conflicting Pressures
Event-Driven Communication Structures
Events Are the Basis of Communication
Event Streams Provide the Single Source of Truth
Consumers Perform Their Own Modeling and Querying
Data Communication Is Improved Across the Organization
Accessible Data Supports Business Communication Changes
Asynchronous Event-Driven Microservices
Example Team Using Event-Driven Microservices
Synchronous Microservices
Drawbacks of Synchronous Microservices
Benefits of Synchronous Microservices
Summary
2. Event-Driven Microservice Fundamentals
Building Topologies
Microservice Topology
Business Topology
The Contents of an Event
The Structure of an Event
Unkeyed Event
Entity Event
Keyed Event
Materializing State from Entity Events
Event Data Definitions and Schemas
Microservice Single Writer Principle
Powering Microservices with the Event Broker
Event Storage and Serving
Additional Factors to Consider
Event Brokers Versus Message Brokers
Consuming from the Immutable Log
Providing a Single Source of Truth
Managing Microservices at Scale
Putting Microservices into Containers
Putting Microservices into Virtual Machines
Managing Containers and Virtual Machines
Paying the Microservice Tax
Summary
3. Communication and Data Contracts
Event-Driven Data Contracts
Using Explicit Schemas as Contracts
Schema Definition Comments
Full-Featured Schema Evolution
Code Generator Support
Breaking Schema Changes
Selecting an Event Format
Designing Events
Tell the Truth, the Whole Truth, and Nothing but the Truth
Use a Singular Event Definition per Stream
Use the Narrowest Data Types
Keep Events Single-Purpose
Minimize the Size of Events
Involve Prospective Consumers in the Event Design
Avoid Events as Semaphores or Signals
Summary
4. Integrating Event-Driven Architectures with Existing Systems
What Is Data Liberation?
Compromises for Data Liberation
Converting Liberated Data to Events
Data Liberation Patterns
Data Liberation Frameworks
Liberating Data by Query
Bulk Loading
Incremental Timestamp Loading
Autoincrementing ID Loading
Custom Querying
Incremental Updating
Benefits of Query-Based Updating
Drawbacks of Query-Based Updating
Liberating Data Using Change-Data Capture Logs
Benefits of Using Data Store Logs
Drawbacks of Using Data Base Logs
Liberating Data Using Outbox Tables
Performance Considerations
Isolating Internal Data Models
Ensuring Schema Compatibility
Capturing Change-Data Using Triggers
Making Data Definition Changes to Data Sets Under Capture
Handling After-the-Fact Data Definition Changes for the Query and CDC Log Patterns
Handling Data Definition Changes for Change-Data Table Capture Patterns
Sinking Event Data to Data Stores
The Impacts of Sinking and Sourcing on a Business
Summary
5. Event-Driven Processing Basics
Composing Stateless Topologies
Transformations
Branching and Merging Streams
Repartitioning Event Streams
Example: Repartitioning an Event Stream
Copartitioning Event Streams
Example: Copartitioning an Event Stream
Assigning Partitions to a Consumer Instance
Assigning Partitions with the Partition Assignor
Assigning Copartitioned Partitions
Partition Assignment Strategies
Recovering from Stateless Processing Instance Failures
Summary
6. Deterministic Stream Processing
Determinism with Event-Driven Workflows
Timestamps
Synchronizing Distributed Timestamps
Processing with Timestamped Events
Event Scheduling and Deterministic Processing
Custom Event Schedulers
Processing Based on Event Time, Processing Time, and Ingestion Time
Timestamp Extraction by the Consumer
Request-Response Calls to External Systems
Watermarks
Watermarks in Parallel Processing
Stream Time
Stream Time in Parallel Processing
Out-of-Order and Late-Arriving Events
Late Events with Watermarks and Stream Time
Causes and Impacts of Out-of-Order Events
Time-Sensitive Functions and Windowing
Handling Late Events
Reprocessing Versus Processing in Near-Real Time
Intermittent Failures and Late Events
Producer/Event Broker Connectivity Issues
Summary and Further Reading
7. Stateful Streaming
State Stores and Materializing State from an Event Stream
Recording State to a Changelog Event Stream
Materializing State to an Internal State Store
Materializing Global State
Advantages of Using Internal State
Disadvantages of Using Internal State
Scaling and Recovery of Internal State
Materializing State to an External State Store
Advantages of External State
Drawbacks of External State
Scaling and Recovery with External State Stores
Rebuilding Versus Migrating State Stores
Rebuilding
Migrating
Transactions and Effectively Once Processing
Example: Stock Accounting Service
Effectively Once Processing with Client-Broker Transactions
Effectively Once Processing Without Client-Broker Transactions
Summary
8. Building Workflows with Microservices
The Choreography Pattern
A Simple Event-Driven Choreography Example
Creating and Modifying a Choreographed Workflow
Monitoring a Choreographed Workflow
The Orchestration Pattern
A Simple Event-Driven Orchestration Example
A Simple Direct-Call Orchestration Example
Comparing Direct-Call and Event-Driven Orchestration
Creating and Modifying an Orchestration Workflow
Monitoring the Orchestration Workflow
Distributed Transactions
Choreographed Transactions: The Saga Pattern
Orchestrated Transactions
Compensation Workflows
Summary
9. Microservices Using Function-as-a-Service
Designing Function-Based Solutions as Microservices
Ensure Strict Membership to a Bounded Context
Commit Offsets Only After Processing Has Completed
Less Is More
Choosing a FaaS Provider
Building Microservices Out of Functions
Cold Start and Warm Starts
Starting Functions with Triggers
Triggering Based on New Events: The Event-Stream Listener
Triggering Based on Consumer Group Lag
Triggering on a Schedule
Triggering Using Webhooks
Triggering on Resource Events
Performing Business Work with Functions
Maintaining State
Functions Calling Other Functions
Event-Driven Communication Pattern
Direct-Call Pattern
Termination and Shutdown
Tuning Your Functions
Allocating Sufficient Resources
Batch Event-Processing Parameters
Scaling Your FaaS Solutions
Summary
10. Basic Producer and Consumer Microservices
Where Do BPCs Work Well?
Integration with Existing and Legacy Systems
Stateful Business Logic That Isn’t Reliant Upon Event Order
When the Data Layer Does Much of the Work
Independent Scaling of the Processing and Data Layer
Hybrid BPC Applications with External Stream Processing
Example: Using an External Stream-Processing Framework to Join Event Streams
Summary
11. Heavyweight Framework Microservices
A Brief History of Heavyweight Frameworks
The Inner Workings of Heavyweight Frameworks
Benefits and Limitations
Cluster Setup Options and Execution Modes
Use a Hosted Service
Build Your Own Full Cluster
Create Clusters with CMS Integration
Application Submission Modes
Driver Mode
Cluster Mode
Handling State and Using Checkpoints
Scaling Applications and Handling Event Stream Partitions
Scaling an Application While It Is Running
Scaling an Application by Restarting It
Autoscaling Applications
Recovering from Failures
Multitenancy Considerations
Languages and Syntax
Choosing a Framework
Example: Session Windowing of Clicks and Views
Summary
12. Lightweight Framework Microservices
Benefits and Limitations
Lightweight Processing
Handling State and Using Changelogs
Scaling Applications and Recovering from Failures
Event Shuffling
State Assignment
State Replication and Hot Replicas
Choosing a Lightweight Framework
Apache Kafka Streams
Apache Samza: Embedded Mode
Languages and Syntax
Stream-Table-Table Join: Enrichment Pattern
Summary
13. Integrating Event-Driven and Request-Response Microservices
Handling External Events
Autonomously Generated Events
Reactively Generated Events
Handling Autonomously Generated Analytical Events
Integrating with Third-Party Request-Response APIs
Processing and Serving Stateful Data
Serving Real-Time Requests with Internal State Stores
Serving Real-Time Requests with External State Stores
Handling Requests Within an Event-Driven Workflow
Processing Events for User Interfaces
Micro-Frontends in Request-Response Applications
The Benefits of Microfrontends
Composition-Based Microservices
Easy Alignment to Business Requirements
Drawbacks of Microfrontends
Potentially Inconsistent UI Elements and Styling
Varying Microfrontend Performance
Example: Experience Search and Review Application
Summary
14. Supportive Tooling
Microservice-to-Team Assignment System
Event Stream Creation and Modification
Event Stream Metadata Tagging
Quotas
Schema Registry
Schema Creation and Modification Notifications
Offset Management
Permissions and Access Control Lists for Event Streams
State Management and Application Reset
Consumer Offset Lag Monitoring
Streamlined Microservice Creation Process
Container Management Controls
Cluster Creation and Management
Programmatic Bringup of Event Brokers
Programmatic Bringup of Compute Resources
Cross-Cluster Event Data Replication
Programmatic Bringup of Tooling
Dependency Tracking and Topology Visualization
Topology Example
Summary
15. Testing Event-Driven Microservices
General Testing Principles
Unit-Testing Topology Functions
Stateless Functions
Stateful Functions
Testing the Topology
Testing Schema Evolution and Compatibility
Integration Testing of Event-Driven Microservices
Local Integration Testing
Create a Temporary Environment Within the Runtime of Your Test Code
Create a Temporary Environment External to Your Test Code
Integrate Hosted Services Using Mocking and Simulator Options
Integrate Remote Services That Have No Local Options
Full Remote Integration Testing
Programmatically Create a Temporary Integration Testing Environment
Testing Using a Shared Environment
Testing Using the Production Environment
Choosing Your Full-Remote Integration Testing Strategy
Summary
16. Deploying Event-Driven Microservices
Principles of Microservice Deployment
Architectural Components of Microservice Deployment
Continuous Integration, Delivery, and Deployment Systems
Container Management Systems and Commodity Hardware
The Basic Full-Stop Deployment Pattern
The Rolling Update Pattern
The Breaking Schema Change Pattern
Eventual Migration via Two Event Streams
Synchronized Migration to the New Event Stream
The Blue-Green Deployment Pattern
Summary
17. Conclusion
Communication Layers
Business Domains and Bounded Contexts
Shareable Tools and Infrastructure
Schematized Events
Data Liberation and the Single Source of Truth
Microservices
Microservice Implementation Options
Testing
Deploying
Final Words
Index