Shahzad Bhatti Welcome to my ramblings and rants!

June 18, 2019

Review of “Designing Distributed Systems”

Filed under: Computing,Technology — admin @ 11:34 am

The “Designing Distributed Systems” book provides design patterns for building distributed systems with support of container technologies such as Kubernetes. The book consists of three sections where first section focuses on single-nodes, second section focuses on long-running services, and third section focuses on batch computation.

Sidecar

The first pattern in the book introduces concept of sidecar pattern for modularity and reusability where a single application requires two containers: application container and sidebar container where sidebar container provides additional functionality such as adding SSL proxy for the service, collecting metrics for the application container. The side-bar container can be configured via dynamic configuration service.

Ambassadors

The ambassador pattern introduces an ambassador container that sits between the application and external services and all incoming/outgoing traffic goes through it. It also helps with modularity and reusability where the ambassador may abstract sharded service (or A/B testing) so that client or service itself doesn’t need to know all details . You may also use ambassador container for service brokering where it looks up an external service and connects to it.

Anti-corruption layer

The anti-corruption layer integrates two systems that don’t share the same semantic data model.

Adapters

The adapter pattern uses special container to modify the interface of application container, e.g. you can deploy monitoring adapter to automatically collect health metrics using Prometheus or other tools. Similarly, you may use adapter container to collect kubernetes logs (stdout/stderr) and reformat the logs before sending them to log aggregator (Fluentd).

Replicated Load-Balanced Services

This pattern is part of long-running services where a load balancer is added in front of the service for scalability. Each service is designed as a stateless so that requests can be sent to any replica of the service behind the load balancer. Each service needs to provide readiness probe so that load balancer knows if it can serve the requests. In some cases, you may need to support session-tracked services where user requests are routed to the same replica using sticky session or consistent hashing function. You may add a caching layer that is deployed along with your service container (as sidebar). Further, you may need to provide rate-limiting and protect against DOS attacks (X-RateLimit-Remaining headers). This pattern can also implement SSL Termination where external traffic is encrypted with different certificate compared with internal traffic (Varnish).

Sharded Services

This pattern partitions the traffic where each shard serves subset of all requests. As opposed to replicated services that are generally used for stateless services, sharded services are used for building stateful services. You may use sharded cache for each shard that sits between user and front-end to optimize end-user performance and latency. You may add replicas for each shard for further redundancy and scalability. Sharding requires selecting a key to route the traffic, e.g. you may use IP-address or consistent hash function to avoid remapping when new shards are added. If one of the shard becomes hot, you can add replicated sharded cache to handle the increased load.

Scatter/Gather

The scatter/gather pattern adds parallelism in servicing requests where work is broken and spawned to multiple services and then result is aggregated before returning to the user. For example, you can implement distributed document search by farming multiple leaf machines that returns matching document and root node aggregates the results. You can also add support for sharded data by searching each shard in parallel and root node generates union of all documents returned by each shard (leaf node). One downside of this pattern is that it may suffer straggler problem as total response time depends on the slowest response so you may need to replicate each shard to improve computational power.

Functions and Event-Driven Processing

This pattern is used to implement function-as-a-service (FaaS) products. FaaS simplifies development and deployment as the code is managed and scaled automatically. However, FaaS requires that you decouple your application into small parts that can be run independently. Faas uses event systems to communicate with each function or create a data pipeline. You can use external data services for storing states that is shared by these functions.

Ownership Election

This pattern helps in multi-node environment where a specific task must be owned by a single process. For example, when you have multiple replicas, you may need to elect master using consensus algorithm such as Paxos, Raft or frameworks such as etcd, ZooKeeper, and consul. You can use distributed locks to implement ownership (optionally with a lease or TTL). You may need to verify if you hold the lock before proceeding, e.g.

func (Lock l) isLocked() boolean {
return l.locked && l.lockTime + 0.75 * l.ttl > now()
}

Work Queue Systems

This pattern is part of batch computation section to handle work items within a certain amount of time. You may use a work-queue manager container along with an ambassador container to connect to external queue source where source might use storage API, network storage, pub/sub systems like Kafka or Redis. Once the queue manager receives a work item, it launches a worker container. Kubernetes contains a Job object that allows for the reliable execution of the work queue. In order to limit number of worker containers running concurrently, you can limit the number of Job objects that your work queue is willing to create. You can also use the multi-worker pattern when different worker containers are transformed into a single unified container that implements the worker interface.

Event-Driven Batch Processing

This pattern allows data pipelining where an output of one work queue becomes input to another work queue, referred as workflow systems. Here are patterns of event-driven processing:
Copier: This pattern just duplicates the work item into two or more identical streams.
Filter: This pattern reduce a stream of work items to a smaller stream of work items by filtering out items that don’t meet particular criteria.
Splitter: This works like filter, but instead of eliminating input, it sends different inputs to different queues based on criteria.
Sharder: This is more generic form of splitter and splits a work item into smaller work items based on sharding function.
Merger: This is opposite of copier and merges two different work queues into a single work queue.

You may use pub/sub API to communicate between different workers.

Coordinated Batch Processing

This is similar to Reduce part of MapReduce pattern where a work is broken up and distributed to multiple nodes in parallel. You may need Join or Barrier Synchronization to wait for intermediate results before proceeding to the next stage of the workflow. For example, reduce phase aggregates merges several outputs into a single output.

Claim-check

Instead of sending large messages to a messaging queue, you can store payload to an external storage service and send reference in the messaging event.

Index Table

If the data-store doesn’t support secondary indexes, you can define a separate index table where the primary key is the secondary field for query such as customer phone number. You can use same partition-key for both fact-table and index tables so that they are colocated.

Compensating Transaction

When invoking several external service with eventual consistency model that can fail, you can use a compensating transaction to undo the operation and revert back changes.

Saga distributed transactions

The saga pattern uses a sequence of transaction to invoke multiple services, however if a step fails then it executes compensating transactions.

Scheduler Agent Supervisor

This pattern coordinates distributed actions as a transaction and it will undo the work if any action fails. It uses a scheduler to orchestrate steps of transaction, an agent to invoke a remote service and a supervisor to monitor the task being performed.

Competing Consumers

This allows multiple concurrent consumers to receive messages from same messaging queue. It allows better scalability, reliability and availability.

Command and Query Responsibility Segregation (CQRS)

CQRS separates query and update operations for a data-store such that commands store task/transaction based data in the data-store. The query model is optimized for high performance read model.

Event Sourcing

The event-sourcing allows storing state of domain object in an append-only store as events that are applied to update state of the application. This allows better performance, scalability and audit-trail, compensating actions.

Asynchronous Request-Reply

The front-end applications often work with synchronous APIs and this pattern uses request-reply facade to decouple the backend asynchronous APIs.

Retry

The retry allows recovering from failures by retrying the failed operation after exponential delay and jitter.

Circuit Breaker

When a network request fails due to transient errors, the circuit breaker can be used to prevent an application from repeatedly retrying the failed operation.

Bulkhead

The bulkhead pattern components of an application are isolated so that failure in one component doesn’t cause cascading failure. For example, you may use different database connection pools, threads pool or rate-limit for different types of requests.

Orchestration vs Choreography

Micro-services can communicate with other services using orchestration model where a centralized service orchestrates requests to other services and acknowledges all operations. Alternatively, you can use message-driven or asynchronous messaging to implement choreography design.

Deployment Stamps

In order to scale deployment by tenants or groups, you can use deployment stamp or scale-unit to host multiple instances of the application. This provides scalability, sharding and separation of data per subset of customers.

External Configuration Store

The external configuration store allows storing application configuration to a centralized location that can be cached and shared across applications.

Edge Workload Configuration

The Edge-workload configuration is often used with IoT deployment to allow scaling of data-store and configuration at the edge. For example, you may define an edge-configuration controller and edge-configuration data-store to host device configurations that are updated asynchronously from cloud configuration controller and data-store.

Federated Identity

The federated identity allows delegating authentication to an external identity provider so that user management, authentication and authorization can be simplified.

Gatekeeper

The gatekeeper acts as a broker between client and services so that requests can be sanitized, throttled and authenticated.

Gateway Aggregation

A client may need to invoke multiple data services to aggregate application state, instead gateway aggregation allows using a gateway to invoke multiple services.

Gateway Offloading

When communicating with a service that requires unique requirements for connections or security, you can use a gateway to proxy all requests so that other services don’t need to implement all those measure.

Gateway Routing

This allows client to use a single endpoint to access all service that can authenticated, throttled and validated consistently.

Geode

This allows deploying applications to geographically node or regions in active/active style so that latency and availability can be improved.

Health Endpoint Monitoring

This allows checking application health from external tools for monitoring purpose.

Leader Election

In order to avoid conflicts, a single task instance can be elected as a leader so that it coordinates actions of other subordinate tasks.

Pipes and Filters

The pipes and filters allow decomposing tasks into a series of components that can be shared to improve performance, scalability, and reusability.

Rate Limiting

The rate-limiting allows controlling resources based on limits so that you can predict throughput accurately.

Sequential Convoy

This allows processing a group of messages in sequential order by using consumers that are partitioned by the group identifier.

Valet Key

The valet key acts as a token to provide access to restricted resource. For example, the client first requests a valet key, which is then used to access the resource within the lease-time.

March 9, 2018

Separating Control/Config Management from the Services

Filed under: REST,Technology — admin @ 9:59 pm

One of pattern I learned earlier in my career was to separate control flow from the data flow. For example, when I first looked at FTP protocol, I noticed it listened to two separate ports for communication between client and server. It used a data port to transfer files and used a control port to send/receive commands for managing transfer. This allows the server to respond quickly if your data port is busy transferring large files. In some ways, this is similar to Bulkhead pattern for partitioning components and limiting the blast radius. When a service reaches its capacity, it will slow down all requests including any requests to control or configure it. Thus, it helps to define a separate channel where you can manage the control-service. Also, you may need to define special access-control policies to manage the control-service. For example, an admin may need to be on a trusted network for administration. In some cases, you may build a control-service for management behind the firewall but the data-service is publicly accessible. Another use-case is to update service’s configuration at runtime where you might store the service configuration via the control-service that can update the configuration and then publishes it to the data-service.

March 6, 2018

Tips from the second edition of “Release It!”

Filed under: Design,Methodologies,Technology — admin @ 4:19 pm

The first edition of “Release It!” has been one of most influential books that I have read and it introduced a number of methods for writing fault tolerant systems such as circuit-breaker, bulkhead patterns so I was excited to read the second edition of the book when it came out. Here are a few tips from the book:

In the first chapter, the author defines stability in terms of robustness, i.e., “A robust system keeps processing transactions, even when transient impulses, persistent stresses, or component failures disrupt normal processing.” He recommends focusing on longevity bugs such as resource leaking, e.g. you can set timeout when invoking a network or database operation or detect dead connections before reading/writing. The author cautions against tightly coupled systems that can easily propagate failures to other parts of the system. Similarly, high number of integration points can increase probability of failure in one of those dependencies. The author suggests looking into the blocking calls that can lead to deadlocks when using multiple threads or scrutinizing resource pool that gets exhausted from blocking operations. Another cause of instability is chain reaction from failure of one of servers that increases load on the remaining servers, which can be remedied using bulkhead or circuit-breaker patterns. High memory usage on the server can also constrain resources and the author recommends using external caching systems such as Redis, memcache, etc. In order to monitor health of the system, the author suggests using a mock transaction to ensure it’s working as expected and keeping metrics on errors especially login errors, high latency warnings. One common self-inflicting failure can be caused by self-denial attacks by marketing campaign that can be mitigated by using shared-nothing architecture or reducing fan-in shared resources. The servers can also be subjected to dogpile effect, where resources after upgrade, cronjob or config change spike up that can be mitigated by using random clock skew, adding random jitter and using exponential backoff. Finally, the author recommends monitoring slow responses, removing unbounded result sets and failing fast.

Here is a summary of stability patterns that the author recommends:

  • Apply timeout with integration points and delayed retries when failure occurs.
  • Apply circuit-breaker to prevent cascading failures (along with timeout).
  • Apply bulk-head pattern to partition the system in the event of chain reaction failure.
  • Steady state
  • Data purging
  • Log files
  • Fail fast, restart fast and reintegrate
  • Let it crash with limited granularity, e.g. boundary of actor.
  • Supervision for monitoring, restarts, etc.
  • Shed Load
  • Back-pressure – queues must be finite for finite response time
  • Governor – create governor to slow rate of actions (from automated response) so that humans can review it.

Next few chapters focus on network, machines and processes for building and deploying the code. The author offers several mechanisms for scaling such as load balancing with DNS, using service registry for upgrading and fail-over, configuration, transparency, collecting logs and metrics, etc. The author recommends load shedding when under high load or use HTTP 503 to notify load balancer. For example, queue length can be calculated as: (max-wait-time / mean-processing-time + 1) * processing-threads * 1.5. You can use listen reject queue to return 503 error to prevent clients from reconnecting immediately. The control-panel chapter recommends tools for administration. It recommends postmortem template such as what happened, apologize, commit to improvement and emphasizes system failures (as opposed to human errors). The author recommends adding indicators such as traffic indicators, business transaction, users, resource pool health, database connection health, data consumption, integration point health, cache health, etc.

The security chapter offers standard best practices from OWASP such as using parameterized queries to protect against SQL injection, using high entropy random session-ids/storing cookies for exchanging session-ids to protect against session hijacking/fixation. In order to protect against XSS (when user’s input is rendered in HTML without escaping) by filtering input and escaping it when rendering. The author recommends using a random nonce and strict SameSite policy to protect against CSRF. Similarly, author recommends using the principle of least privilege, access control, etc. The admin tools can offer tools for resetting circuit breakers, adjust connection pool sizes, disabling specific outbound integrations, reloading configuration, stopping accepting load, toggling feature flags.

For ease of deployment, the author recommends automation, immutable infrastructure, continuous deployment, and rolling changes incrementally.

The author suggests several recommendations on process and organization such as OODA loop for fast learning, functional/autonomous teams, evolutionary architecture, asynchrony patterns, loose clustering and creating options for future.

Lastly, the author offers chaos engineering as a way to test resilience of your system using Simian army or writing your own chaos monkey. In the end, the new edition offers a few additional chapters on scaling, deployment, security, and chaos engineering and more war stories from author’s consulting work.

January 15, 2017

Review of “Whiplash: How to Survive Our Faster Future”

Filed under: Future,Technology — admin @ 4:38 pm

I read “Whiplash: How to Survive Our Faster Future” by Jai Ito and Jeff Howe over the holidays. Joi Ito is a director of the MIT Media Lab. The MIT Media Lab was created by Nicholas Negroponte in 1985 to build an environment where best ideas from schools of arts and science can be married to build next revolutionary discoveries.

This book narrates anecdotes of how technology revolutionized human development in past and how it continues to disrupt our lives today. As a consequence of Moor’s law and the Internet, technology is changing at an exponential speed. In such rapidly changing environments, this book provides key lessons that can be used to prepare us for uncertain future and paradigm shifts. In such exponential times, the invention of new technologies far outpaces the moral and ethical consequences of those breakthroughs. As technologies can be used for both good and bad, they offer both salvage and demise of humanity.

Here are primary principles that authors present to shape the new world:

Emergence over Authority:

The invention of Internet has facilitated communication and collaboration all over the world, where best ideas can be easily shared and exchanged. As a result, institutes that had central authorities are disintegrating. The authors present several examples of Emergence vs Authority such as Blogs vs Newspapers, Wikipedia vs Encyclopedia and central governments vs social networks based political revolutions.
In emergent systems, participants use simple rules and exchange information to build complex systems. Examples of emergent systems are ant colony, slime mold, brain, flocking birds, stock exchanges, biology, etc. The authoritarian systems enable incremental changes whereas emergent systems are more adoptive and foster non-linear progress.

Pull Over Push:

Push-based systems control their access whereas pull based use transparency and two-way communication and are able to cope with the crisis far better than push based systems. The authors recited an example of pull-based when meltdown of Fukushima nuclear plant occurred as a result of severe earthquake and fourteen feet tsunami. Joi and a team of volunteers across the world collaborate and built Geiger counters to take accurate readings of radiation. Other examples of pull based systems are crowdfunding and crowdsourcing.

Compasses over Maps:

A map has detailed knowledge and an optimal route whereas compass offers more autonomy and offer more flexibility in an unpredictable environment. The authors stated examples of the education system where standardized tests and curriculum deprive students of creativity and passion for learning. They used the culture of Media Lab as an illustration where the vision is based on compass heading. It provides a framework for individual progress leaving flexibility for interactions between groups.

Risk over Safety:

Traditional businesses are more risk averse where new ventures are thoroughly analyzed and they spend millions in studies. However, the cost of experimenting new ideas has drastically been reduced in today’s market and it offers much better return on investment.

Disobedience over Compliance:

Innovation requires creativity and breaking rules so a high-impact institutes require a culture of disobedience. It needs a culture where criticism and diverse ideas are embraced.

Practice over Theory:

Due to low-cost of launching new products, an innovating organization requires a culture where experiments are valued more than detailed planning.

Diversity over Ability:

The authors provided lessons from biochemistry companies that used gamers to design protein molecules. They used gamers with diverse background and those gamers had better pattern recognition than the biochemists with PhD. Most organizations believe in diversity, but most organizations lack diversity especially in high-tech companies such as Facebook, Yahoo and Google.

Resilience over Strength:

Resilient organizations are like the immune system that can successfully recover from failures. The authors gave examples of cyber-security where there are threats from various sources and successful defense requires treating security systems as biological systems and building strong immune systems against those security risks.

Systems over Objects:

Systems over objects emphasize understanding the connections between people, communities, and the environment. Instead of optimizing an individual or an organization, we need to optimize the impact of innovations on an entire natural system.

Conclusion:

In this final chapter, authors gave examples AI and machine learning where deep learning and reinforced learning has allowed machines to beat human experts in Chess and Go. The authors cite “The Singularity is Near” by Ray Kuzweil, who predicts that we will have intelligent explosion by 2045. In this world, we will have to think about how humans and machines will work together.

February 23, 2009

Software Estimation

Filed under: Project Management,Technology — admin @ 6:04 pm

Software estimation is a difficult art that I am still learning despite developing software for more than twenty years. I have worked on a number of projects that started with some broad vision and manager asked me how many man-months will it take. You feel like a guy who is asked how long will it take you to survey a cave without going inside (see Software Estimates and the Parable of the Cave). So based on some initial requirements, you make up some numbers. But, often that number translates into commitment and some target date. This issue has been also brought up by Software Estimation by Steve McConnell, Manage It by Johanna Rothman, Lean Software Development by Mary Poppendieck and a number of other people. So it must be made clear that your estimate is not the target date.

As a project is always constrained by iron triangle of schedule/cost/functionality or sometime referred to as cost/quality/schedule or cost/resourcs/schedule. It is crucial to find what’s driving the project as also suggested by Johanna Rothman in her book Manage It. I have seen a number of cases where dates were arbitrary picked, sometime referred to as “happy date”. Though, at other times, dates may depend on marketing campaign, seasons, tax time, Olympics, etc. So, you can negotiate between functionality and schedule based on what’s driving the project. Following are some of techniques that I have found useful with estimation:

  • Get the vision and requirements straight – It’s important about the charter, constraints and requirements for the project as any misdirection here would lead to disaster. Luke Hohmann in his book Beyond Software Architecture recommends starting with good vision and mission statement. Johanna Rothman also recommends creating a project charter before starting the project.
  • Probablistic based estimation – Despite the fact, you are often pressured to produce more precise estimates even though they would be inaccurate, it is better to give estimate with some probablity. Both Johanna Rothman and Steve McConnell cite cone of uncertainty, where your estimate becomes more accurate as project progresses.

  • Based on best/worst/most-likely case – use following formula from Steve McConnell’s book can be used when estimates are more accurate:
expected_case = (best_case + (4 * most_likely) + worse_case) / 6

If estimates are not accurate, then Steve McConnell recommends

expected_case = (best_case + (3 * most_likely) + (2 * worse_case)) / 6

Bob Martin also similar formula from his article PERT, CPM, and Agile Project Management:

Mean     = (best_case + worst_case + (4 * most_likely) ) / 6

Variance = ((worst_case_best_case) / 6) ^ 2
  • Iterative development – No matter if you are working on small or large project, the only way to bring some reality and feedback on initial estimate is to develop iteratively starting with highest valued features.
  • T-shirt based estimation – I find t-shirt based estimation useful when estimating with minimal information available. For example, you may have to estimate projects that you can deliver in Q1, Q2, etc and you can order them in small, medium, large and compare them against their business value.
  • Spiking can also help in areas that are new to the team and spending a little time creating walking skeleton or tracer bullet can give you some idea on the size of the effort for the project.
  • Delphi estimation – where PM and team prepares task list, assumptions and estimate in private and reviews them together.
  • Divide and conquer/Decomposition/WBS – as with any large effort, breaking a project into smaller subsystems, components, services and tasks will help estimate better. In general any errors in estimation of smaller tasks will cancel each other.
  • Estimate fine grained tasks – I can rarely estimate with some accuracy for tasks that are longer than a few days so it’s important to estimate only fine grained tasks. XP has a concept of inch pebble and story points that can help in this case. The idea is that each task is either done or not done.
  • Planning poker a technique from Agile Estimating and Planning by Mike Cohn, where each member of the team pick an estimate for a story based on fibonacci numbers, but don’t show until everyone selects some number. The members then pick some average or may ask member with highest or lowest estimates to explain.
  • Historical data – though I rarely see PM track estimates but tracking them can help future projects and new projects can use LOC, man-months, function-points, # of services, files, interfaces, bugs from prior projects for estimation.
  • Schedule chicken – Kent Beck often talks about schedule chicken where you have some some meeting about who is ontrack and you hope there is someone who is behind so that you don’t have to admit you are behind as well. Integrity is big part of the XP and agile methodologies so it encourages transparency and honesty instead of schedule chicken.
  • Better to overestimate than underestimate – programmers often underestimate and though there is risk of student syndrome or Parkinson’s law but it’s better to overestimate.
  • Don’t question developer’s estimate – even though developers tend to underestimate, some managers still question them, which is not a good idea.
  • In XP or Scrum, you use story points, which can be ideal hours or based on some multiplier. These numbers are generally follow fibonacci sequence such as 1, 2, 3, 5, 8, 13, 21.
  • Function points use number of external input/output/queries, internal logical files/external interface files and it can be used as unit of measurements similar to story points.
  • Estimation quality factory (EQF) as proposed by Tom Demarco in his paper A Defined Process For Project Postmortem Review can be used to check how accurate estimates are.
  • Include vacation, sick, holidays as well as non-development activities such as testing, deployment, configuration, migration, etc in your project plan.
  • Scheduling is all about ordering with highest value features. I find rolling-wave scheduling based on milestones useful when planning iterations.

Summary

I often find projects turn into death march projects due to overly optimistic estimates and “queen of denial” manager who holds developers’ estimates as commitment and refuses to accept the reality. One way to overcome bad estimation is to adopt iterative development that delivers small features based on the value proposition, which creates biggest value for the business. Another way is to use advice from the Rational Unified Process that uses risk management to prioritize the highest risk tasks first. Though, some managers are keen to accept more work than the team can handle in order to aim high but it takes a courage to say NO. In the end, under-promise and over deliver as it can save your credibility and not to mention unnecessary overtime and stress on the team.

« Newer Posts

Powered by WordPress