🍒More System Design Learnings

Stateful vs. Stateless

”State” refers to the condition of a system, component, or application at a particular point in time.

1. Stateful

Not scalable and fault tolerant since the state is managed by the unique server and when the server goes down, all data is lost.

2. Stateless

State is typically stored on a separate database, accessible by all the servers. This creates a fault-tolerant and scalable architecture since web servers can be added or removed as needed, without impacting state data.

Reverse Proxy

Imagine you’re at a large event trying to order a meal from a variety of food stands. Instead of going directly to each stand to see if they have what you want, you go to a central booth (the reverse proxy). You tell the booth what you’re looking for, and they communicate with all the food stands on your behalf. Once they find a stand that can serve your request, they get the meal for you and bring it back, without you needing to know where the meal came from.

In this analogy:

• The event is the internet.

• You are a client (like a web browser).

• The food stands are different servers (web servers, application servers, etc.)

• The central booth is the reverse proxy

= A reverse proxy is a web server that centralizes internal services and provides unified interfaces to the public. Requests from clients are forwarded to a server that can fulfill it before the reverse proxy returns the server’s response to the client.

Key points and benefits

• Increased Security: The reverse proxy hides the identities of the backend servers. This is similar to how the central booth hides the specific details of each food stand. Attackers or unwanted visitors cannot directly access the servers, improving security

• SSL Termination: The reverse proxy can handle encrypting and decrypting data (SSL termination), so the individual servers behind it don’t have to. It’s as if the central booth could handle payments, so the food stands don’t have to manage money, simplifying their operations

• Serving Static Content: The reverse proxy can also directly serve static content (like images, videos, etc.), reducing the load on the backend servers. It’s as if booth had common condiments and utensils available, so you don’t have to go to a stand just for a napkin

Disadvantages

• Increased Complexity: Adding a reverse proxy introduces a new component that needs to be managed and configured, which can complicate your setup

• Single Point of Failure: If not set up with high availability in mind, the reverse proxy can become a bottleneck or a single point of failure, meaning if it goes down, the clients can’t reach the servers at all

API Gateway vs. Load Balancers in Microservices

API Gateway

• acts as a single entry point for all API requests

• provides features such as request routing, rate limiting, authentication, and API versioning

• hide the complexities of the underlying microservices from the client applications

• supports multiple protocols, such as HTTP, WebSocket, and MQTT

Load Balancer

• responsible for distributing incoming request across multiple instances of a microservice to improve availability, performance, and scalability

• helps to evenly distribute the workload across multiple instances

• ensures that each instance is utilized to its fullest potential

• only supports protocols at the transport layer, such as TCP and UDP

In other words, API Gateway provides higher-level features related to API management, while Load Balancer provides lower-level features related to traffic distribution across multiple instances of a microservice.

API Gateway in Microservices

Is one of the essential pattern used in microservices architecture that acts as a reverse proxy to route requests from clients to multiple internal services. It also provides a single entry point for all clients to interact with the system, allowing for better scalability, security, and control over the APIs

• handles common tasks such as authentication, rate limiting, and caching, while also abstracting away the compleixity of the underlying services

• by using an API Gateway, you can simplify the client-side code, reduce the number of requests that need to be made, and provide a unified interface for clients to interact with microservices

Load Balancer in Microservices

A load balancer is a component that distributes incoming network traffic across multiple servers or nodes in a server cluster

• helps to improve performance, scalability, and availability of applications and services by evenly distributing the workload among the servers

• A load balancer ensures that no single server is overloaded with traffic while others remain idle, which can lead to better resource utilization and increased reliability of the overall system

Consistent Hashing

is a technique designed to distribute data across a cluster of servers in a way that minimizes reshuffling when the cluster’s size changes (i.e., when servers are added or removed). It’s particularly useful in a distributed systems for tasks like load balancing, caching, and data partioning.

1. Hash Space as a Circle: Think of the hash space (the range of possible hash values) as a circle or “ring”

2. Placement of Servers: Each server is assigned a position on this circle based on the hash of its identifier

3. Data Assignment: To determine where data should be stored, the data key is hashed, and this hash value is used to find its place on the circle. The data then stored in the server positioned clockwise closest to this hash value

• Adding servers works well but you can have skewed distribution when losing servers. To solve this, we can do virtual servers/replicas (multiple hash functions). Number of hash functions (k) can be log(m)

  • by increasing the number of points (virtual servers/replicas) that represent a server on the hash circle, you can achieve a more even distribution of data among servers. This helps to avoid scenarios where some servers get overloaded while others remain underutilized

Code Implementation

'''consistent_hashing.py is a simple demonstration of consistent
hashing.'''

import bisect
import hashlib

class ConsistentHash:
  '''ConsistentHash(n,r) creates a consistent hash object for a 
  cluster of size n, using r replicas.

  It has three attributes. num_machines and num_replics are
  self-explanatory.  hash_tuples is a list of tuples (j,k,hash), 
  where j ranges over machine numbers (0...n-1), k ranges over 
  replicas (0...r-1), and hash is the corresponding hash value, 
  in the range [0,1).  The tuples are sorted by increasing hash 
  value.

  The class has a single instance method, get_machine(key), which
  returns the number of the machine to which key should be 
  mapped.'''

  def __init__(self,num_machines=1,num_replicas=1):
    self.num_machines = num_machines
    self.num_replicas = num_replicas
    hash_tuples = [(j,k,my_hash(str(j)+"_"+str(k))) \
                   for j in range(self.num_machines) \
                   for k in range(self.num_replicas)]
    # Sort the hash tuples based on just the hash values
    hash_tuples.sort(lambda x,y: cmp(x[2],y[2]))
    self.hash_tuples = hash_tuples

  def get_machine(self,key):
    '''Returns the number of the machine which key gets sent to.'''
    h = my_hash(key)
    # edge case where we cycle past hash value of 1 and back to 0.
    if h > self.hash_tuples[-1][2]: return self.hash_tuples[0][0]
    hash_values = map(lambda x: x[2],self.hash_tuples)
    index = bisect.bisect_left(hash_values,h)
    return self.hash_tuples[index][0]

def my_hash(key):
  '''my_hash(key) returns a hash in the range [0,1).'''
  return (int(hashlib.md5(key).hexdigest(),16) % 1000000)/1000000.0

def main():
  ch = ConsistentHash(7,3)
  print "Format:"
  print "(machine,replica,hash value):"
  for (j,k,h) in ch.hash_tuples: print "(%s,%s,%s)" % (j,k,h)
  while True:
    print "\nPlease enter a key:"
    key = raw_input()
    print "\nKey %s maps to hash %s, and so to machine %s" \
        % (key,my_hash(key),ch.get_machine(key))

if __name__ == "__main__": main()

Message Queue

Servers are processing jobs in parallel. A server can crash and the jobs running on the crashed server still needs to get processed.

A notifier constantly polls the status of each server and if a server crashes it ttakes all unfinished jobs (listed in some database) and distributes it to the rest of the servers. Because distribution uses a load balancer (with consistent hashing) duplicate processing will not occur as job_1, which might be processing on server_3 (alive) will land agaon on server_3, and so on. This ”notifier with load balancing” is a “Message Queue”.