🍒More System Design Learnings
Stateful vs. Stateless
”State” refers to the condition of a system, component, or application at a particular point in time.
1. Stateful
Not scalable and fault tolerant since the state is managed by the unique server and when the server goes down, all data is lost.
2. Stateless
State is typically stored on a separate database, accessible by all the servers. This creates a fault-tolerant and scalable architecture since web servers can be added or removed as needed, without impacting state data.
Reverse Proxy
Imagine you’re at a large event trying to order a meal from a variety of food stands. Instead of going directly to each stand to see if they have what you want, you go to a central booth (the reverse proxy). You tell the booth what you’re looking for, and they communicate with all the food stands on your behalf. Once they find a stand that can serve your request, they get the meal for you and bring it back, without you needing to know where the meal came from.
In this analogy:
The event is the internet.
You are a client (like a web browser).
The food stands are different servers (web servers, application servers, etc.)
The central booth is the reverse proxy
= A reverse proxy is a web server that centralizes internal services and provides unified interfaces to the public. Requests from clients are forwarded to a server that can fulfill it before the reverse proxy returns the server’s response to the client.
Key points and benefits
Increased Security: The reverse proxy hides the identities of the backend servers. This is similar to how the central booth hides the specific details of each food stand. Attackers or unwanted visitors cannot directly access the servers, improving security
SSL Termination: The reverse proxy can handle encrypting and decrypting data (SSL termination), so the individual servers behind it don’t have to. It’s as if the central booth could handle payments, so the food stands don’t have to manage money, simplifying their operations
Serving Static Content: The reverse proxy can also directly serve static content (like images, videos, etc.), reducing the load on the backend servers. It’s as if booth had common condiments and utensils available, so you don’t have to go to a stand just for a napkin
Disadvantages
Increased Complexity: Adding a reverse proxy introduces a new component that needs to be managed and configured, which can complicate your setup
Single Point of Failure: If not set up with high availability in mind, the reverse proxy can become a bottleneck or a single point of failure, meaning if it goes down, the clients can’t reach the servers at all
API Gateway vs. Load Balancers in Microservices
API Gateway
acts as a single entry point for all API requests
provides features such as request routing, rate limiting, authentication, and API versioning
hide the complexities of the underlying microservices from the client applications
supports multiple protocols, such as HTTP, WebSocket, and MQTT
Load Balancer
responsible for distributing incoming request across multiple instances of a microservice to improve availability, performance, and scalability
helps to evenly distribute the workload across multiple instances
ensures that each instance is utilized to its fullest potential
only supports protocols at the transport layer, such as TCP and UDP
In other words, API Gateway provides higher-level features related to API management, while Load Balancer provides lower-level features related to traffic distribution across multiple instances of a microservice.
API Gateway in Microservices
Is one of the essential pattern used in microservices architecture that acts as a reverse proxy to route requests from clients to multiple internal services. It also provides a single entry point for all clients to interact with the system, allowing for better scalability, security, and control over the APIs
handles common tasks such as authentication, rate limiting, and caching, while also abstracting away the compleixity of the underlying services
by using an API Gateway, you can simplify the client-side code, reduce the number of requests that need to be made, and provide a unified interface for clients to interact with microservices
Load Balancer in Microservices
A load balancer is a component that distributes incoming network traffic across multiple servers or nodes in a server cluster
helps to improve performance, scalability, and availability of applications and services by evenly distributing the workload among the servers
A load balancer ensures that no single server is overloaded with traffic while others remain idle, which can lead to better resource utilization and increased reliability of the overall system
Consistent Hashing
is a technique designed to distribute data across a cluster of servers in a way that minimizes reshuffling when the cluster’s size changes (i.e., when servers are added or removed). It’s particularly useful in a distributed systems for tasks like load balancing, caching, and data partioning.
Hash Space as a Circle: Think of the hash space (the range of possible hash values) as a circle or “ring”
Placement of Servers: Each server is assigned a position on this circle based on the hash of its identifier
Data Assignment: To determine where data should be stored, the data key is hashed, and this hash value is used to find its place on the circle. The data then stored in the server positioned clockwise closest to this hash value
Adding servers works well but you can have skewed distribution when losing servers. To solve this, we can do virtual servers/replicas (multiple hash functions). Number of hash functions (k) can be log(m)
by increasing the number of points (virtual servers/replicas) that represent a server on the hash circle, you can achieve a more even distribution of data among servers. This helps to avoid scenarios where some servers get overloaded while others remain underutilized
Code Implementation
'''consistent_hashing.py is a simple demonstration of consistent
hashing.'''
import bisect
import hashlib
class ConsistentHash:
'''ConsistentHash(n,r) creates a consistent hash object for a
cluster of size n, using r replicas.
It has three attributes. num_machines and num_replics are
self-explanatory. hash_tuples is a list of tuples (j,k,hash),
where j ranges over machine numbers (0...n-1), k ranges over
replicas (0...r-1), and hash is the corresponding hash value,
in the range [0,1). The tuples are sorted by increasing hash
value.
The class has a single instance method, get_machine(key), which
returns the number of the machine to which key should be
mapped.'''
def __init__(self,num_machines=1,num_replicas=1):
self.num_machines = num_machines
self.num_replicas = num_replicas
hash_tuples = [(j,k,my_hash(str(j)+"_"+str(k))) \
for j in range(self.num_machines) \
for k in range(self.num_replicas)]
# Sort the hash tuples based on just the hash values
hash_tuples.sort(lambda x,y: cmp(x[2],y[2]))
self.hash_tuples = hash_tuples
def get_machine(self,key):
'''Returns the number of the machine which key gets sent to.'''
h = my_hash(key)
# edge case where we cycle past hash value of 1 and back to 0.
if h > self.hash_tuples[-1][2]: return self.hash_tuples[0][0]
hash_values = map(lambda x: x[2],self.hash_tuples)
index = bisect.bisect_left(hash_values,h)
return self.hash_tuples[index][0]
def my_hash(key):
'''my_hash(key) returns a hash in the range [0,1).'''
return (int(hashlib.md5(key).hexdigest(),16) % 1000000)/1000000.0
def main():
ch = ConsistentHash(7,3)
print "Format:"
print "(machine,replica,hash value):"
for (j,k,h) in ch.hash_tuples: print "(%s,%s,%s)" % (j,k,h)
while True:
print "\nPlease enter a key:"
key = raw_input()
print "\nKey %s maps to hash %s, and so to machine %s" \
% (key,my_hash(key),ch.get_machine(key))
if __name__ == "__main__": main()Message Queue
Servers are processing jobs in parallel. A server can crash and the jobs running on the crashed server still needs to get processed.
A notifier constantly polls the status of each server and if a server crashes it ttakes all unfinished jobs (listed in some database) and distributes it to the rest of the servers. Because distribution uses a load balancer (with consistent hashing) duplicate processing will not occur as job_1, which might be processing on server_3 (alive) will land agaon on server_3, and so on. This ”notifier with load balancing” is a “Message Queue”.
Last updated