Designing a Load Balancer from First Principles

The Load Balancer's Job

Distribute incoming requests across backend servers. Sounds simple. Production load balancing is not.

Algorithm Selection

Algorithm	Best For	Avoid When
Round Robin	Even workloads	Variable request cost
Least Connections	Long-lived connections	Connection pooling
IP Hash	Session affinity	Dynamic scaling
Consistent Hashing	Cache servers	Backend failures

Consistent Hashing

Problem: Adding a server when using simple hash % N remaps most keys.

Consistent hashing maps both servers and keys to a ring:

         hash=0
          │
    0.75 ─┼─ server-C
          │
    0.50 ─┼─ key-1 ──> server-B
          │
    0.25 ─┼─ server-A
          │          key-2 ──> server-C
   -0.00 ─┴─ server-B

Adding server-D only affects keys between C and D.

Health Checks

Passive: Mark unhealthy on 5xx Active: Periodic HTTP checks

hljs yaml

[object Object],
  ,[object Object], ,[object Object],
  ,[object Object], ,[object Object],
  ,[object Object], ,[object Object],
  ,[object Object], ,[object Object],
  ,[object Object], ,[object Object],

Failing health checks should trigger gradual drain, not instant removal.

Circuit Breaker

When downstream is failing, stop sending traffic:

hljs python

[object Object], ,[object Object],:
    CLOSED = ,[object Object],      ,[object Object],
    OPEN = ,[object Object],          ,[object Object],
    HALF_OPEN = ,[object Object], ,[object Object],

    ,[object Object], ,[object Object],(,[object Object],):
        ,[object Object], ,[object Object],.state == OPEN:
            ,[object Object], time_since_open > timeout:
                ,[object Object],.state = HALF_OPEN
            ,[object Object],:
                ,[object Object], CircuitOpen()

        ,[object Object],:
            result = fn()
            ,[object Object],.on_success()
            ,[object Object], result
        ,[object Object], Exception ,[object Object], e:
            ,[object Object],.on_failure()
            ,[object Object], e

The Complete Picture

Client
  │
  ▼┌────────────────────────────────────┐
   │ DNS Load Balancer (Geo-distributed)│
   └────────────────────────────────────┘
  │
  ▼┌────────────────────────────────────┐
   │ L4 Load Balancer (TCP/UDP)         │
   │ - Consistent hashing               │
   │ - Connection draining              │
   └────────────────────────────────────┘
  │
  ▼┌────────────────────────────────────┐
   │ L7 Load Balancer (HTTP)            │
   │ - Path-based routing               │
   │ - Rate limiting                    │
   │ - SSL termination                  │
   └────────────────────────────────────┘
  │
  ▼
Backend Pool

L4 for performance, L7 for intelligence.