Imagine you have built a web application.

At the beginning, everything works perfectly.

A few hundred users are accessing your application, and a single server is more than enough to handle the traffic.

But then the application starts growing.

More users arrive.

More requests are generated.

More data needs to be processed.

Eventually, you reach a point where one server is no longer enough.

This creates an important System Design question:

What happens when one server cannot handle all the traffic?

One common solution is to add multiple servers and distribute incoming traffic between them.

But that creates another question:

Who decides which server should handle each request?

That's where a Load Balancer comes in.

1. The Problem: One Server Is Not Enough

When an application is small, we might start with a simple architecture where users send requests to one application server.

This architecture is easy to build and understand.

But as traffic increases, the server has to handle more and more work.

For example, an e-commerce application may need to handle:

• Product searches

• User authentication

• Product details

• Cart operations

• Order creation

• Payment requests

• Database operations

• Business logic

If thousands of users send requests at the same time, the server can become overloaded.

This can result in:

• Higher CPU and memory usage

• Increased response time

• Request failures

• Poor user experience

• Application downtime

There is another major problem.

If the only server fails, the entire application can become unavailable.

This creates what we call a Single Point of Failure (SPOF).

2. The Solution: Load Balancing

Instead of depending on a single server, we can run multiple application servers.

But we need something to distribute incoming traffic between them.

That's the role of a Load Balancer.

A Load Balancer sits between clients and backend servers.

It receives incoming requests and forwards them to appropriate backend instances.

The basic idea is:

Users → Load Balancer → Multiple Application Servers

Now, instead of one server handling all incoming traffic, multiple servers can share the workload.

This provides a foundation for horizontal scaling.

3. What Is a Load Balancer?

A Load Balancer is a component that receives incoming traffic and distributes requests across multiple backend servers according to a defined strategy.

Its primary purpose is to prevent traffic from being concentrated on a single server.

But a Load Balancer can do more than simply distribute requests.

Depending on the architecture and technology being used, it can also help with:

• Server health checks

• Traffic management

• High availability

• TLS termination

• Connection management

• Routing decisions

• Monitoring

The exact responsibilities depend on the implementation.

The important concept is:

A Load Balancer creates a controlled entry point for traffic going toward multiple backend instances.

4. How Does a Load Balancer Work?

Let's follow a simple request.

A user opens an application and sends a request.

The request first reaches the Load Balancer.

The Load Balancer examines the available backend servers and selects one according to its configured routing strategy.

The request is then forwarded to that server.

The server processes the request and generates a response.

The response is returned to the client.

The simplified flow is:

Client → Load Balancer → Application Server → Response

The client generally doesn't need to know which specific backend server processed the request.

This abstraction allows backend servers to be added, removed, or replaced without requiring the client to know about those internal changes.

5. How Does the Load Balancer Choose a Server?

A Load Balancer needs a way to decide where each request should go.

Different load-balancing algorithms can be used depending on the workload and system requirements.

There is no single algorithm that is perfect for every application.

Round Robin

Round Robin distributes requests sequentially across available servers.

For example:

Request 1 → Server 1

Request 2 → Server 2

Request 3 → Server 3

Request 4 → Server 1

Request 5 → Server 2

This approach is simple and can work well when servers have similar capacity and requests have relatively similar workloads.

Least Connections

With Least Connections, the Load Balancer considers the number of active connections handled by each server.

Suppose:

• Server 1 → 80 active connections

• Server 2 → 25 active connections

• Server 3 → 40 active connections

A new connection could be directed toward Server 2 because it currently has fewer active connections.

This can be useful when requests or connections have different processing durations.

Weighted Routing

Sometimes all servers do not have the same capacity.

For example:

• Server 1 → 2 CPU cores

• Server 2 → 4 CPU cores

• Server 3 → 8 CPU cores

A weighted strategy can assign more traffic to more capable servers.

This allows the system to account for differences in server capacity.

IP Hashing

IP Hashing uses information such as the client's IP address to determine the backend server.

This can provide a more consistent mapping between a client and a backend server.

However, the usefulness of this approach depends on the application's requirements and network conditions.

The important lesson is:

Load-balancing strategy should be selected according to the workload and architecture, not simply because one algorithm is popular.

6. What Happens When a Server Fails?

This is one of the most important reasons Load Balancers are used.

Suppose we have three application servers.

Server 1 is healthy.

Server 2 becomes unhealthy.

Server 3 is healthy.

If the Load Balancer continues sending requests to Server 2, users may receive errors.

A properly configured Load Balancer can use health checks to determine whether backend servers are healthy.

Depending on the system, health checks can involve:

• HTTP requests

• TCP checks

• Application-specific health endpoints

For example, an application may expose an endpoint such as:

/health

The Load Balancer can periodically check this endpoint.

If the server is healthy, it can remain in the active pool.

If the server becomes unhealthy, the Load Balancer can stop sending new traffic to that instance.

Traffic can then continue toward healthy servers.

This improves the system's availability and fault tolerance.

7. Load Balancing and Horizontal Scaling

Load Balancing becomes especially useful when combined with horizontal scaling.

Instead of continuously making one server more powerful, we can add additional application instances.

For example, an application might start with one server.

As traffic increases, additional servers can be added.

The Load Balancer then distributes incoming requests across the available instances.

This creates a scalable architecture where the application layer can grow according to traffic requirements.

This is an important difference between:

Vertical Scaling

Making one server more powerful.

and

Horizontal Scaling

Adding more servers and distributing traffic between them.

Load Balancers are commonly associated with horizontal scaling because they provide a mechanism for distributing traffic across multiple application instances.

8. Real-World Example: E-Commerce

Consider an e-commerce platform during a major sale.

Thousands or even millions of users may simultaneously:

• Browse products

• Search for products

• Add products to carts

• Create orders

• Check order status

• Make payments

If all traffic is sent to one server, that server may become overloaded.

Instead, the application can run multiple backend instances.

The Load Balancer distributes incoming traffic across those instances.

A simplified architecture would be:

Users → Load Balancer → Application Servers → Database

Now, instead of one application server processing every request, multiple servers can work together.

If traffic increases further, additional application instances can potentially be added.

This is one of the fundamental patterns used when designing scalable backend systems.

9. Does a Load Balancer Solve Everything?

No.

This is an important System Design lesson.

Adding a Load Balancer does not automatically make an application highly available.

We also need to consider what happens if the Load Balancer itself fails.

Imagine an architecture with:

Users → One Load Balancer → Multiple Servers

The backend servers may all be healthy.

But if the only Load Balancer becomes unavailable, users may still lose access to the application.

The Load Balancer has now become another Single Point of Failure.

This is why production systems need to consider redundancy and failure handling for the traffic-distribution layer as well.

Depending on the infrastructure, multiple load-balancing instances or managed highly available load-balancing services can be used.

The broader lesson is:

Removing one Single Point of Failure should not accidentally create another one.

10. Load Balancer vs API Gateway

Load Balancers and API Gateways are often discussed together, but they are not exactly the same thing.

A Load Balancer primarily focuses on distributing traffic across backend instances.

An API Gateway can handle API-level concerns such as:

• Authentication

• Authorization

• Rate limiting

• API routing

• API policies

• Request transformation

• Response transformation

They can also exist together.

For example:

Client → API Gateway → Load Balancer → Application Servers

In this architecture, the API Gateway can handle API-level policies while the Load Balancer distributes traffic across available backend instances.

However, the exact architecture depends on the system requirements and infrastructure.

The important thing is to understand the responsibility of each component instead of adding components simply because they are common in architecture diagrams.

11. Important Design Considerations

When designing a Load Balancing layer, several questions need to be considered.

Which algorithm should we use?

Should we use:

• Round Robin?

• Least Connections?

• Weighted Routing?

• IP Hashing?

The answer depends on the workload.

How do we detect failures?

Health checks are important for identifying unhealthy backend instances.

Can the Load Balancer itself fail?

If yes, redundancy may be required.

Does the application require session persistence?

If user sessions depend on a particular server, routing decisions become more important.

Can the system scale automatically?

In cloud environments, Load Balancers are often combined with auto-scaling mechanisms.

How do we monitor the system?

Useful metrics can include:

• Request rate

• Response latency

• Error rate

• Active connections

• Backend health

• Traffic distribution

These considerations are what turn a basic architecture into a production-oriented architecture.

12. The Bigger System Design Picture

A Load Balancer is usually one component of a much larger architecture.

A simplified web architecture might contain:

Client → CDN → API Gateway → Load Balancer → Application Servers → Cache → Database

Each component solves a different problem.

CDN

Helps deliver static or cacheable content closer to users.

API Gateway

Handles API-level concerns such as authentication, authorization, routing, and rate limiting.

Load Balancer

Distributes traffic across backend instances.

Application Servers

Execute business logic and process requests.

Cache

Can reduce repeated access to backend data stores and improve response time.

Database

Stores persistent application data.

The important System Design skill is understanding why each component exists and what problem it solves.

13. Benefits of Load Balancing

A properly designed Load Balancing layer can provide several important benefits.

Scalability

Traffic can be distributed across multiple backend instances.

Availability

If one backend instance fails, healthy instances can continue serving traffic.

Fault Tolerance

The system can continue operating despite individual server failures.

Performance

Traffic can be distributed instead of overwhelming one machine.

Flexibility

Different routing strategies can be selected according to system requirements.

Maintainability

Traffic can potentially be moved away from instances during maintenance or deployments.

However, these benefits depend on correct configuration and the overall architecture.

A Load Balancer is only one part of a reliable system.

14. The Most Important System Design Lesson

When learning System Design, it's easy to memorize architecture diagrams.

For example:

Client → Load Balancer → Servers → Database

But memorizing the diagram is not enough.

The more important question is:

Why is each component there?

For a Load Balancer, we should be able to answer:

Why do we need it?

To distribute incoming traffic across backend instances.

What happens if a server fails?

Health checks can identify unhealthy instances and traffic can be redirected toward healthy instances.

What happens when traffic increases?

Additional backend instances can potentially be added.

What happens if the Load Balancer fails?

Redundancy and fault-tolerance mechanisms need to be considered.

How do we know which server is healthy?

Through health checks and monitoring.

This way of thinking is much more valuable than simply memorizing definitions.

15. Key Takeaway

A Load Balancer helps transform a system from:

One server handling everything

into:

Multiple servers working together behind a traffic-distribution layer.

It can play an important role in:

Scalability

Availability

Performance

Fault Tolerance

But a Load Balancer by itself does not automatically make a system scalable or highly available.

The entire architecture needs to be designed with:

• Failure handling

• Health checks

• Redundancy

• Monitoring

• Appropriate routing

• Capacity planning

in mind.

Final Thought

A system doesn't become reliable simply because we add more servers.

The real engineering challenge is making those servers work together, handling failures gracefully, and continuing to serve users as traffic grows.

That's where Load Balancing becomes an important building block of modern System Design.

The next time you see:

Client → Load Balancer → Multiple Servers

don't just recognize the diagram.

Ask yourself:

What problem is the Load Balancer solving?

What happens when a server fails?

What happens when traffic suddenly increases?

What happens when the Load Balancer itself fails?

Those questions are where real System Design thinking begins.

🧠 Quick Check

You have three application servers:

• Server A → Healthy

• Server B → Unhealthy

• Server C → Healthy

What should a properly configured Load Balancer do?

A) Continue sending traffic to Server B
B) Stop sending new traffic to Server B
C) Shut down the entire application
D) Send every request to Server A

Answer: B — Stop sending new traffic to the unhealthy server and route requests toward healthy instances.