Four protocols, four different questions

These technologies do not all solve the same problem. Each one asks a different question about how data should move.

TCP vs Homa at a glance

TCP is a general-purpose, connection-oriented byte-stream protocol. Homa is an RPC-oriented transport built for very low latency inside data centers.

The biggest difference: who controls the send button

In TCP, the sender decides when to transmit, limited only by its congestion window. When many senders target one server, they all decide to send at once, and queues build. In Homa, a sender can push a small first slice of an RPC without asking. After that, the receiver hands out grants that decide which sender goes next.

Application
   ↓
Byte stream
   ↓
TCP connection
   ↓
Congestion window
   ↓
Network

Application RPC
   ↓
Message
   ↓
Initial unscheduled packets
   ↓
Receiver grants transmission
   ↓
Priority scheduling
   ↓
Network

A worked example

A server receives three RPCs at the same time: A = 2 KB, B = 20 KB, and C = 5 MB. With TCP, packets from all three compete in the same queues, so the tiny RPC can sit behind packets from the 5 MB transfer. Homa puts the short ones first:

Queue: [C][C][B][C][A][C][C][B][C]...
        A waits behind C's packets

2 KB  → 20 KB → rest of 5 MB
 A ✓     B ✓     C ...

That is why Homa focuses on tail latency (p99, p99.9) rather than average throughput. Try it yourself in the simulator.

Why TCP struggles inside a data center

TCP sees bytes, not messages. If an application sends RPC A (2 KB), RPC B (5 MB), and RPC C (4 KB), TCP sees one long run of bytes from 0 to about 5,006,000. It has no idea that C is tiny and latency-sensitive while B is huge. In microservice architectures, that blind spot is expensive.

Incast: many senders, one receiver

A front-end service fans a request out to 100 backends. Each replies with 64 KB at roughly the same moment: 100 × 64 KB = 6.4 MB converging on one receiver. The senders together can push hundreds of gigabits, but the receiver link may be only 100 Gbps. Packets pile up in the top-of-rack or leaf switch.

             Request
                │
       ┌────────▼────────┐
       │  Front-end API  │
       └────────┬────────┘
       fan out to 100 servers
  ┌─────────────┼─────────────┐
  ▼             ▼             ▼
Server 1     Server 2 ... Server 100
  └─────────────┼─────────────┘
                ▼
     6.4 MB lands at once

Servers
  │ │ │ │ │ │ │ │
  ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
┌──────────────────┐
│ Switch queue     │
│ ███████████████  │ ← growing
└────────┬─────────┘
      100 Gbps
         ▼
      Receiver

DCTCP: earlier, gentler warnings

Microsoft Research built Data Center TCP around this exact problem. The team measured a 6,000-server production cluster and found that bandwidth-heavy background flows were building queues in switches and hurting latency-sensitive foreground traffic.

DCTCP keeps everything familiar about TCP: connections, sequence numbers, ACKs, byte-stream semantics, and congestion windows. What changes is the feedback. Instead of waiting for a full buffer and a dropped packet, switches use ECN (Explicit Congestion Notification) to mark packets once the queue passes a threshold. The sender then measures how much of its traffic was marked and cuts its window in proportion, instead of treating congestion as a yes/no event.

Queue
███████████████████████
                      X packet loss
                        ↓
                    slow down hard

Queue
██████
     ↑ ECN mark at threshold K
       ↓
   slow down in proportion

The limit: DCTCP is still reactive. Congestion starts, the network marks it, and the sender responds. It shrinks queues; it does not schedule them.

Homa: schedule the receiver’s link

Homa starts from a simple observation: most data-center traffic is RPCs, so why model it as byte streams? Instead of connection → stream → bytes, Homa works with RPC → message → packets. Because it knows where each message starts and ends, it knows how much of each one is left to send.

Shortest Remaining Processing Time (SRPT)

Homa approximates SRPT. Rather than splitting bandwidth equally, it gets short messages out of the way first.

RPC A ████
RPC B ████
RPC C ████
RPC D ████
(everyone progresses slowly)

2 KB   ████ ✓
4 KB   ██████ ✓
20 KB  ████████████ ✓
5 MB   ████████████████████...

Unscheduled and scheduled packets

A sender may transmit the first part of a message right away, without permission. These are unscheduled packets, and they let short messages finish in one shot. The rest of a larger message moves as scheduled packets, sent only when the receiver issues a grant.

Sender A ──┐
Sender B ──┤
Sender C ──┤
Sender D ──┼──► Receiver
Sender E ──┤
Sender F ──┘
Potential incast.

Sender A ─────┐
Sender B ─────┤
Sender C ─────┼──── Receiver
Sender D ─────┘        │
               remaining bytes:
               A: 2 KB  B: 5 MB  C: 8 KB
              ┌────────┼─────────┐
              ▼        ▼         ▼
           grant A  grant C   wait B

Why receiver control works

A receiver with a 100 Gbps NIC can only ever take in 100 Gbps. Under TCP, 50 machines might each try to send toward it at line rate, and the network has to absorb the mismatch. Homa lets the receiver answer one question: who gets my downlink next? The bottleneck is scheduled, not just reacted to.

RDMA, RoCE, and InfiniBand

Homa and DCTCP deal with network congestion. RDMA deals with CPU overhead. At 100, 200, or 400+ Gbps, pushing millions of packets through the kernel stack burns a lot of CPU.

NIC
 ↓
interrupt / polling
 ↓
kernel network stack
 ↓
TCP processing
 ↓
socket buffer
 ↓
copy
 ↓
application

Application memory
       │
       ▼
      NIC
       │
=======NETWORK=======
       │
      NIC
       │
       ▼
Remote application memory

The NIC places data straight into registered application memory with very little CPU involvement. The result is very low latency, very high throughput, and low CPU use. That is why RDMA dominates distributed AI training, MPI/HPC, high-performance storage, and GPU-to-GPU communication.

RoCEv2: RDMA over Ethernet

Application
    │
 RDMA verbs
    │
  RDMA NIC
    │
  RoCEv2
    │
 UDP / IP
    │
 Ethernet

PFC  Priority Flow Control
ETS  Enhanced Transmission Selection
ECN  Explicit Congestion Notification

Goal: make packet loss extremely rare

RDMA expects a network where loss almost never happens, and Ethernet does not promise that on its own. Microsoft’s Azure Local guidance calls for careful DCB configuration, including PFC and ETS, when using RoCE. Getting a RoCE fabric right takes real network engineering work.

Two attacks on two bottlenecks

                 Network congestion
                        ▲
                        │
                 Homa attacks this
                        │
TCP ────────────────────┼────────────
                        │
                 RDMA attacks this
                        │
                        ▼
                   CPU overhead

The two ideas can coexist. The Homa Linux research found that once network congestion was under control, software processing overhead became the main limit, and the authors argued that future gains would need transport processing moved into NIC hardware.

How they compare

Stars are conceptual ratings, not benchmark numbers.

Homa’s benchmark, in context

Queue simulator

Three RPCs arrive at one receiver at the same moment. Compare a FIFO switch queue with fair sharing (TCP-like) against priority scheduling with shortest-first ordering (Homa-like).

Where this lands in Azure

Different Azure scenarios sit in different parts of this picture. That distinction is easy to miss.

App Service
     ↓
AKS
     ↓
Redis
     ↓
Azure SQL

= TCP/IP world
+ Accelerated Networking

GPU
 │ matrix operation
 │ AllReduce
 │ network  ← stall here = idle GPUs
 │
GPU
 │ next operation

= RDMA over InfiniBand

Ordinary Azure applications: TCP/IP

For web apps, APIs, and databases, you are in TCP/IP. The levers are connection reuse, HTTP/2 or HTTP/3, good placement (for example proximity placement groups), Private Link, and right-sized VMs and NICs. Accelerated Networking enables SR-IOV so traffic goes from the NIC straight to the VM, bypassing the host and virtual switch while network policies are enforced in hardware. This lowers latency, jitter, and CPU use. For best results, enable it on at least two VMs in the same virtual network; it has little effect on latency across virtual networks or to on-premises. For most Azure architectures, this matters far more than Homa.

Azure HPC and AI: RDMA over InfiniBand

Most Azure HPC VM sizes, plus selected N-series sizes marked with “r”, add a second network interface for RDMA over InfiniBand alongside the standard Ethernet NIC. Almost all newer RDMA-capable sizes are SR-IOV enabled. RDMA runs only over the InfiniBand network, not over Ethernet. To use RDMA between VMs, place them in the same virtual machine scale set or availability set, and keep your virtual network address space clear of 172.16.0.0/16, which the RDMA network reserves.

Azure Local: RoCE and iWARP

On Azure Local (on-premises, Azure-connected infrastructure), Microsoft supports RDMA through both RoCE and iWARP. RoCE needs the DCB settings described above.

Quick decision guide

Template: explaining this to a partner

Which transport fits your workload?

Answer the questions on the right. The flowchart lights up the path you take and ends at the recommendation from the decision guide above.

Break the problem down, branch by branch

Every technology on this page attacks one of two bottlenecks. Click any node to expand or collapse it.

A Homa exchange, step by step

Two senders target one receiver: A has a 2 KB RPC, C has a 5 MB RPC. Press play to watch unscheduled packets, grants, and scheduled packets in order.

The matrix as a shape

The same star ratings from section 07, plotted on eight axes. Toggle each protocol to compare profiles.

Where each one changes the stack

Gradient layers show what each technology replaces or adds. Striped layers are skipped.

Frequently asked questions