Network: Part 6. Transport Layer

The era of adaptive systems

In September 1981, Jon Postel released RFC 793 — the specification of the Transmission Control Protocol (TCP), on which work had been going on for almost ten years by that point. The document described a mechanism that was supposed to provide "reliable stream transmission of data between a pair of processes in heterogeneous computer networks," and it did not contain a single word about the web, streaming, clouds, or mobile networks — none of this existed yet. Forty-five years later, TCP carries practically all of the planet's textual and a significant part of its media information: every page opening, every purchase, every bank transfer begins with the three handshake packets invented in an era when the entire Internet consisted of several hundred hosts.

The paradox of the transport layer is that it builds order out of chaos. The network layer beneath it — IP — guarantees nothing: neither delivery, nor order, nor the absence of duplicates. And on top of this "mail that loses letters," the transport layer erects a reliable byte stream over which applications communicate as if an ideal wire were laid between them. Moreover, it is precisely the transport layer — not the routers — that today controls the speed of the entire Internet: the congestion control algorithms in the TCP stacks of billions of devices collectively decide who occupies the shared channels and at what speed. The global network has no central regulator — its role is played in a decentralized manner by the window algorithms invented by Van Jacobson in 1988.

This architecture gives rise to a series of critical questions:

1. The architectural question. Why did the layer end up with exactly two "official" protocols — the heavyweight reliable TCP and the minimalist UDP — and why did the second become a "construction set" from which the industry assembled RTP, DTLS, QUIC, and WebRTC? What is lost and what is gained in this asymmetry?
2. The reliability question. How do you assemble an ordered stream without a single loss out of a network that loses packets? The answer is the sliding window, acknowledgments, and timers — but the task also has a fundamental side: the "two generals problem" proves that perfect reliability is in principle unattainable, and all the engineering of the layer is work with this limit.
3. The speed question. The throughput of a connection equals the window divided by the round-trip time (RTT). The sixteen-bit window of the original TCP limited the flow to 64 kilobytes "in flight" — with modern RTT and gigabit channels, this would not have been enough already in the nineties. The story of the "long fat pipes" is the story of fighting the protocol's own header.
4. The question of congestion and fairness. TCP treats packet loss as a congestion signal and cuts the speed in half; from this simple rule is born the global fairness of channel sharing — and a mass of pathologies: unfairness to distant connections, the "sawtooth" of throughput, blindness to buffering (bufferbloat), the wars of the CUBIC and BBR algorithms.
5. The evolution question. TCP turned out to be a hostage of its own success: it lives in the kernels of the operating systems of billions of devices, and the intermediate equipment has "ossified" around its headers in such a way that new protocol options are almost impossible to deploy. The answer became QUIC — a transport in user space, on top of UDP, with fully encrypted headers — which in ten years traveled the path from a Google experiment to a standard on which more than 40% of websites run.
6. The security question. The TCP handshake is an attack surface: SYN flood exhausts the queues of servers, the predictability of sequence numbers allowed session hijacking (the famous Mitnick attack of 1995), and "blind" connection resets are used even by state firewalls. The transport has no built-in cryptography — it was added on top, and this seam still determines the look of the protocols.
7. The performance question. At speeds of 100 Gbit/s, the bottleneck becomes not the network, but processing: system calls, memory copying, interrupts. The C10K problem, event APIs, zero-copy, kernel bypass, and RDMA — this whole industry exists because the price of delivering a byte to the application in the twenty-first century has equaled the price of transmitting it over the wire.

Next we will go through the entire transport layer: from the historical separation of TCP and IP and the Berkeley sockets — through ports, UDP, the structure of the TCP segment, handshakes and the finite state machine, reliability and timers, flow and congestion control (from Tahoe of 1988 to BBR) — to the performance of kernel stacks, alternative protocols (SCTP, MPTCP), QUIC and HTTP/3, real time and RPC, security and practical traps like TIME_WAIT and "sticky packets." In the finale we will trace the lifecycle of one connection — from SYN to the last ACK.

The birth of the transport layer

From a single protocol to the TCP/IP pair

In Cerf and Kahn's 1974 article, the TCP protocol was a single whole: it was responsible for addressing, for routing datagrams across network boundaries, and for reliable delivery between processes. By 1977 it became clear that one protocol was solving tasks that were too different — and, worse, doing it with different required speeds. The split occurred in 1978, largely under the influence of Danny Cohen, who was working on transmitting voice over ARPANET: his experiments showed that for speech, reliable delivery with retransmissions is harmful — an overdue packet is useless, and the delay of retransmission kills interactivity. Applications needed two different services: a reliable stream for files and an unreliable datagram for voice.

Thus the pair was formed: IP took upon itself the delivery of datagrams between hosts, TCP — the reliable stream between processes. In August 1980, RFC 768 came out — the specification of UDP, a protocol that added to IP exactly two properties: ports and a checksum. In September 1981 — RFC 793 (TCP) together with RFC 791 (IP) and RFC 792 (ICMP). And in 1989, RFC 1122, edited by Robert Braden, brought together the requirements for hosts: the document that to this day formally defines what a correct TCP/IP stack is obliged to do.

This separation of layers — "the network delivers to the machine, the transport delivers to the process" — is precisely the defining idea with which the whole conversation begins. IP does not know how many processes run on the host; TCP and UDP do not know how many networks lie between the hosts. Each layer solves its own task and nothing more.

The two generals problem: the fundamental limit of reliability

Before dissecting the mechanisms, it is worth understanding the theoretical boundary they run into. The "two generals problem" is a thought experiment formalized in the mid-1970s (Akyounlou, Eswaran, and Huber, 1975; the classic formulation was given by Jim Gray in 1978): two generals besieging a city must coordinate a simultaneous attack, but the messages between them pass through a valley where they are intercepted. However many confirmations the generals send each other, the last confirmation can be lost — and neither of them can be absolutely sure that the other is sure. The formal result: reliable agreement on state over an unreliable channel is impossible in the general case.

For the transport layer this is not philosophy but an engineering constant. When a client sends a request and does not receive a response, it fundamentally cannot distinguish three situations: the request was lost; the server did not execute it; the request arrived, the server executed it, but the response was lost; the server died in the process. Hence — the eternal dilemma of "exactly once": without an ideal channel, only "at least once" (with the risk of repeats) and "at most once" (with the risk of non-execution) are achievable. TCP chooses at-least-once for data inside an established connection, and applications for which non-repeatability is critical are forced to make operations idempotent and introduce request identifiers. The whole theory of distributed systems — from repeated RPC calls to database commit protocols — grows out of this impossibility discovered at the dawn of network design.

Sockets: the Berkeley interface that defeated everyone

The mechanisms of the layer are meaningless without an interface through which applications use them. In 1983, in the 4.2BSD system, the team of Bill Joy and Sam Leffler introduced the sockets API: socket(), bind(), listen(), connect(), accept(), send(), recv(), close(). The genius of the interface was not in the functions, but in the metaphor: a socket looked like a file descriptor. The programmer wrote network code with the same system calls as file code — read/write worked both with the disk and with the network, and the Unix philosophy "everything is a file" unexpectedly covered distributed systems too.

The competitor was serious: AT&T was promoting the Transport Layer Interface (TLI) in System V — a "correct" API not tied to the file model. BSD won the war of interfaces: first because 4.2BSD was distributed to universities cheaply, then because all the loud applications of the era grew on sockets — telnet, FTP, later the web. TLI remained a footnote in history, and sockets became so universal that Winsock, POSIX sockets, and even modern asynchronous APIs continue their semantics. A socket is the point at which the abstraction "connection" turns into a concrete pair (IP address, port) for UDP and a quadruple (source IP address and port, destination IP address and port) for TCP.

The analogy for remembering is classic: the IP address is the house, the port is the apartment, the socket is the specific apartment in which the process lives. The postman (IP) delivers the letter to the house; the mailbox with the apartment number (port) decides which resident (process) it will go to.

The tasks of the transport layer

Let us formulate the layer strictly. The transport layer provides data transmission between processes on top of a network that delivers packets between hosts. Its unit of data is the segment; for comparison: the data link layer operates with frames, the network layer — with packets. The encapsulation chain looks like this:

application data → TCP segment → IP packet → Ethernet frame → signal

The service of the layer is described by a set of primitives — operations that an application can request from the transport: LISTEN (wait for an incoming connection), CONNECT (establish a connection), SEND (transmit data), RECEIVE (accept data), DISCONNECT (terminate the connection). In the Berkeley API, these primitives correspond almost one-to-one to listen(), connect(), send(), recv(), and close() — a rare case when the abstraction of the standard and the abstraction of the implementation coincided completely.

Delivery between processes: ports

An IP address identifies a host, but dozens of processes run on a host, and several of them may need to receive data simultaneously. A port is a sixteen-bit number identifying an endpoint inside the host; the pair (IP, port) is called the connection's endpoint. Numbers familiar to everyone: 80 — HTTP, 443 — HTTPS, 22 — SSH, 53 — DNS, 25 — SMTP. The port space is divided into three ranges (the registry is kept by IANA, the rules are RFC 6335): "well-known" ports 0–1023, to which standard services are assigned and which in Unix systems historically can only be occupied by a privileged process (root); "registered" 1024–49151 for applications; and "dynamic" (ephemeral) — approximately 49152–65535, from which the OS allocates outgoing connections. In Linux the ephemeral port range by default is 32768–60999, that is, about 28 thousand ports, and this figure will come up later when we talk about exhaustion.

Multiplexing and demultiplexing

Multiplexing is the ability of many applications to simultaneously use one transport stack and one IP address. Demultiplexing is the reverse operation: to deliver the arrived segment to the right process. And here hides a subtlety that is often missed: in UDP, demultiplexing goes by the pair (destination IP, destination port), while in TCP — by the full quadruple (both addresses and both ports). The difference is fundamental: one UDP socket bound to port 53 receives datagrams from all senders at once (this is how DNS servers work), whereas a TCP connection is uniquely defined by the quadruple, so thousands of clients can be connected to one and the same server port 443 — each connection is unique by its client address and port. It is precisely this that allows a web server with one listening socket to serve tens of thousands of simultaneous sessions.

The services of the layer: from a reliable stream to a "construction set"

The transport layer provides two polar services. A reliable byte stream with connection establishment, guarantees of delivery and order, flow control and congestion control — TCP. And a minimal unreliable datagram service without connections and guarantees — UDP. All the rest of the variety (RTP, SCTP, QUIC, DTLS) is built between these poles or on top of one of them — and this, looking ahead, is the main plot of the forty-year evolution of the layer.

UDP: the minimal transport

What UDP does — and what it fundamentally does not do

UDP (User Datagram Protocol, RFC 768, August 1980) is the smallest protocol of the family: its header occupies eight bytes and contains exactly four fields: source port, destination port, datagram length, and checksum. No connections, no acknowledgments, no retransmissions, no order, no congestion control. A datagram is sent with one system call and goes into the network as is — "send and forget." If the application sent a three-kilobyte message, the receiver will get exactly one three-kilobyte message: message boundaries are preserved, unlike in TCP.

For these eight bytes the application receives exactly two properties that bare IP does not have: multiplexing by ports and (optionally in IPv4, mandatorily in IPv6) integrity control. Everything else UDP leaves for the application itself to implement — and this is not a shortcoming, but an architectural position: minimal service, maximum freedom. Not for nothing is UDP called "a construction set for your own protocols": almost everything invented in transport over the last forty years was built on top of it.

The checksum and the pseudo-header

The UDP checksum is arranged curiously: it is computed not only over the UDP fields and the data, but also over a "pseudo-header" — a fragment of the IP header with the source and destination addresses and the protocol number. Formally this is a violation of layering (the transport peeks into the network layer); practically — elegant insurance: a datagram delivered to the wrong host due to a routing failure will be discarded, because the addresses in the pseudo-header will not match. In IPv4 the checksum is allowed not to be used (a zero value) — a concession to the slow hosts of the eighties, which turned into a source of hard-to-catch errors; in IPv6 it is mandatory. A separate variation is UDP-Lite (RFC 3828, 2004), allowing the checksum to cover only part of the data: a codec can protect the frame header and come to terms with bits in the low-order planes of the image, which on noisy wireless channels gives a picture with artifacts instead of the loss of the frame.

Where UDP is the right choice

The list of UDP applications reads like a list of everything that is sensitive to delay or simply does not need reliability: DNS (a short request — a short answer; retransmission is cheaper than establishing a connection), VoIP and video conferencing (an overdue packet is useless, and a lost frame is better replaced by the next one), online games (the state of the world is updated dozens of times per second; an old update is not "fixed" but thrown away), streaming broadcast, DHCP, NTP, SNMP. The common logic: when the cost of waiting for a lost packet is higher than the cost of losing it, reliability becomes an anti-function. Everything real-time belongs to the same class — RTP, WebRTC, and related protocols will be discussed separately.

The price of freedom: the absence of congestion control and "ossification"

Freedom has a reverse side too. UDP does not slow down under congestion: an application sending datagrams without limits is capable of flooding the shared channel — that is exactly why "honest" implementations (RTP, QUIC) are obliged to add their own congestion control on top, and RFC 8085 (2017) "UDP Usage Guidelines" directly prescribes that application protocols either limit their rate or behave "TCP-friendly." The second problem is middleboxes: the intermediate equipment for decades "learned" on TCP and treats bare UDP with suspicion; some networks cut UDP traffic or limit it; part of the carrier equipment breaks datagrams by fragmenting them in an unsafe way. Therefore modern UDP rarely travels "naked": on top of it there almost always lies another layer — DTLS (TLS for datagrams), or QUIC, or RTP with its own headers — which at the same time protects the protocol from the interference of middleboxes and makes the traffic less distinguishable for filters. This race between protocols and intermediate equipment is the harbinger of the big theme of "ossification," to which we will return in the section about QUIC.

TCP: the protocol on which the reliable Internet rests

The TCP segment: the header field by field

TCP transmits data in segments; each segment is a header (minimum 20 bytes, up to 60 with options) and a payload. Let us dissect the header (RFC 793 with subsequent additions) in the order the fields follow.

Source Port / Destination Port (16 bits each) — the quadruple, together with the IP addresses, uniquely identifies the connection.

Sequence Number (32 bits) — the number of the first byte of data of this segment in the stream; in a segment with the SYN flag — the initial sequence number (ISN), from which the counting begins.

Acknowledgment Number (32 bits) — the number of the byte that the sender of the acknowledgment expects to receive next; the acknowledgment is cumulative: "everything up to this byte has been received."

Data Offset (4 bits) — the length of the header in 32-bit words (the analog of IHL in IP).

Flags (8 bits) — control bits: CWR and ECE (the ECN echo — congestion signals from RFC 3168), URG (urgent data), ACK (the acknowledgment field is valid), PSH (a request to hand the data to the application immediately), RST (emergency reset), SYN (connection establishment), FIN (termination). Of the eight flags, four participate in the everyday life of a connection: SYN, ACK, FIN, RST. Two others are relics of the early era. URG marks "urgent data": the mechanism was inherited from telnet, where the interrupt command was transmitted this way, but implementations diverged even in the interpretation of the urgent pointer (RFC 793 and BSD understood it differently), and today it is practically dead. PSH is a request to the receiver's stack to "give the data to the application immediately, without waiting for the buffer to fill": not a rule, but a hint, which modern implementations often ignore — interactivity is provided by settings like TCP_NODELAY.

Window Size (16 bits) — the size of the receiver's window: how many more bytes the sender may send without waiting for it to unload. Sixteen bits are enough for only 64 kilobytes — this limitation became one of the main problems of TCP and gave birth to the window scaling option; about it in the section on flow control.

Checksum (16 bits) — the checksum of the header, the data, and the IP pseudo-header, as in UDP.

Urgent Pointer (16 bits) — the offset of the urgent data; practically dead.

Options — the variable part: MSS (the maximum segment size that the sender of the option is ready to accept), Window Scale (the window multiplier), SACK Permitted (permission for selective acknowledgments), Timestamps (time marks for precise RTT measurement and protection against the "wrapping" of numbers). The options are negotiated in the handshake — and it is precisely with them that TCP acquired most of its modern functionality.

MSS and the connection with MTU

MSS (Maximum Segment Size) is the maximum amount of data in a segment. It is computed from the path MTU: for classic Ethernet, MTU 1500 minus 20 bytes of the IP header minus 20 bytes of the TCP header give MSS 1460. Negotiating MSS in the handshake is the first echelon of protection against fragmentation at the network layer: the parties agree in advance not to send each other segments that would have to be cut. For tunnels (IPsec, GRE, PPPoE), the MSS is recomputed, and home routers often apply MSS clamping — "pinching" the option in passing handshakes so that clients do not barge through with packets that do not fit into the tunnel.

A stream of bytes, not of messages

TCP is a byte-oriented protocol: the application writes into the connection a sequence of bytes, and the transport is obliged to deliver it to the receiver in the same order, without gaps and repeats. Message boundaries do not exist here. Two send() calls with "Hello" and "World" may arrive in one recv() as "HelloWorld," or they may arrive in three pieces "He," "lloWo," "rld": TCP has the right to glue and split the data however it likes, as long as the bytes go in order. This is not a bug, but a consequence of optimizations (Nagle's algorithm accumulates small portions, the stack merges segments), but for developers it is an eternal source of errors known as "sticky packets." There is one rule: framing is the duty of the application protocol. Boundaries are set either by length (a prefix: first 4 bytes of "how much to read next"), or by a separator (lines ending with \n — this is how HTTP, Redis, IRC, SMTP work), or by a fixed record size. A protocol that "just sends JSON into a socket" and relies on the boundaries of send() will break at the first fragmentation — sooner or later, usually in production.

Sequence numbers and ISN: from the Mitnick attacks to randomization

Sequence numbers solve three tasks at once: they set the order of the bytes, they enable cumulative acknowledgments, and — what is most important historically — they protect against duplicates. A packet that lingered in the network and arrived again will be discarded if its bytes are already covered by the acknowledged range: the sequence number turns an "old" segment into garbage. The counter is 32-bit and on modern channels wraps around in seconds — at 1 Gbit/s in about half a minute, at 10 Gbit/s even faster — therefore, together with timestamps, the PAWS mechanism (Protection Against Wrapped Sequences) works, cutting off segments with "old" timestamps.

A separate drama is connected with the choice of the initial number (ISN). RFC 793 prescribed increasing the counter by one every four microseconds — predictable. At Christmas 1995, Kevin Mitnick used this for one of the most famous attacks in history: he "blinded" a trusted host (flooding it with SYNs), computed by the algorithm the sequence number it would generate next, and, on behalf of the trusted machine, established an rlogin session with the computers of Tsutomu Shimomura — the security specialist who was the one catching Mitnick. The attack did not require intercepting traffic: it was enough to guess the ISN. The answer was RFC 1948 (1996) by Steven Bellovin: the initial number must be a cryptographically strong function of the connection quadruple and a secret key. Modern stacks generate the ISN exactly this way — and the sequence prediction attack, with which the career of Shimomura the "hacker fighter" began and Mitnick's decline started, is a thing of the past.

The TCP checksum: the weak link that is not changed

The sixteen-bit ones' complement sum is a cheap but weak code: it does not detect, for example, the permutation of two 16-bit words. In 2000, the study by Stone and Partridge "When The CRC and TCP Checksum Disagree" measured reality: checksum failures occurred orders of magnitude more often than theory predicted (on certain routes — up to one packet in a thousand), which means that memory and hardware make mistakes more often than was thought, and the TCP checksum regularly catches what the data link CRC did not catch. Compatibility and cost do not allow replacing it with something strong; instead, the industry went the way of a division of labor: the integrity of application data is ensured by TLS and application protocols, and the TCP checksum remained a fast filter of gross errors — often computed by the network card (checksum offload) so as not to burn CPU cycles.

The lifecycle of a connection

The three-way handshake: why exactly three

Establishing a TCP connection is an exchange of three segments: the client sends a SYN with its initial number X; the server answers with a SYN-ACK, acknowledging X+1 and reporting its own number Y; the client finishes with an ACK acknowledging Y+1. After this, both parties are synchronized on numbers and the connection is established — the ESTABLISHED state, in which the data transfer takes place.

Why not two stages? Precisely because of the "lingering duplicates," the very ones that sequence numbers protect against. Imagine an old SYN that lay in the network for minutes and arrived at the server when the client had long since changed its mind about connecting: with a two-stage scheme, the server would open a connection into nowhere and hold resources for a phantom client. The third step gives the client the opportunity to confirm (or, seeing something unexpected, to reset) — the connection opens only when both parties have convinced themselves that they are talking to a live interlocutor about the present moment. Simultaneous opening (both parties send SYN towards each other) is also provided for by the protocol — it passes through the SYN-RECEIVED state at both — but in practice it is rarely encountered.

In the handshake, too, the options are negotiated: MSS, window scaling, SACK permission, timestamps. This is the only moment when the parties "agree on the language" — after it, the set of capabilities of the connection is frozen for its whole life. A separate upgrade is TCP Fast Open (RFC 7413, 2014): a cookie obtained during a previous connection allows attaching data right to the SYN and saving a whole RTT at the start. For the web, where connections live for milliseconds, this is substantial — and exactly the same motivation will later be realized by QUIC with its 0-RTT and 1-RTT.

The finite state machine: eleven states

TCP is not just a protocol, but a state machine, and knowing its map distinguishes an engineer from a user of the API. The full set: CLOSED, LISTEN (the server waits), SYN-SENT (the client sent SYN), SYN-RECEIVED (the server received SYN, sent SYN-ACK), ESTABLISHED (data transfer), and then — the cascade of closing: FIN-WAIT-1 (sent FIN), FIN-WAIT-2 (our FIN is acknowledged, waiting for the interlocutor's FIN), CLOSE-WAIT (received the interlocutor's FIN, the application has not yet closed the socket), LAST-ACK (sent our own FIN, waiting for the last ACK), TIME-WAIT (waiting for old segments to disappear), and again CLOSED.

The normal lifecycle looks like this. At opening, the client passes the path CLOSED — SYN-SENT — ESTABLISHED; the server — LISTEN — SYN-RECEIVED — ESTABLISHED. Closing is a "four-stage handshake": each party independently sends a FIN and receives an ACK for it, because the stream is half-duplex — having closed the direction "from oneself," one can still receive data. The actively closing party (usually the client, but often the server too) passes FIN-WAIT-1, FIN-WAIT-2, and TIME-WAIT to CLOSED; the passive one — CLOSE-WAIT and LAST-ACK to CLOSED. The diagnostic value of the map is enormous: thousands of sockets in CLOSE-WAIT are not a network problem, but a program that does not call close() (the FIN was received, the application "forgot" to close its side); thousands of sockets in TIME-WAIT are the normal consequence of active closing, and it is treated differently.

TIME-WAIT: the state that everyone hates in vain

TIME-WAIT lasts 2×MSL (MSL is the maximum segment lifetime; RFC 793 suggested two minutes, practical implementations usually take 30 seconds — hence the fixed 60 seconds of TIME-WAIT in Linux). What is it needed for? Two reasons. The first is the reliability of the last ACK: if it is lost, the interlocutor will repeat its FIN, and the party in TIME-WAIT will be able to answer again — without this, the other side would get stuck in LAST-ACK, and the connection would close "dirty." The second is protection against ghosts: the old segments of a dead connection wandering through the network must die before a new connection with the same quadruple of addresses and ports appears. TIME-WAIT is a quarantine guaranteeing that the previous life of the quadruple has ended before the next one begins.

The practical pain is known to everyone who has operated a high-load backend: a web server or proxy actively closing tens of thousands of short connections per second accumulates tens of thousands of sockets in TIME-WAIT; the ephemeral ports (those very ~28 thousand in Linux) are exhausted, and new outgoing connections are not created. The medicines: close the connection on the client side, not the server side (then TIME-WAIT lives at the client); reuse connections (keep-alive is the best prevention); SO_REUSEPORT (Linux 3.9, 2013), allowing several sockets to listen on one port and share the load; tcp_tw_reuse for outgoing connections. But the famous tcp_tw_recycle — an aggressive shortening of TIME-WAIT by timestamps — turned out to be dangerous behind NAT (one timestamp for many clients broke the logic) and was removed from the Linux kernel in 2017. The moral: TIME-WAIT is not a disease but an immunity; one must fight the cause of its accumulation, not the state itself.

RST: the emergency reset and the weapon of censorship

The RST flag tears the connection down immediately, without handshakes: nobody is listening on the port (the classic — a SYN to a closed port gets an RST), a segment arrived at a nonexistent connection, the application demands an emergency close (SO_LINGER with zero), or data arrived in an invalid window. RST also has a dark side: the "blind" injection of a reset. An attacker who does not see the traffic but knows (or has guessed) the quadruple and the approximate sequence number can send a forged RST and tear someone else's connection. This is not theory: measurements of 2005–2006 showed that the "Great Chinese Firewall" massively applies RST injections to tear down "undesirable" sessions, sending several packets at once with different sequence numbers — so that at least one hits the window. RFC 5961 (2010) made such attacks harder (a suspicious RST must now be answered with a challenge ACK, not by tearing the connection), but it did not fully remove the problem of "blind" interference: a transport without cryptography is forgeable in principle.

Half-open connections and keepalive

If a host suddenly dies (a crash, a pulled cable), its interlocutor does not learn about it: the connection remains "half-open" — one side considers it alive, the other no longer exists. Data sent into nowhere will accumulate in the buffers until they are exhausted. The remedy is keepalive: if there is no traffic in the connection for a long time (by default in Linux — two hours), a party sends an empty probe; several unanswered probes in a row declare the connection dead. The two-hour timeout suits almost nobody, so in production it is reduced (or, which is more correct, application-level heartbeats are used — protocols like WebSocket ping/pong, gRPC keepalive, or AMQP heartbeat, which live in the data stream and do not depend on the stack's settings). TCP keepalive is the last line of defense, not a working tool.

Reliability: acknowledgments, timers, retransmissions

Cumulative ACKs and duplicate acknowledgments

The basis of TCP's reliability is acknowledgments: the receiver in every ACK reports "the next expected byte N," which is equivalent to "all bytes up to N have been received." The acknowledgment is cumulative — compact and resistant to losses of the ACKs themselves (a lost acknowledgment is replaced by the next one), but coarse: if one segment disappeared from the middle of the stream, the receiver keeps sending one and the same number — duplicate ACKs. For the sender, three duplicates in a row are almost certainly a sign of a loss (not a delay): there is no reason to wait for a timeout; the segment is resent immediately. This is how fast retransmit works, added to the stack in 1988.

The cumulative scheme has a blind spot: the sender does not know what has been received after the hole. SACK (Selective Acknowledgment, RFC 2018, 1996) closed it — an option allowing the ACK to additionally list received out-of-order blocks: "I am waiting for byte 1000, but 2000–3000 and 4000–5000 I have already received." With SACK, retransmission becomes pinpoint — only what was actually lost is resent. Together they give behavior close to the selective repeat model from the textbooks: basically TCP is go-back-N with fast recovery, and with SACK — almost selective repeat. The modern addition is the RACK mechanism (standardized by RFC 8985 in 2021): the loss is determined not by counting duplicates, but by time — a segment is considered lost if a later one has already been acknowledged and this one is "overdue" relative to the measured RTT. RACK works where duplicate counting is powerless (short flows, a small window), and has become standard in modern kernels.

Timers and RTT estimation: adaptivity as a principle

The main timer of TCP is RTO (Retransmission Timeout): how long to wait for an ACK before retransmitting. Too short a timer breeds false retransmissions (duplicates load the network and knock down congestion control); too long — leaves the connection idle after a real loss. A fixed value is impossible: the RTT in the Internet varies from milliseconds in data centers to seconds on satellite channels and "breathes" during a single session. The solution is adaptive estimation: the smoothed average SRTT plus an estimate of the variance RTTVAR; by the Jacobson-Karels formula (1988) RTO = SRTT + 4·RTTVAR (standardized in RFC 6298; the lower bound in Linux is 200 ms). A separate subtlety is Karn's algorithm (Phil Karn, 1987): it is impossible to measure RTT on segments transmitted again — it is unclear whether this is the acknowledgment of the first transmission or the second; such samples are discarded, and on repeated losses the RTO grows exponentially (backoff).

A connection has several timers: RTO — for data, the persist timer — for watch duty at a zero window (see flow control), keepalive — for detecting the death of the interlocutor, the FIN-WAIT-2 timer — so that the party that has already closed its own direction does not hang forever waiting for the other's FIN. The totality of these mechanisms is what makes TCP "self-governing": a connection without human intervention survives losses, delays, pauses of the receiver, and partly — the death of one of the parties.

Buffering at both ends

Reliability costs memory. The sender stores the sent but not yet acknowledged bytes (they may need to be transmitted again) and the data accepted from the application but not yet fitting into the window. The receiver stores segments received out of order (a hole in the middle of the stream obliges keeping the "tail" until it is filled) and the data the application has not yet taken. Hence — the direct connection of the transport with system performance: the sizes of the socket buffers limit the achievable speed (the window cannot exceed the buffer), and pathological scenarios (a slow application on the receive side, deep reordering in the network) turn the buffers into a source of delays and pressure on memory. A big buffer is not equal to good performance — this maxim, proven on network queues by the phenomenon of bufferbloat, applies in full measure to the end nodes as well.

Delivery order and head-of-line blocking

TCP guarantees order: segments that arrived ahead of time are buffered and given to the application only as a continuous prefix of the stream. The price is head-of-line blocking: if segment No. 2 is lost, the bytes of segments No. 3 and No. 4, already received and lying in the buffer, are not given to the application — the stream has "frozen" waiting for the hole, even if the lost data is not urgently needed. For transferring a file this is the right price; for multiplexed applications — a catastrophe: in HTTP/2, dozens of logical streams share one TCP connection, and the loss of one segment stops them all. This fundamental contradiction of "order versus independence" is the main motive for the appearance of QUIC, where every stream is reliable by itself and a loss in one does not block the others.

Flow control: do not overload the receiver

The sliding window and the receiver's window

Flow control solves a task different from congestion: the sender must not send data faster than the receiver manages to process it. The mechanism is the sliding window. The receiver in every segment (in the Window Size field) reports how many bytes it is ready to accept beyond what has already been acknowledged; the sender keeps "in flight" no more than this volume. As the application takes data from the buffer, the window grows; as data is sent, the effectively acknowledged boundary "slides" forward. The actual transmission limit is the minimum of two windows: the sender has the right to keep in flight no more than is simultaneously allowed by the receiver's window (rwnd — flow control) and the congestion window (cwnd — congestion control). Two independent brakes, and at every moment the stricter one is in effect.

The very idea of "send N segments without waiting for acknowledgments" is a generalization of three textbook schemes of reliable transmission. Stop-and-wait (sent one packet, wait for the ACK, send the next) is simple and hopelessly inefficient: its throughput equals the packet size divided by the RTT — on a 1 Gbit/s channel with a 100 ms delay, a one-and-a-half-kilobyte packet gives about 120 kbit/s — a ten-thousandth of what is available. Go-Back-N (a window without selective repeat: a packet was lost — retransmit everything after it) wastes bandwidth on extra transmissions. Selective Repeat (exactly what was lost is retransmitted) is optimal in bandwidth, but requires out-of-order buffers at the receiver. TCP, as already said, is a practical hybrid: cumulative ACKs gravitate to go-back-N, while fast retransmit and SACK bring the behavior closer to selective repeat.

The zero window and the persist timer: the anatomy of a deadlock

The application on the receive side can slow down — and the window collapses to zero. The sender stops. A dangerous situation arises: when the receiver frees its buffer and sends a window update, this message can be lost — and both parties will wait for each other forever: the receiver thinks it announced the window, the sender — that it is still zero. The deadlock is treated with the persist timer: having frozen at a zero window, the sender periodically sends tiny window probes — one-byte probes forcing the receiver to re-announce the window. The update can be lost again, but the probe will be sent again — the circle is broken. This is a classic example of how TCP patches the holes of the protocol with timers: any control segment can disappear, so every "silence" is obliged to have its own watchful timer.

Silly Window Syndrome and Nagle's algorithm

Another pathology is the "silly window syndrome" (SWS). A fast receiving application that takes data by the teaspoon announces tiny windows; the sender stuffs the segments with one or two bytes — and an avalanche of packets with 40-byte headers and a byte of payload goes over the network: the efficiency falls to single-digit percent. The syndrome is treated from two sides. The receiver (David Clark's solution) does not announce a window until it can accept at least a segment or half the buffer — better to stay silent than to open a window for one byte. The sender (Nagle's algorithm, RFC 896, January 1984) accumulates small portions of data until a full segment accumulates or an ACK arrives for what was sent earlier — and glues them into one packet. John Nagle invented the algorithm while fighting congestion collapse in the Ford Aerospace network: telnet sessions generated a packet per keystroke, and the network was drowning in one-byte segments.

The irony is that Nagle himself later admitted: his algorithm, paired with delayed ACK (the receiver holds the acknowledgment for up to ~40 ms, hoping to piggyback it onto response data), creates an evil combination — the sender waits for the ACK to send what it accumulated, the receiver waits for data to send the ACK; the result is periodic 40–500-millisecond "stickiness" of interactive applications. That is why in low-latency systems (trading terminals, games, SSH-like sessions) Nagle's algorithm is disabled with the TCP_NODELAY flag — and the author himself recommends doing this everywhere where interactivity is more important than saving packets. Linux and BSD also offer the reverse tool — TCP_CORK / TCP_NOPUSH: "do not send anything except full segments until I remove the cork," convenient for sending a header and body in one motion.

Window scaling and the "long fat pipes"

The sixteen-bit window field is a time bomb laid in 1981. A maximum of 65,535 bytes "in flight" means a limiting throughput of window/RTT: at an RTT of 100 ms — 5.2 Mbit/s; at a satellite RTT of 600 ms — less than a megabit. Channels grew faster than delays fell, and by the end of the 1980s the class of "long fat networks" (LFN) arose — channels with an enormous product of throughput and delay. This product is called the bandwidth-delay product (BDP) and shows how many bytes must be on the way to fill the channel entirely: 1 Gbit/s at 100 ms is 12.5 megabytes in flight; a transatlantic 10 Gbit/s at 80 ms — 100 megabytes.

The answer was the TCP Extensions for High Performance: first RFC 1072 (1988), then RFC 1323 (1992) by Van Jacobson, Robert Braden, and Dave Borman — they added the window scale option (a multiplier up to 2³⁰; the window stretches to a gigabyte), timestamps (precise RTTM and PAWS), and, as the foundation, the concept of LFN. It was precisely these extensions that allowed TCP to survive the transition of the Internet into the gigabit era: without window scaling, the modern Internet physically could not fill its backbones. Today, windows of tens of megabytes are routine, and stacks automatically adjust the buffer sizes to the measured BDP (receive buffer autotuning). The reverse side is the demand on memory: a "fat pipe" obliges holding megabytes of unread data in the buffers of both ends.

Congestion control: how TCP runs the Internet

The legacy of 1988: Tahoe, Reno, and the four algorithms

The network layer deliberately does not control the speed of senders — a datagram network has neither the state of flows nor a return channel to the sources. Therefore congestion control lives in the transport: each TCP stack independently feels out its share of the common bandwidth. The algorithms that Van Jacobson invented for this after the collapse of 1986 (and which entered the 4.3BSD-Tahoe release of 1988) are still the skeleton of any congestion control.

Slow start: the connection begins with a window of one segment (the modern standard is an initial ten, RFC 6928) and, for every ACK that arrives, increases the window by another segment, that is, doubles it every RTT — exponential acceleration, until the window hits the ssthresh threshold or a loss appears. The name is ironic: the start seemed "slow" against the background of stop-and-wait. Congestion avoidance: after the threshold, the window grows linearly — an increment of about one segment per RTT, a cautious approach to the boundary beyond which losses begin. Fast retransmit: three duplicate ACKs are treated as the loss of one segment — retransmit immediately, without waiting for the RTO. Fast recovery (added in 4.3BSD-Reno, 1990): after a fast retransmission, the window is not zeroed, but cut in half and work continues — the loss of one packet does not yet mean the collapse of the network. The combination of Reno behavior with SACK received the name NewReno and was canonized in RFC 2581 (1999) and the current RFC 5681 (2009).

AIMD: the sawtooth that leads to fairness

The total dynamics of the window — additive increase and multiplicative decrease (AIMD) — draws the famous "sawtooth": cwnd slowly crawls up; upon a loss it instantly falls by half; it grows again. The throughput of one connection on average is approximately 0.75·cwnd/RTT. The sawtooth looks like waste (the channel is periodically underloaded), but it has a deep property, proven in the classic work of Chiu and Jain (1989): of all pairs "increase/decrease," only the additive-multiplicative scheme converges to a fair and efficient division of the channel among competitors. Geometrically: the vector of additive increase points toward the fairness diagonal at 45°; multiplicative decrease projects the system back onto the line of efficient utilization — iteration after iteration, all flows are pulled toward equal shares. Tens of thousands of independent TCP flows divide the Internet without a single coordinator — a distributed algorithm that works because each of its parts selfishly observes AIMD. This is probably the largest example of a self-organizing control system in the history of technology.

The Mathis formula and the price of losses

The behavior of this sawtooth is described by the formula of Mathis and co-authors (1997): the throughput of a connection is approximately equal to MSS / (RTT·√p), where p is the probability of loss. The consequences are ruthless: the speed falls inversely proportional to the square root of the losses (losses of 1% cut the speed tenfold relative to losses of 0.01%) and inversely proportional to the RTT. Hence — all the woes of distant connections: satellite and transoceanic channels with a high RTT and the slightest losses give a miserable speed even on a fat pipe; hence too — the incentive to bring content closer to users (CDNs cut the RTT) and to guard against losses (buffers, FEC). The formula also explains why "one loss per gigabit" is a catastrophe for a loss-based algorithm: the window recovers to its previous values for minutes.

Unfairness and the wars of algorithms

The fairness of AIMD has caveats. First, RTT unfairness: connections with a short RTT receive ACKs more often and grow faster — in a shared channel, the near flow eats more than the far one. Second, the incompatibility of different algorithms: flows that react more aggressively to bandwidth (or do not react to losses at all) displace the classic AIMD. Both problems are not bugs, but properties of a decentralized game without rules: "fairness" in the Internet is an equilibrium, not a guarantee.

History knows an attempt to play by other rules: TCP Vegas (Brakmo and Peterson, 1995) oriented itself not on losses, but on the growth of delay — an increasing RTT meant that the queues were growing, and Vegas braked in advance, without letting it come to losses. The algorithm was more elegant and friendlier… and lost in the competitive struggle: in a channel shared by Vegas and loss-based Reno, the second occupies more bandwidth (it does not slow down before losses), and it is disadvantageous for an operator to enable Vegas alone. A classic prisoner's dilemma at the scale of a protocol: individual politeness is punishable until everyone is polite. Vegas remained in history, and its ideas were reborn in BBR — under different starting conditions.

The era of gigabit channels gave birth to the family of "high-speed" algorithms. BIC (2004) and its successor CUBIC (Ha, Rhee, and Xu, 2008) grow the window not linearly, but by a cubic function of the time elapsed since the last loss: a fast return to the previous speed, a plateau in its neighborhood, a cautious probing higher. CUBIC since 2006 (kernel 2.6.19) is the default algorithm in Linux and the de facto standard of the Internet; in 2018 it was canonized as RFC 8312. Microsoft in those same years promoted Compound TCP (the sum of loss and delay components) — it lived in Windows and neatly lost to CUBIC in mixed traffic.

BBR: a model instead of losses

In 2016, Neal Cardwell and the Google team proposed a radical alternative: BBR (Bottleneck Bandwidth and Round-trip propagation time). Instead of reacting to losses as the only signal, BBR builds a model of the channel: it continuously estimates the throughput of the bottleneck (BtlBw) and the minimum "propagation" delay (RTprop) and keeps the amount of data in flight near their product — that very BDP. The idea: congestion sets in not when packets start being lost, but when in-flight exceeds the BDP; up to that point you can drive fast, beyond it — it is pointless (the data stands in queues, the speed is the same, the delay is higher). BBR cyclically probes both quantities (the ProbeBW phase iterates over rate coefficients; the ProbeRTT phase periodically drains the window to a minimum in order to measure the unoccupied delay).

The results turned out to be impressive: on transatlantic channels with background losses, BBR gave a multiple gain over CUBIC; Google deployed it on its long-haul backbone and in YouTube even before publication. But the first version also generated conflicts: BBR ignored losses, "pushing through" its estimate, and in shared bottlenecks with loss-based flows it behaved aggressively, while on small buffers it itself became a source of losses. BBRv2 (2019) and BBRv3 (2023) added a reaction to losses and ECN, "ceiling" constraints — the protocol grew up. Another detail is telling: since 2021 the Linux kernel allows loading congestion control algorithms as eBPF programs (BPF struct_ops) — researchers and operators experiment with new algorithms without kernel patches; congestion control turns from a monolith into a plugin.

ECN and data center TCP

The alternative to losses as a signal is explicit marking: ECN (RFC 3168, 2001) allows a router to set the bit "it is crowded here," and the TCP stack — to signal its partner with the ECE/CWR flags and reduce the window without a single lost drop. In the backbone Internet, ECN took root sluggishly (support of the whole chain is needed), but in data centers it became the foundation: the DCTCP algorithm (Alizadeh et al., SIGCOMM 2010) reads not the fact of marking, but its fraction, and proportionally reduces the window — with marking of only a few percent, the queues are kept tiny, the delays microsecond-scale, and the throughput almost full. DCTCP showed that the transport can use network telemetry far more finely than "loss/no loss" — and set the direction in which L4S and the ECN modes of BBRv2/v3 develop.

Wireless networks: when a loss is not congestion

The fundamental assumption of loss-based control — "a loss means the network is full" — collapses in wireless environments: Wi-Fi and cellular channels lose packets due to interference, fading, and roaming, with completely empty queues. TCP "gets scared" and cuts the speed several times over, although there is nobody to slow down for. The solutions divide into classes: special algorithms that take into account the nature of the losses (Westwood+ estimates the available bandwidth by the pace of ACKs), proxies with connection splitting (split-TCP, PEP — carrier gateways terminating TCP closer to the subscriber and keeping the radio segment under their control), and a transition to transports that measure rather than guess (BBR, QUIC with configurable control). The problem is as old as the mobile Internet and has not been solved to the end — people have simply learned to work around it.

Performance: when the stack itself becomes the bottleneck

The price of a byte: system calls, copies, interrupts

At speeds of 10–100 Gbit/s and RTTs of single-digit milliseconds, the limitation moves from the network into the host: the processor does not have time to process the packets. The path of a segment to the application includes a network card interrupt (or polling in polling mode), climbing up the stack in the kernel, several copies of the data between buffers, system calls for receiving and sending. Each packet costs microseconds of CPU; at 100 Gbit/s with a standard MSS this is tens of millions of packets per second — more than a general-purpose kernel is capable of processing. Hence — a whole industry of optimizations from the end of the 1990s onward.

The first class is moving work to hardware: checksum offload (the checksum is computed by the network card), TSO/GSO (segmentation of a large block into MSS-sized packets is deferred to the card or to the last moment in the kernel), GRO/LRO (the reverse operation on reception — gluing small packets into large ones before climbing up the stack), interrupt coalescing (one interrupt for a group of arrived frames instead of an interrupt for each frame). The second class is reducing copies: sendfile() and splice() (Linux, late nineties) allow transferring data from a file to a socket, bypassing user space — static content and media are offloaded without extra cycles; zero-copy reception gives the application a kernel page directly. The third class is bypassing the kernel entirely: DPDK and related frameworks take the network card's queue into a user process, which polls it in a busy-poll loop without interrupts and system calls; user-space stacks (mTCP, Seastar, F-Stack) implement their own TCP on top of this. The price is a dedicated CPU core for polling and the loss of universality; the gain is tens of millions of packets per second on one server. The extreme point of this evolution is RDMA (RoCE in data centers): a transport that works past both the kernel and the CPU, transferring the delivery of memory from process to process to the network card.

The C10K problem and event APIs

A separate axis of performance is not bytes, but connections. The classic "thread per connection" model runs into the limits of thousands of simultaneous threads, while a web server needs tens and hundreds of thousands. In 1999, Dan Kegel formulated the "C10K problem" — ten thousand simultaneous clients — and around it the event-driven model grew: one thread serves thousands of sockets, switching only to where there is readiness. select/poll (linear polling, O(n)) were replaced by kqueue (FreeBSD, 2000), epoll (Linux, 2002), and IOCP (Windows) — event registers with O(1) access; nginx (2004) proved that on such an architecture one server holds hundreds of thousands of connections. The modern continuation is io_uring (Linux 5.1, 2019, Jens Axboe): a submission queue and a completion queue in shared memory, allowing exchange with the kernel in batches of operations almost without system calls — a step toward the same kernel bypass, but without the loss of universality.

Incast and the specifics of data centers

Inside the data center, the transport has its own pathology — TCP incast: dozens and hundreds of servers, on request, answer one recipient simultaneously; the total burst instantly overflows the switch buffer; dozens of packets are lost; dozens of connections synchronously go into backoff — the useful throughput falls by an order of magnitude. The answers: DCTCP with its microscopic queues on ECN, prioritization and precise queue control, and radical alternatives like Homa (Montazeri et al., SIGCOMM 2018) — a transport with priority for short messages and a receiver that manages the sending — which show: the universal TCP is not optimal even in conditions where the network is perfectly manageable. Data center transport is a separate ecosystem living by other laws than the Internet.

Header compression

For large packets, the 40 bytes of IP+TCP headers are a negligible share; for IoT telemetry and voice packets (payload 20–160 bytes) — up to half the frame. The header compression mechanism (ROHC, RFC 3095, 2001) exploits the redundancy of the flow: most fields between neighboring packets change predictably or do not change at all, so deltas are transmitted over the channel, and the full header is restored at reception. In cellular networks, ROHC is mandatory for VoLTE — without it, conversational traffic would not fit into the radio air. On the web, the same role is played by QPACK (compression of HTTP headers in HTTP/3) — headers of the application level, while QUIC encrypts the transport ones entirely.

Alternatives and extensions of TCP

SCTP: a transport for telecom

SCTP (Stream Control Transmission Protocol, RFC 2960, 2000; the current version is RFC 4960, 2007) was designed for telephone networks: carrying SS7 signaling over IP required a reliable transport with functions that TCP did not have. SCTP is message-oriented (it preserves message boundaries), supports multistreaming (several independent substreams in one connection — a loss in one does not block the others: head-of-line blocking was solved long before QUIC) and multihoming (one connection with several addresses at each end — path redundancy). The handshake is four-stage, with a cookie mechanism of protection against flooding (unlike TCP, no state is stored on the server until the handshake completes). The fate of SCTP is instructive: in telecom (DIAMETER, Sigtran) it took root; in WebRTC the data channel works over SCTP-over-DTLS-over-UDP; but it did not become a general-purpose transport — the intermediate equipment and NAT did not understand it, and the ecosystem was already tailored to TCP and UDP. QUIC learned the lesson: one must build on top of a protocol that middleboxes definitely do not touch.

DCCP and other experiments

DCCP (Datagram Congestion Control Protocol, RFC 4340, 2006) offered a datagram transport with built-in negotiable congestion control (CCID profiles: TCP-like and the "equation-based" TFRC — for flows that need a speed "like TCP" but without retransmissions). The idea was sound; the deployment — almost zero: middleboxes strangled yet another new transport, and QUIC would later absorb this lesson — disguise yourself as what the network is already used to. From the same series is UDP-Lite. They all remained in the IANA registries as monuments to the main problem of transport: a new protocol is useless until its traffic passes through the existing Internet.

MPTCP: several paths at once

MPTCP (Multipath TCP; architecture RFC 6182, protocol RFC 8684, 2020) extends TCP to work over several paths at once: one connection for the application, several subflows over different interfaces and routes, common byte numbering and a single congestion control. The motivation is mobility: a phone with Wi-Fi and LTE simultaneously can aggregate bandwidth and seamlessly survive the departure of one of the networks. Apple included MPTCP in iOS 7 (2013) for Siri, later — for maps and the assistant; operators (traffic proxying) and data centers use it for balancing. MPTCP did not go out into the general Internet — again middleboxes: equipment that did not know the MPTCP options broke the handshakes; the fallback mode helped (at the slightest problem the connection degrades to ordinary TCP). MPTCP is important evidence: even an extension standardized by the IETF and implemented by giants runs into the "immune system" of the network.

Why TCP is so hard to change

Let us add up the observations into a diagnosis. TCP lives in the kernels of billions of devices, whose updates stretch over years and decades (embedded systems, IoT, old OSes). Its headers are visible to the intermediate equipment, which over forty years has learned to read them, modify them, and block the "incomprehensible" — any new option risks being discarded by a firewall on the way. This phenomenon was called ossification: the network "ossifies" around the current profile of protocols, and evolution becomes impossible without disguise. Attempts to extend TCP (new options, Accurate ECN, alternative algorithms) advance with the difficulty of decades, whereas the algorithms inside the existing header (CUBIC, BBR, RACK) change relatively freely — they are not visible from outside. The conclusion the industry drew by the mid-2010s: for the transport to become evolvable again, it must (a) be moved from the kernel into a space where updates are measured in weeks, (b) be hidden behind an already untouchable protocol (UDP 443 passes almost everywhere), and (c) be encrypted so that middleboxes cannot ossify around the new format. All three points are about QUIC.

QUIC: a transport from user space

From a Google experiment to an IETF standard

The story began in 2012 at Google: Jim Roskind designed an experimental protocol intended to speed up search and YouTube. Publicly, QUIC was announced in 2013, deployed in Chrome and on Google's servers; in 2016 the IETF established a working group for standardization, and in May 2021 came out RFC 9000 (the transport), RFC 9001 (integration of TLS 1.3), RFC 9002 (loss detection and congestion control). In June 2022, HTTP/3 was standardized (RFC 9114) — the application profile on top of QUIC. The path from draft to a global standard with billions of users took less than a decade — unprecedentedly fast for a transport; for comparison: MPTCP took about ten years from its first specification (2011) to the final protocol standard (2020) — and still remained a niche solution.

The architecture of QUIC: what exactly was redesigned

QUIC works on top of UDP: for the network these are ordinary datagrams to port 443; for middleboxes — an opaque stream (and after ossification around QUIC — a stream that is dangerous to touch). Inside is a reassembled transport.

Cryptography is built in, not added on. The TLS 1.3 handshake is combined with the transport one: 1-RTT for a full connection from scratch and 0-RTT for a repeated one (data leaves in the first packet; the price is the risk of replay attacks, so 0-RTT is allowed only for idempotent operations). Practically the whole QUIC header is encrypted — an observer sees a minimum of metadata; TCP, where all the flags and windows are open, looks against this background like a protocol from the eighties, which it is.

Streams instead of a single byte stream. A QUIC connection contains many logical streams, each reliable and ordered by itself, but independent of the others. The loss of a segment blocks only its own stream: head-of-line blocking at the transport level is eliminated (inside a stream — no, and this is a conscious boundary: applications that need order are provided it; those that do not — it is not imposed). HTTP/3 puts each request/response in its own stream.

Connection ID instead of the quadruple. The connection is identified by a random ID, not by the quadruple of addresses and ports: a phone that switched from Wi-Fi to LTE keeps the session (including video) without reconnecting — connection migration out of the box. TCP fundamentally cannot do this: a change of address for it means the death of the connection.

Packet numbers instead of sequence numbers. In TCP, one counter serves both the order of bytes and the identification of segments — hence the eternal ambiguity at retransmissions (which exact segment did the ACK come for?) and Karn's algorithm as a patch. QUIC separated the concepts: the byte stream is numbered inside the stream, and each packet receives a unique, non-reusable number — a retransmission is a new packet, acknowledgments are unambiguous, the RTT estimate is precise without special reservations.

Loss detection without RTO. RFC 9002 defines losses by acknowledgments and time (a relative of RACK): the retransmission timer as the primary mechanism is not needed — fewer false alarms and faster recovery. Congestion control is a plugin: the base implementation is loss-based (NewReno-like), Google uses BBR; the algorithm can be changed without touching the network.

User space. QUIC lives in the application's library, not in the kernel: updates come out together with Chrome and nginx, not with the OS distribution; A/B experiments with algorithms are part of the ordinary development cycle. The price is CPU: encryption and processing in user space are more expensive than in an optimized kernel (the 2022 studies — "QUIC Is Not Quick Enough over Fast Internet" — showed that at short RTTs of fast channels TCP is sometimes faster, until GSO/GRO for QUIC and hardware offloads appeared), and compatibility: part of the networks block UDP, so clients must be able to fall back to TCP (the Happy Eyeballs strategy for QUIC).

Deployment in numbers

Data on the deployment of QUIC have been published since 2017, when the article by Langley and co-authors (SIGCOMM) summed up the first results: the latency of Google Search fell by 8% for desktops and 3.6% for mobile users, rebufferings on YouTube — by approximately 15–18%, and QUIC already accounted for about 7% of world internet traffic. By 2026 the picture is different: HTTP/3 is supported by more than 40% of websites (by W3Techs measurements, October 2026); the share of HTTP/3 in world web requests fluctuates around 20%, and in the traffic of large CDNs it reaches 35–45%. QUIC has ceased to be "Google's protocol": Cloudflare, Fastly, Akamai, Microsoft (MsQuic), and Meta implement it; mobile applications all over the world use it for APIs. The next step is already visible: MASQUE — the standardization of proxying over HTTP/3 (iCloud Private Relay is built on it), and Media over QUIC (MoQ) — an attempt to reassemble live streaming on the transport principles of QUIC.

What QUIC changes conceptually

The main shift is not even in performance, but in the model of evolution. TCP was updated "by the whole Internet at once" — through kernels, through middleboxes, through decades; QUIC is updated like an application — by a library in one process. For the first time, the transport became software in the full sense: with release cycles, experiment flags, and rollback. The reverse side is fragmentation (every application has "its own TCP" with its own bugs and settings), the growth of CPU cost, and the loss of visibility for operators: the fact that middleboxes can no longer interfere is at the same time a protection of privacy and the end of a whole industry of traffic optimization. The dispute about whether this is good or bad is essentially a dispute about who owns the network: the endpoints or the infrastructure between them. The end-to-end architecture, formulated in 1981, has won a belated but decisive victory.

Real time and remote calls

Why real time renounces reliability

The requirements of live traffic are the opposite of file traffic: low delay, jitter stability, and synchronization are important, while the completeness of data is secondary. A video call frame that arrives 300 ms late is needed by nobody — the conversation has moved on; it is better to lose a frame and interpolate than to stop the stream for the sake of retransmission. Hence — RTP (Real-time Transport Protocol, RFC 3550, 2003; originally 1996), a superstructure over UDP adding exactly what datagrams lack for media: sequence numbers (to detect a loss and rearrange what arrived out of sync), timestamps (to restore the rhythm of playback and synchronize audio with video), and source identifiers; the control protocol RTCP in parallel exchanges quality statistics. A lost RTP packet is not retransmitted — the receiver conceals the loss (packet loss concealment): it repeats the previous fragment, interpolates, inserts silence. Between RTP and the application lies the jitter buffer: it smooths the unevenness of packet arrival at the price of a small fixed delay — a balance that adaptive buffers adjust on the fly.

The peak of this evolution is WebRTC (developed since 2011 under the aegis of the W3C and IETF): a stack in which media goes over SRTP on top of UDP, control channels — over DTLS, and arbitrary data — over SCTP-over-DTLS; ICE lays the path through NAT, and congestion control (GCC) is implemented on top — by delays and losses. Calls in a browser without plugins became possible precisely because the transport layer was allowed to be assembled anew from the UDP "construction set." To the same family belong the streaming protocols of the new generation — SRT and RIST in broadcasting, MoQ for live delivery over QUIC.

RPC: calling a function across the ocean

The second great application abstraction of the transport is the remote procedure call (RPC). The idea, proposed back in Bruce Nelson's dissertation (1981) and popularized by Sun RPC (1984), on which NFS runs: the call result = remote_function() must look local, while the transport, serialization of arguments (XDR, later protobuf and others), waiting for the response, and retries must hide inside. Behind the elegance stands the two generals problem already familiar to us: a call timeout is fundamentally indistinguishable from "the request did not arrive" and "the request was executed, the response was lost." A repeated call may execute the operation twice — debit the money twice. Therefore mature RPC systems introduce request identifiers and idempotency on the server side (a repeated request with the same ID returns the previous result instead of being executed again), and honestly document the semantics as at-least-once. The modern standard is gRPC (Google, 2015): protobuf serialization over HTTP/2 (multiplexed streams, bidirectional streaming), with HTTP/3 support. The microservice architecture is, in essence, the industrialization of RPC, and its main headaches (timeouts, retries, cascading failures) are direct consequences of the fundamental unreliability of the transport on which it stands.

QoS versus QoE: metrics and sensations

The quality of transport is measured in two coordinate systems. QoS (Quality of Service) is technical metrics: throughput, delay (RTT), jitter, losses. QoE (Quality of Experience) is what the user feels: page opening time, the number and duration of video "freezes," the intelligibility of speech in a call. The two systems do not reduce to each other: with formally good QoS the video may lag (the adaptive player misjudged the channel), and with bad QoS the user may be satisfied (the losses are concealed by the codec). The engineering practice of the 20th century optimized QoS; the streaming services industry of the 21st switched to QoE — because it is precisely the frequency of rebuffering, not the average bitrate, that predicts viewer churn. Measuring both is a science of its own: the network is non-stationary, the data are noisy, and the result depends on the point of measurement (active probes distort the picture, passive ones do not see the whole picture); the RTT measured by an ICMP ping and the RTT that the TCP handshake sees can differ because of traffic prioritization.

Security of the transport layer

SYN flood and SYN cookies

The handshake is the most vulnerable phase of a connection: upon receiving a SYN, the server allocates state (buffers, an entry in the queue of half-open connections) before it has convinced itself that the client exists. SYN flood clogs the queue with fake SYNs from nonexistent addresses — legitimate clients cannot connect. The attack is as old as TCP itself (one of the first loud cases was the Mitnick attack of 1995, which blinded a host with a flood before hijacking a connection). The countermeasure is SYN cookies (Daniel Bernstein and Eric Schenk, mid-1990s): when the queue overflows, the server stops storing state and encodes it right into its ISN — a cryptographic hash of the connection parameters and a secret key; the state is "restored from the packet" when the third step of the handshake arrives. SYN cookies entered Linux back in 2.0 and are enabled by default; modern variants (SYNPROXY) terminate the handshake at the firewall, not letting the flood through to the server at all. SYN flood today is a routine tool of the DDoS arsenal, neutralized at the infrastructure level.

Predictability and forgery: from Mitnick to RFC 5961

The sequence number prediction attack that made Mitnick famous was closed by ISN randomization (RFC 1948), but the related class is alive: "blind in-window attacks" — the injection of data, SYN, or RST on behalf of one of the parties by an attacker who does not see the traffic but has guessed the quadruple and a number within the receive window. RFC 5961 (2010) tightened the rules: an RST outside the exact window is ignored; a suspicious one is answered with a challenge ACK; data in the window outside the expected one is discarded. Only channel cryptography fully solves the problem — TLS makes injection meaningless (fake bytes will not pass the MAC check), and QUIC closes the metadata too. For the control traffic of infrastructure (BGP sessions) there exists a separate mechanism — TCP-AO (RFC 5925, 2010), the heir of TCP MD5 (RFC 2385, 1998): authentication of every segment with a shared key.

Ports as an attack surface

Ports are not only addressing, but also a vulnerability. Ephemeral ports are exhausted on proxies and balancers (tens of thousands of TIME-WAIT against ~28 thousand ports per pair of addresses); collisions of ephemeral ports in their time allowed attacking DNS by predicting the source port of the request — paired with guessing the transaction ID this gave the famous Kaminsky attack (2008), after which DNS resolvers randomized both the ports and the identifiers (the recommendations are RFC 6056). Port scanning is the basic reconnaissance of any intrusion. A separate relic is the privileged ports below 1024 in Unix: historically only a process with superuser rights could occupy them, and this served as a mechanism of trust — for example, rlogin took seriously only connections originating from a low port; the mechanism long ago became an anachronism, but by inertia lives on in systems.

Transport without encryption as the norm of the past

It is worth recording the shift that happened before our eyes. Until the mid-2010s, transport metadata — who, to where, with packets of what size, at what speed — was open to all observers along the way; on this were built both optimization (proxies, caches, PEP) and surveillance (traffic analysis reveals services even inside encrypted TLS, by the sizes and rhythm of packets). TLS 1.3 (2018) hid the handshake; QUIC — almost the entire transport header. The network loses observability; the endpoints gain privacy; operators respond with attempts to classify traffic by indirect signs and to block the "incomprehensible" (the story with the slowing of UDP traffic in certain networks is an episode of this struggle). Transport security has ceased to be an add-on and has become its property — and this is, perhaps, the most important change of the layer in thirty years.

The path of one connection: a synthesis

Let us gather all the mechanisms into one story — the life of a typical HTTPS session of 2026, in which a user opens a page.

Everything begins before the transport: DNS (a UDP datagram to port 53 — a short life without connections) returns the address of the server. The client chooses an ephemeral port (for example, 52431 from the range 32768–60999) and forms the quadruple — the future identifier of the connection. The first packet is a SYN with an ISN generated as a cryptographic hash of the quadruple and a secret key, and with options: MSS 1460, window scale (multiplier 128 — the base 64 KB has long been too little on gigabit channels), SACK permitted, timestamps. The server answers with a SYN-ACK with its own ISN; the client sends an ACK — three packets, one RTT, the connection is ESTABLISHED. If TCP Fast Open is enabled and the cookie is at hand — the data will go already in the first SYN. On top of this, TLS 1.3 starts immediately (another RTT) — and only then the first byte of the HTTP request.

Data transfer is the work of two windows. The stream is cut into segments by MSS; each goes out as an IP packet (if necessary — through TUN/provider tunnels, where the MSS is squeezed by clamping); the network card computes the checksums and glues the segments (TSO). Slow start doubles the window every RTT — the first 10 segments (IW10) go out at once; the window hits ssthresh; the growth becomes linear: the phase of congestion avoidance begins, the AIMD "sawtooth." The receiver acknowledges bytes cumulatively, postponing the ACK up to 40 ms in the hope of a piggyback; the receiving application announces the window — flow control insures against the overflow of its buffers. Somewhere in the middle of the download, the provider's router drops a segment: the receiver three times acknowledges the previous boundary — fast retransmit sends a copy without waiting for the RTO (which is anyway adapted to the measured SRTT + 4·RTTVAR); SACK reports that the "tail" beyond the hole has already been received — only the missing segment is retransmitted; fast recovery cuts cwnd in half and continues work. If the server is behind a CDN with BBR on board, it will endure the loss differently: the algorithm will compare in-flight with its estimate of the BDP and correct the speed by the model, not by the fact of the drop.

The request is served, the page is received. The client calls close() — a FIN goes out; the server, having given the last bytes, answers with an ACK, then with its own FIN; the client acknowledges and enters TIME-WAIT for 60 seconds — a quarantine against stray segments. On the server, which closed passively, the CLOSE-WAIT state will last exactly as long as the application does not call close(). After a minute, the client's quadruple is free for a new life — and in the provider's logs, several thousand packets remained from the connection, each of which passed through all the machinery described in this article: sequence numbers and cumulative acknowledgments, two windows and the "sawtooth," timers and probes, network card offloads and the state table. If, however, both the client and the server understand HTTP/3 — this whole scene did not exist at all: instead of it — one QUIC connection, whose TLS handshake took one RTT (and a repeated one — zero), whose streams do not block each other, and which will survive the client's switch from Wi-Fi to a cellular network without noticing the change of IP address.

Conclusions

The transport layer is the place where the Internet becomes usable for applications, and the place where its fundamental contradictions manifest themselves most sharply.

Reliability is assembled from unreliability — and runs into a theoretical limit. The sliding window, cumulative acknowledgments, adaptive timers, selective repeat — from these mechanisms TCP builds a guaranteed byte stream on top of a mail that loses letters. But the two generals problem reminds us: the absolute is unattainable; applications are doomed to idempotency and "at least once" — and mature systems are designed with this knowledge, not against it.

The minimalism of UDP turned out to be a strategy. The eight-byte protocol "guaranteeing nothing" became the platform of the era: RTP and WebRTC, DTLS and, above all, QUIC — all are built from the UDP construction set. When the transport froze in kernels and middleboxes, the only free layer turned out to be the simplest one.

Congestion control is precisely the management of the Internet. Jacobson's AIMD turned billions of selfish flows into a self-organizing system with provable convergence to fairness; the Mathis formula is its passport: the speed runs into the square root of the losses and is inversely proportional to the RTT. The history of the algorithms — Tahoe, Reno, Vegas, CUBIC, BBR — is the history of the search for a congestion signal: from losses to delays, from delays to a measurable model of the channel.

The practice of the layer is full of traps that are understood only from its structure. TIME-WAIT is an immunity, not a disease; CLOSE-WAIT is a bug of the application, not of the network; "sticky packets" are not a bug of TCP, but the duty of the application protocol to do framing; Nagle together with delayed ACK is a known combination that breaks interactivity (treated with TCP_NODELAY); performance is the BDP against the sizes of buffers and windows, and at modern speeds — also the price of system calls, copies, and interrupts (from epoll and io_uring to DPDK and RDMA).

Ossification defeated TCP — and gave birth to QUIC. The protocol that served as the backbone of the Internet for forty years lost the ability to change: kernels are updated over decades, middleboxes block the unfamiliar, options are not deployed. The answer is moving the transport into user space, on top of the untouchable UDP, under full cryptography: QUIC is updated like a library, migrates between networks by connection ID, eliminates inter-stream blocking, and already supports more than 40% of sites. For the first time, the transport became ordinary software — with all the benefits of fast releases and all the risks of fragmentation.

Security moved inside the transport. From SYN cookies and ISN randomization, born of the attacks of the nineties — to TLS 1.3 and the encrypted headers of QUIC: the layer that for half a century was transparent by design became opaque by necessity. Privacy versus observability is the new watershed along which the next evolution of networks will run.

The final picture of the layer is this: TCP is an adaptive control system that observes the network through ACKs and losses, makes decisions about windows and retransmissions, and continuously corrects its own behavior; UDP is the minimal framework from which every application builds a transport for itself; and QUIC is the proof that even a forty-year standard can be reassembled anew if you hide it where the network can no longer reach. The next step up is the application layer: the protocols for the sake of which all this machinery exists. HTTP and DNS, mail and streaming, CDN and P2P — they all speak in the ports and sockets, windows and segments described here, and, as the twelfth "truth of network design" from the April Fools' RFC 1925 proclaims (tracing back to Saint-Exupéry): perfection is achieved not when there is nothing left to add, but when there is nothing left to take away. The transport layer is the best proof of this maxim in the Internet: both TCP of 1981, and the eight-byte UDP, and QUIC, which removed from sight everything that could be removed.