Sockets, Buffers, and the Stream

Be conservative in what you do, be liberal in what you accept from others.

— Jon Postel, RFC 793

Every program that talks to a network eventually calls a function named something like recv(). It’s entirely possible to write networked software for years without asking what that function does, what it’s reading from, or why every serious library wraps it in a “buffered” something-or-other. This is the article I wish I’d read before that.

It starts below the socket, with what the network actually delivers, and works up through the socket, the buffers on both sides of it, the protocols that give the bytes meaning, and the async machinery most of us meet the socket through. Each section leans on the one before it. The margin notes hold the history and the commentary; skip them and the article still works.

1. The stream

The network is made of packets, and your program will never see one. The wire carries packets, chunks of at most about 1,500 bytes on Ethernet, roughly 1,460 of them yours once the headers are paid for. Packets get lost. They arrive out of order. Occasionally they arrive twice. That’s the deal you make for routing data across tens of thousands of independently owned networks with no central coordinator.Vint Cerf and Bob Kahn published the design that became TCP in 1974 as a single protocol that both routed packets across networks and made them reliable. In 1977 Jon Postel argued that this violated layering: the reliability machinery belonged in the endpoints, not the network. So it was split. IP delivers packets and promises nothing; TCP, sitting on top, turns them into a stream.

TCP’s job is to hide all of it. It numbers every byte, has the receiver acknowledge what arrived, retransmits what didn’t, and holds early arrivals until the gaps fill. What it presents to your program is a stream: an ordered, reliable, unbroken sequence of bytes with no trace of how they were chopped up in transit. A 10,000-byte message leaves as seven-ish packets and arrives as whatever the receiver asks for: 4,096 bytes on one read, 2,000 on the next. Nothing in your program needs to know the packet size. If you catch yourself reasoning about “the TCP limit” while sizing an array, stop: that limit lives below the floor you’re standing on.

An IP address says which machine. A port, a 16-bit number from 0 to 65535, says which program on it. A connection is identified by four things together, the four-tuple:Whenever a limit in this subject looks like it was pulled from a hat, check whether the hat is a power of two. 65,535 is 2¹⁶ − 1: a port has to fit in the sixteen bits the TCP header gives it, and that’s the biggest number sixteen bits can hold. The same shape explains the 255 in every IPv4 address and the 4,096 that turns up in §3. Computers don’t have a favourite number. They have a favourite shape, and it’s 2ⁿ − 1.

(203.0.113.7 : 51384)  →  (198.51.100.20 : 443)     // client addr:port, server addr:port

The server’s port is the same for every visitor, but each visitor comes from a different address, or at least a different ephemeral port, which the kernel picks for the client’s end, so ten thousand connections to port 443 stay distinct. Change any one of the four and it’s a different connection. One rule to file away: only one program can listen on a given port on a given address. A second gets “address already in use.” §6 explains how machines appear to break that rule.

2. The socket

A socket is an endpoint for communication that the kernel creates and manages on your behalf. When your program calls socket(), the kernel allocates an object internally and hands back a small integer, a file descriptor. The kernel keeps the connection state, the peer’s address, and its own queues of bytes going out and coming in. You hold a handle.In the early 1980s DARPA paid Berkeley’s Computer Systems Research Group to put TCP/IP into a Unix that universities could actually get. BBN (Bolt Beranek and Newman, the Cambridge, Massachusetts firm that had built the ARPANET’s original packet switches in 1969) had already written an implementation under a separate contract and handed it over. Berkeley, being Berkeley, rewrote it, and it ran faster, which BBN was not thrilled about. The result shipped in 4.2BSD in August 1983 with the sockets interface on top; Kirk McKusick’s essay “Twenty Years of Berkeley Unix” tells the story from the inside. Windows later got a near-verbatim copy as Windows Sockets, and a networking tutorial from 1985 mostly still compiles.

Making it a file descriptor was the best decision in the API’s history. Unix already had read(), write(), and close() for files, terminals, and pipes, and sockets fit the same mould, so code that reads from a disk can read from a machine on another continent without knowing the difference.

Getting to “connected” is a small, asymmetric dance:“Handshake” is borrowed from electrical engineering, where it meant the exchange of control signals two devices go through before either sends data: a modem’s screech was literally its handshake, and serial cables carried wires named “request to send” and “clear to send.” TCP’s version is a three-step nod, SYN, SYN-ACK, ACK: I’d like to talk, here’s my starting number. Fine, here’s mine. Got it. Three, because after two the server has no idea whether its reply arrived. Ray Tomlinson, who also put the @ in email addresses, gets the credit for working out that it had to be three.

// server
int s = socket(AF_INET, SOCK_STREAM, 0);
bind(s, &addr, sizeof addr);     // claim port 8080
listen(s, 128);
int c = accept(s, NULL, NULL);   // waits for a client; c is the conversation
recv(c, buf, sizeof buf, 0);  send(c, reply, len, 0);  close(c);

// client
int s = socket(AF_INET, SOCK_STREAM, 0);
connect(s, &server_addr, sizeof server_addr);  // the handshake
send(s, request, len, 0);  recv(s, buf, sizeof buf, 0);  close(s);

send() and recv() are write and read with an extra flags argument. bind() is where “address already in use” comes from. And note that accept() returns a new socket for each client; the original keeps listening, and “socket” means both, a naming crime we’re stuck with.

socket() takes three arguments, and they settle a question that trips people up: if sockets are “protocol agnostic,” why pick a protocol? The first argument is the domain: AF_INET, AF_INET6, or AF_UNIX for the local machine. The second is the type: SOCK_STREAM for a reliable byte stream, SOCK_DGRAM for individual messages, SOCK_RAW for building packets yourself. The third is usually 0, meaning “the obvious one.” So: a socket knows nothing about application protocols. HTTP, SMTP, your game’s message format, the socket has no idea any of it exists; it ships bytes. But you do choose the transport, and the kernel implements it. Agnostic about what the bytes mean; opinionated about how they’re delivered.In OSI terms: layers 1 through 4, physical, data link, network, and transport, live in the kernel. Layers 5 through 7, session, presentation, and application, which the internet protocols collapse into a single “application” layer, live in your program. The socket is the line between 4 and 5, not a layer of its own. Picture the counter at a post office. Everything behind it, the sorting, the trucks, the planes, the carrier’s route, is the postal system’s business: layers 1 to 4. Everything in front of it, what you wrote, what language it’s in, whether it’s a bill or a love letter, is yours: 5 to 7. The counter is where envelopes change hands, and nobody would call the counter a stage of delivery.

That also places it in the layer diagrams. The socket isn’t the transport layer; TCP and UDP are, inside the kernel. The socket is the doorway your program uses to reach them, and two kinds of socket prove the doorway isn’t nailed to one layer. A raw socket opens onto IP directly, which is how ping and traceroute work and why they need root. A Unix domain socket has no network layers at all: it’s named by a filesystem path like /var/run/docker.sock, the kernel copies bytes straight between processes, it’s faster than TCP to localhost, it can be permission-controlled like a file, and it can pass an open file descriptor to another process, which no network socket can. Docker, PostgreSQL, systemd, and D-Bus all use them, and they shipped in the same 1983 release as the internet domain; they are not a later add-on.

3. The three buffers

Your program can’t touch the network card. It runs in user space; hardware happens in kernel space; and crossing between them is a system call: switch modes, save registers, run kernel code, copy results back, switch back. One costs a few hundred nanoseconds to a microsecond. Reading a value already in your memory costs a few nanoseconds. Call it a hundredfold. Every buffer in this section exists to avoid crossing that line more often than necessary, and there are three of them between the wire and your parser.

The first two live in the kernel. Every socket has a receive queue and a send queue. An incoming byte goes network card, driver, IP, TCP reassembly, receive queue, and waits there until your program calls recv(), when the kernel copies bytes into your memory. By then it’s bytes, not packets: headers stripped, order restored. The kernel sizes the queue automatically and grows it for a fast connection with a slow reader.Flow control was in TCP from the 1974 paper. Congestion control was not, and in October 1986 the internet found out the difference: a link between Lawrence Berkeley Laboratory and the Berkeley campus, rated at 32 kilobits per second, slumped to about 40 bits per second as every host retransmitted into a network that was already full. Van Jacobson and Mike Karels diagnosed it and added slow start and congestion avoidance, written up at SIGCOMM in 1988. Flow control protects the receiver from being overrun. Congestion control protects the network.

Suppose your program stops reading. The queue fills. What happens to the bytes still arriving? Mostly they don’t arrive, on purpose. Every acknowledgment your kernel sends carries a number, the receive window: “I have this much free space.” The sender may never have more than that many unacknowledged bytes in flight. Read, and the window grows. Stop reading, and it shrinks to zero and the sender stops completely, probing now and then to ask whether space has opened. The data waits in the sender’s send queue, and if that fills too, the sending program blocks in send(). Back-pressure travels the whole way to the other application, and its trigger is your program not reading, not packet loss. A packet that does arrive with no room is dropped, unacknowledged, and retransmitted later; that’s the safety net, not the mechanism.

The third buffer is yours, and it exists because of the hundredfold. A parser wants small, specific pieces: a line, then a line, then exactly 1,256 bytes. If each were a recv(), it would spend its life crossing into the kernel. So the program owns an array, say 4,096 bytes, plus a position marker. When the parser wants a line, it looks in the array first. Empty? One system call fills it with whatever the kernel has, often a whole header section and the start of a body. From then on “read a line” means “scan the array to the next \r\n and move the marker.” No kernel. Just memory.

It works until the marker reaches the end of what was fetched. That’s the buffer running empty, and it isn’t mysterious: the array held a finite slice of the stream, you consumed it, and the next read goes back to the socket for another slice, possibly a smaller one if that’s all the kernel had.

Why 4,096? Convention. It matches the memory page size and exceeds nearly any HTTP header section, so one system call typically covers all the small reads that follow. Go’s bufio defaults to 4 KB, Java’s BufferedReader to 8 KB, Node’s streams to 16 KB; bigger means fewer system calls and more memory per connection. What the size is not related to is packet size, because packets dissolved into a stream inside the kernel before your program saw a byte. Libraries wrap this arrangement in a buffered stream that presents the socket’s own read interface while doing far fewer real reads. Hold onto that object; it’s the answer to a puzzle in §5.

4. Framing, with HTTP as the example

A stream has no boundaries, which is the point of it and the source of the most common bug in hand-written socket code. Send three 100-byte messages and the receiver may get one 300-byte recv(), or 250 then 50, or 1 then 299. One send() is not one recv(). And localhost lies about this: on one machine sends usually arrive intact, so the code passes its tests and corrupts data in production the first time a boundary falls mid-message. Everyone does it once.

Since the stream won’t draw lines, the protocol must. There are only a handful of ways: delimiters (“a line ends at \r\n”), length prefixes (“the next 4 bytes say how long the message is”), fixed sizes, and self-describing chunks (“size, bytes, size, bytes, a size of zero”). Whatever the rule, a single recv() may return the tail of one message and the head of the next, so the parser keeps what it has in its buffer and acts only when a complete unit is present.Why \r\n and not just \n? Because the early protocols were designed to be typed at a Teletype, where “carriage return” and “line feed” were two separate mechanical actions: slam the print head left, then advance the paper. Telnet standardized the pair, SMTP and HTTP inherited it, and we’re still sending a byte whose job was to move a metal carriage. HTTP itself began, in 1991, as one line: the client sent GET /path, the server sent the document and hung up. No headers, no status codes. It was text because text is debuggable; everything since is elaboration, right up to HTTP/2, which went binary in 2015 and ended the era of typing at web servers.

HTTP is one such rule set, and it lives entirely in your program, not in the socket. A browser writes this into the socket:

GET /index.html HTTP/1.1\r\n
Host: example.com\r\n
\r\n

The server parses it by the rules, first line is method, path, version, then headers until the blank line, and writes back:

HTTP/1.1 200 OK\r\n
Content-Type: text/html\r\n
Content-Length: 1256\r\n
\r\n
<html>... exactly 1256 bytes ...

The parser knows where it is in the stream because it’s a state machine that remembers what it has done. Reading the status line. Reading headers until an empty line, noting Content-Length along the way. Reading a body with 1,256 bytes to go. Done; the next byte begins a new response. It’s one component reading in sequence, not different parts of the program grabbing their pieces, and it doesn’t know Content-Type is coming; headers arrive in any order, so it reads a line and then looks. Only when the whole response is assembled does it hand a structured object to the application. With HTTPS, TLS sits between HTTP and the socket, so the socket sees scrambled bytes; nothing else changes. You can do all of this by hand: nc example.com 80, type the request, read the reply. There is no better cure for the feeling that HTTP is something magical the browser does.

5. Many connections at once

By default a socket blocks: recv() on an empty queue parks the thread until something arrives, and so do accept(), connect(), and a send() into a full queue. Simple, file-like, and the reason a naive server needs a thread per connection, each parked in recv(). Ten is nothing. Ten thousand is a problem, because each thread reserves a stack and the scheduler has to juggle them all. This model ran from 1983 until it fell over.

The alternative starts with a flag. A non-blocking socket returns immediately, either with data or with an error literally named “would block.” One thread can now service many sockets, but not by looping over recv(), which pegs a CPU doing nothing. It needs to sleep until something is ready, which is what select(), poll(), and their descendants epoll (Linux) and kqueue (BSD, macOS) provide: hand the kernel a set of sockets, sleep, wake with a list of the ready ones, do the reads and writes that won’t block, repeat. That loop is the heart of nginx, Node.js, Redis, Go’s scheduler, and asyncio. Blocking, non-blocking, and “async” are not three kinds of socket: the first two are a flag on the same socket, and the third is a programming style built on the second.It fell over in public in 1999, when Dan Kegel put up a page titled “The C10K problem”: a cheap PC could plausibly serve ten thousand connections, and the operating system interfaces were what stopped it. select() took a bitmask of descriptors and rescanned all of them on every call, so its cost grew with connections rather than with the ones that had anything to say. kqueue (FreeBSD, 2000) and epoll (Linux, 2002–03) fixed that by letting you register a socket once. Twenty years later Jens Axboe’s io_uring went further, with submission and completion rings shared with the kernel so batches of reads need no system call at all. It’s a proactor, and the biggest change to Linux I/O in two decades.

The loop comes in two shapes, differing in who does the I/O and what the wake-up means.

A restaurant pager versus room service. The pager says your table is ready; you still walk over. Room service knocks when the food is already at the door. The reactor is simpler and lets you decide how to read, but each wake-up still costs a system call for the read itself; the proactor folds the read into the notification but pins your buffer until the kernel is done with it. Libraries pick one shape and emulate the other: libuv is a reactor on epoll and rides completion ports on Windows; .NET exposes BeginReceive-style completion and, on Linux, implements it over epoll, doing the recv() the instant the socket reports ready.

Now the puzzle. async/await, promises, and Begin/End pairs are wrappers over all of the above: a non-blocking socket plus a readiness or completion mechanism, with “wake me when ready” turned into “resume this function when ready.” Put the buffered stream from §3 in front of such a socket and follow one HTTP response through it. The parser asks for a line. The buffer is empty, so the stream issues a read; nothing has arrived, so the call returns “pending” and the callback fires later. Truly asynchronous. But that read pulled in 4 KB, the whole header section. The next request for a line is satisfied from the array: no socket, no kernel, finished before the call that started it returned.

In a completion-style API this is called completing synchronously, and .NET’s IAsyncResult has a CompletedSynchronously flag for exactly it: the Begin method invokes the callback right then, on the calling thread, instead of handing it to the event loop. Every header line completes that way. Only a body larger than what’s left in the buffer drains it and flips back to asynchronous.

IAsyncResult BeginRead(byte[] dest, int count, AsyncCallback cb, object state)
{
    if (buffered > 0) {                     // bytes already here: copy, call back now
        int n = Math.Min(count, buffered);
        Array.Copy(buffer, pos, dest, 0, n);
        pos += n; buffered -= n;
        var done = new SyncResult(n, state);  // CompletedSynchronously == true
        cb(done);
        return done;
    }
    return socket.BeginReceive(buffer, 0, buffer.Length, cb, state);   // this one waits
}

The flag exists because of a hazard. If the callback’s job is “process the line, then BeginRead the next,” and that also completes synchronously, then the callback calls BeginRead, which calls the callback, and every line adds a stack frame until the stack overflows. Well-behaved code loops on the synchronous path and reserves the callback for the case that actually waited. So the answer to “why did my asynchronous read complete synchronously?” is: a buffer above the socket already had the bytes, and going to the kernel for them would have wasted the exact thing buffers exist to save.

6. Sharing a port

Back to the rule from §1: one listener per port per address. Three things let a machine appear to break it.HTTP/1.0 had no Host header, so a server couldn’t tell which website a request was for, and the only fix was one IP address per site; hosting companies burned through IPv4 addresses running virtual hosts. HTTP/1.1, in 1997, made Host mandatory. Then TLS brought the problem back, because the certificate must be chosen before any HTTP is readable. Server Name Indication, first specified in 2003, has the client announce the hostname in the clear during the handshake, and it took about a decade of old browsers dying before everyone could rely on it.

First, one listener handles thousands of connections, because connections are identified by the four-tuple and accept() hands each one to the listener as its own socket. “One program per port” and “ten thousand users on 443” were never in conflict.

Second, when a machine hosts several services behind 443, one program listens and it’s a traffic director. nginx, HAProxy, or a load balancer routes each request by the Host header, by path, or for HTTPS by the server name the client announces during the TLS handshake, which lets the proxy pick a certificate before decrypting anything. The services themselves listen on 3000, 8080, or a Unix domain socket that the outside world never reaches.

Third, the rule is per address. A machine with several IPs, or a host full of containers each with its own, can have a listener on 443 for each. The one real exception is SO_REUSEPORT, which lets several processes of the same program share a listener while the kernel spreads new connections across them, for worker pools and zero-downtime restarts. On Linux they must run as the same user, precisely so a stranger can’t attach to your port and start receiving a share of your traffic.

7. Firewalls, and WebSocket as the worked example

Firewalls tend to block every inbound port except the ones explicitly opened, which for a web server means 80 and 443. Can’t those be attacked too? Yes. The logic isn’t that they’re safe; it’s that a port is only dangerous if something listens on it, so fewer open ports means fewer programs an attacker can reach. Port 22 isn’t risky; an SSH daemon with password logins might be. When a firewall drops everything but 443, it’s saying: only the web server is reachable, and the database, the admin panel, and the forgotten monitoring agent are not. That’s reducing the attack surface. The web server itself is a risk accepted on purpose: SQL injection and authentication bugs travel over ordinary HTTPS, and the firewall’s view of them is “a GET to 443,” which is exactly what it was told to allow. Other tools work at that layer.The perimeter firewall is, more or less, a reaction to November 1988, when Robert Morris’s worm spread through a large fraction of the internet by walking in through services listening on ports nobody was thinking about: a debug mode left on in sendmail, a buffer overflow in finger. The lesson drawn was that you can’t audit every listening program, so make most of them unreachable. Bill Cheswick and Steve Bellovin’s 1994 book Firewalls and Internet Security turned that into doctrine.

There’s a direction to it: inbound is blocked, outbound on 80 and 443 is allowed so people can browse. And because 443 is nearly always open outbound, everything now travels over it, VPNs, chat, updates, malware phoning home. Defenders answered with deep packet inspection and TLS interception; protocol designers answered by making their traffic look even more like browsing. That’s ossification: the network gets so tuned to the protocols it knows that anything new has to disguise itself as one of them.

WebSocket is the worked example. Despite the name, it isn’t a socket. It’s an application protocol over an ordinary TCP socket, named to sound like the raw socket browsers won’t give JavaScript. It starts as HTTP:

GET /chat HTTP/1.1\r\n
Upgrade: websocket\r\n
Connection: Upgrade\r\n
\r\n

The server answers 101 Switching Protocols, and from that byte both sides stop speaking HTTP. The TCP connection stays; the rules change. The new rules add what a raw socket lacks, framing: each message gets a two-to-fourteen-byte header with its length and type, so the library hands you whole messages. And unlike HTTP, the connection is persistent and full-duplex; either side sends whenever it likes, which is what chat, live dashboards, and multiplayer games need.

Why start as HTTP? Because a GET to 443 with normal headers sails through the middleboxes that would block a new protocol on a new port. The disguise is imperfect: some caching proxies kept treating the bytes as HTTP, and researchers showed a malicious page could craft WebSocket data that looked like an HTTP request and poison the proxy’s cache for other users. So every frame from a browser is masked, XOR-ed with a random key, so it can never resemble a valid request. Same socket, same three buffers; only the convention for what the bytes mean has changed, and a 4 KB read still cuts frames in half, so the library reassembles them like any other parser.

8. Misconceptions that survive first contact

send() returning means the kernel accepted the bytes, not that they were delivered. It copied them into the send queue. The only proof of delivery is an acknowledgment from the application on the other end. And send() may accept fewer bytes than you offered; you have to loop.The small-write delay is Nagle’s algorithm, from a 1984 RFC by John Nagle, who noticed a Telnet session sending one keystroke per packet: 40 bytes of headers per byte of payload. His fix, don’t send a new small packet while an earlier one is still unacknowledged, is still the default. Its bad interaction with delayed acknowledgments, where each side politely waits for the other, wasn’t his idea, and he has said so, repeatedly, on the internet.

A dead connection is silent. A crashed peer, an unplugged cable, and a NAT router that forgot the connection all look identical to “nothing to say,” for hours. Detecting death takes TCP keepalive, application-level pings (WebSocket’s ping frame exists for this), or a timeout on your reads.

Closing isn’t instant. The side that closes first parks the connection in TIME_WAIT for a minute or so, so late packets can’t be mistaken for a new connection. A client that churns through tens of thousands of short connections can run out of ephemeral ports this way, which is one reason connection pools and HTTP keep-alive exist.

Half-closed is a real state. shutdown() can say “I’m done sending” while still receiving, and on the other side recv() returning zero bytes means “they’re done,” not “error.” Code that never handles the zero spins or hangs.

Small writes can be mysteriously slow. By default TCP may hold a small write briefly to see whether more is coming, and combined with the receiver’s habit of delaying acknowledgments this can add tens or hundreds of milliseconds to a chatty protocol. TCP_NODELAY turns the holding off, and most latency-sensitive libraries set it.

9. References and further reading

• V. Cerf and R. Kahn, “A Protocol for Packet Network Intercommunication,” IEEE Transactions on Communications, 1974.
• RFC 793, “Transmission Control Protocol,” 1981; superseded by RFC 9293, 2022.
• M. K. McKusick, “Twenty Years of Berkeley Unix,” in Open Sources: Voices from the Open Source Revolution, O’Reilly, 1999.
• V. Jacobson and M. Karels, “Congestion Avoidance and Control,” SIGCOMM, 1988.
• RFC 2068, “Hypertext Transfer Protocol — HTTP/1.1,” 1997; current: RFC 9110 and RFC 9112, 2022.
• RFC 3546, “Transport Layer Security (TLS) Extensions,” 2003; current: RFC 6066, 2011.
• W. Cheswick and S. Bellovin, Firewalls and Internet Security, Addison-Wesley, 1994.
• RFC 6455, “The WebSocket Protocol,” 2011.
• RFC 896, J. Nagle, “Congestion Control in IP/TCP Internetworks,” 1984.
• Microsoft .NET documentation for IAsyncResult.CompletedSynchronously.

Further reading, in the order I’d read them.

1. Brian “Beej” Hall, Beej’s Guide to Network Programming. The shortest path from §2 to a program that runs.
2. W. R. Stevens, B. Fenner, A. Rudoff, UNIX Network Programming, Volume 1, 3rd ed., 2003. Where to go when Beej stops answering; every state in §8 has a chapter.
3. Dan Kegel, “The C10K problem.” The hinge into §5, written while it was happening.
4. Michael Kerrisk, The Linux Programming Interface, 2010, the chapters on sockets and alternative I/O models. The exact epoll semantics libraries rely on.
5. Jens Axboe, “Efficient IO with io_uring.” Where the proactor shape is going on Linux, from the person taking it there.