The Internet is Cool. Thank you, TCP

The internet is incredible. It’s nearly impossible to keep people away from. But it can also be unreliable: packets drop, links congest, bits mangle, and data corrupts. Oh, it’s dangerous out there! (I’m writing this in Kramer’s tone)

So how is it possible that our apps just work? If you’ve networked your app before, you know the drill: socket()/bind() here, accept() there, maybe a connect() over there, and it just works. Reliable, orderly, uncorrupted data flows to and fro.

Websites (HTTP), email (SMTP) or remote access (SSH) are all built on top of TCP and just work.

Why TCP

Why do we need TCP? Why can’t we just use the layer below, IP?

Remember, the network stack goes: Physical –> Data Link (Ethernet/Wi-Fi, etc) –> Network (IP) –> Transport (TCP/UDP).

IP (Layer 3) operates at the host level, while the transport layer (TCP/UDP) works at the application level using ports. IP can deliver packets to the correct host via its IP address, but once the data reaches the machine, it still needs to be handed off to the correct process. Each process “binds” to a port: its address within the machine. A common analogy is: the IP address is the building, and the port is the apartment. Processes or apps live in those apartments.

Another reason we need TCP is that if a router (a piece of infra your average user does not control) drops packets or becomes overloaded, TCP at the edges (on the users’ machines) can recover without requiring routers to participate. The routers stay simple, the reliability happens at the endpoints.

Packets get lost, corrupted, duplicated, and reordered. That’s just how the internet works. TCP shields developers from these issues. It handles retransmission, checksums, and a gazillion other reliability mechanisms. If every developer had to implement those themselves, they’d never have time to properly align their flexboxes, a truly horrendous alternate universe.

Jokes aside, the guarantee that data sent and received over a socket isn’t corrupted, duplicated, or out of order, despite the underlying network being unreliable, is exactly why TCP is awesome.

Flow and Congestion Control

When you step back and think about network communication, here’s what we’re really trying to do: machine A sends data to machine B. Machine B has a finite amount of space and must store the incoming data somewhere before passing it to the application, which might be asleep or busy. This temporary storage takes the name of a receive buffer and is managed by the kernel:

sysctl net.ipv4.tcp_rmem => net.ipv4.tcp_rmem = 4096 131072 6291456, a min of 4k, default of 128k and max of 8M.

The problem is that space is finite. If you’re transferring a large file (hundreds of MBs or even GBs), you could easily overwhelm the destination. The receiver therefore needs a way to tell the sender how much more data it can handle. This mechanism is called flow control, and TCP segments include a field called the window, which specifies how much data the receiver is currently willing to accept.

Another issue is overwhelming the network itself, even if the receiving machine has plenty of buffer space. You’re only as strong as your weakest link: some links carry gigabits, others only megabits. If you don’t tune for the slowest link, congestion is inevitable.

Fun fact: in 1986, the Internet’s bandwidth dropped from a few dozen KB/s to as low as 40 bps (yes, bits per second! yes, those numbers are wild!), in what became known as congestion collapse. When packets were lost and systems retried sending them, they made congestion even worse: a doom loop. To fix this, TCP incorporated ‘play nice’ and ‘back off’ behaviors known as congestion control, which help prevent the Internet from clogging itself to death.

Some Code: A Plain TCP Server

With all low-level things like TCP, C examples are the way to go. Just show it like it is.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <signal.h>

int sockfd = -1, clientfd = -1;
void handle_sigint(int sig) {
printf("\nCtrl+C caught, shutting down...\n");
if (clientfd != -1) close(clientfd);
if (sockfd != -1) close(sockfd);
exit(0);
}

int main() {
signal(SIGINT, handle_sigint);
sockfd = socket(AF_INET, SOCK_STREAM, 0);
int opt = 1;
// SO_REUSEADDR to force bind to the port even if an older socket is still terminating (TIME_WAIT)
setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));

struct sockaddr_in addr = { .sin_family = AF_INET, .sin_port = htons(8080), .sin_addr.s_addr = INADDR_ANY };
bind(sockfd, (struct sockaddr*)&addr, sizeof(addr));
listen(sockfd, 5);
printf("Listening on 8080...\n");

clientfd = accept(sockfd, NULL, NULL);
char buf[1024], out[2048];
int n;
while ((n = recv(clientfd, buf, sizeof(buf) - 1, 0)) > 0) {
buf[n] = '\0';
int m = snprintf(out, sizeof(out), "you sent: %s", buf);
printf("response %s %d\n", out, m);
send(clientfd, out, m, 0);
}
close(clientfd); close(sockfd);
}

This create a TCP server that echoes what the client sends prefixed with ‘You sent:’.

1
2
3
4
5
6
# compile and run server
gcc -o server server.c  && ./server
# connect client
telnet 127.0.0.1 8080
# hi
# you sent: hi

127.0.0.1 (localhost) could be replace with a remote IP and it should work all the same.

We used the following primitives/functions follow the Berkley Socket way of doing things (released with BDS 4.2):

• SOCKET: create an endpoint (structure in the kernel).
• BIND: associate to a port.
• LISTEN: get ready to accept connection and a specify queue size of pending connection (beyond that size, drop!)
• ACCEPT: accept an incoming connection (TCP Server)
• CONNECT: attempt connection (TCP client)
• SEND: send data
• RECEIVE: receive data
• CLOSE: release the connection

In the example above, we’re using client/server dynamics in a request/response pattern. But I can add the following after send:

1
2
3
4
send(clientfd, out, m, 0);
sleep(5);
const char *msg = "not a response, just doing my thing\n";
send(clientfd, msg, strlen(msg), 0);

Compile, run, and telnet:

1
2
3
4
5
client here
you sent: client here
client again
not a response, just doing my thing
you sent: client again

I typed in the telnet terminal: client here, then client again. I only got you sent: client here, then the server was sleeping. My second line, client again, was patiently waiting in the receive buffer. The server sent not a response, just doing my thing, then picked up my second TCP packet and replied with you sent: client again.

This is very much a duplex bidirectional link. Each side sends what it wishes, it just happens that at the beginning, one listens and the other connects. The dynamics afterwards don’t have to follow a request/response pattern.

Catfishing Curl: A Dead Simple HTTP Server

Let’s create a very simple HTTP/1.1 server (later versions are trickier).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
// same as before
printf("Listening on 8080...\n");
int i = 1;
while (1) {
clientfd = accept(sockfd, NULL, NULL);
char buf[1024], out[2048];
int n;
while ((n = recv(clientfd, buf, sizeof(buf) - 1, 0)) > 0) {
buf[n] = '\0';
int body_len = snprintf(out, sizeof(out), "[%d] Yo, I am a legit web server\n", i++);

char header[256];
int header_len = snprintf(
header, sizeof(header),
"HTTP/1.1 200 OK\r\n"
"Content-Type: text/plain\r\n"
"Content-Length: %d\r\n"
"Connection: close\r\n"
"\r\n",
body_len
);
printf("header: %s\n", header);
printf("out: %s\n", out);
send(clientfd, header, header_len, 0);
send(clientfd, out