Letting services fetch any host without becoming an open relay

A service on CrabTalk declares the hosts it fetches in its crab.toml:

[http]
fetch = ["api.github.com", "*.example.com"]

A crawler, a link previewer or a feed reader cannot list its hosts in advance, so * matches every host. We asked whether to keep it. This post is what * does and does not open, and the one change we made.

What * does not open

Our network. On Cloud, every fetch goes through an egress client that resolves names itself and refuses loopback, private, link-local, CGNAT and unique-local addresses, the cloud metadata address 169.254.169.254 included. The check runs at resolution, on every connection, so a name that resolves to a public address once and a private one the next (DNS rebinding) is still refused. A literal IP is checked on the URL. Fetches do not follow redirects; a redirect is answered to the service, and following it is a new request, checked again.

Your secrets. A service names a secret as {secret:NAME} and the host writes the value in on the way out. It does so only for a host the secret's owner set for it under the secret destination rules, and those hosts are named in full: * is not accepted there. Each sealed value is bound to its host list with AES-256-GCM, so editing a stored row to add a host makes it fail to open. A service that declares * and sends {secret:KEY} to a host of its choosing is refused before anything is sent.

What * does open

Our IPs. A public service that declares * and takes a URL from its caller lets anyone point our nodes at any target: scraping, credential stuffing, floods. The target sees CrabTalk, not the service, and the cheapest defence it has is to block every CrabTalk node at once.

Banning * does not fix that. A service that lists a thousand hosts relays to those thousand hosts, and the uses that need * go elsewhere.

Naming the service on every request

Cloudflare Workers have the same problem, and their answer is a header: every subrequest a Worker sends with fetch() carries CF-Worker, set to the zone that owns the Worker. A site can log it, rate-limit on it, or block one zone instead of all of Cloudflare.

We did the same. Every request a service sends from Cloud now carries:

CrabTalk-Service: <owner id>/<name>

The node sets it after the service's own headers, replacing any value the service set, so a service cannot leave it off or name another. It carries the owner's id rather than its handle because a handle can change and the header should keep pointing at the same account. The same id resolves at /services/{owner}/{name}.

A site that sees abuse from CrabTalk can now block one service, and tell us which one.

What is next

The header makes a service identifiable; it does not stop one. A published abuse contact and a way for us to stop a single service are next.

What holds for every service today is on one page: Security.