Replaying

A call that fails keeps what it ran with and a tape of its answers from outside the service. crab replay runs it again on your machine from those, so a failure in production reproduces locally, with its logs.

What a failed call keeps

• Only a call whose tool failed keeps them. A call that answered, refused its caller or was refused by the host keeps neither.
• The input: the image's digest, the tool, its arguments, its caller and its daily client key. The caller includes the outside IP when the host supplied it.
• Older inputs without a daily client key replay with client() returning None.
• The tape: every syscall in order, with a sha256 of the request as the service made it and the answer it got, the clock and the random source included. A request is recorded before its secrets are substituted, so no secret is on the tape.
• A tape holds at most 4 MiB of answers. Past it, it stops recording and is marked cut short.
• A host keeps at most 64 MiB of uncollected tapes per service, dropping the oldest past it. Cloud pulls failed-call inputs and tapes into private object storage, retaining them for 30 days even after a service moves or is removed.
• Reading them takes an owner of the service's account, or a key bound to it.

Replaying from the tape

Find the call's id with crab calls, then replay it:

crab calls notes --outcome failed
crab replay notes 1234

crab replay reads the call from the context in use, fetches the image it ran from the host and checks its digest, then runs the call in the CLI's own process. Each syscall is answered from the tape, and the service's logs print as it runs.

• A request whose digest differs from the recorded one stops the replay, naming the syscall: the code took another path than it did in production.
• A tape cut short is replayed as far as it goes.
• It exits 2 when the call fails or traps again. A call that answers prints its result and says the call it replays failed.

Checking a fix

From the service's crate:

crab replay notes 1234 --build

--build builds the current crate as crab build does and replays the same input and tape against it. It prints the recorded and built image digests. The built image may differ; syscall requests must still match the tape. A mismatch names the syscall where the code took another path. A successful call exits 0; another failure or trap exits 2.

--build and --on exclude each other.

Replaying against another host

crab replay notes 1234 --on default

--on runs the call's tool, arguments and caller on that context's host, against its own state, with no tape. It warns when the service deployed there runs another digest than the call did. --on default is the local host.

• Calls and logs
• Developing