Storage and recovery

How a node stores the chain is its own business. The physical layout does not change any protocol hash or charge. This page describes the reference node's choices.

In outline: block bytes go into append-only files, and everything else into one embedded database (two stores). For every block it executes, the node keeps the changes needed to undo it, so it can move to a heavier branch by undoing blocks back to the fork and connecting the other branch (connecting the best branch); a fork older than the changes it keeps is reached by a replay from genesis (deep reorganizations). After a crash, the node recovers on its own from what reached the disk (restarts).

Two stores#

• Block files. Canonical block bytes live in checksummed, append-only segment files under blocks/, each up to 128 MiB, or holding one larger block alone.
• The state database. An embedded redb database, chain.redb, stores where each block is, compact block metadata, the current records, the state-tree nodes, per-block changes and transaction indexes.

The whole state does not need to fit in memory. During execution, the VM works on a small overlay containing only the records it touches, with a journal for nested calls.

Block files#

Segment files are named blocks/00000000.dat, blocks/00000001.dat and so on. Each starts with a 40-byte header, the 8 bytes TINGBLK\0 followed by the 32-byte ChainId, and then holds one frame per block:

u64 little-endian length ‖ canonical block bytes ‖ H("node-block", block bytes)

H is the protocol's domain-separated hash. The block in a frame is at least 302 bytes, the size of an empty block with no producer data, and a frame never spans two files: a new file begins when the next frame would not fit in 128 MiB. A frame larger than that begins a file of its own and is the only frame in it, so no block is too large to store. A block's location in the database is a u32 file number, a u64 byte offset and a u64 block length, and the database's file cursor marks the end of the last indexed frame.

The node keeps the file it appends to open, and up to 16 files open for reading, whose frames it reads by position: opening a file costs more than the frame, notably where a scanner inspects every opening. It keeps in memory the blocks it stored last and the blocks larger than a slice it served last, up to 4,096 of them and 64 MiB: a block is connected right after it is stored, and a large one crosses to a peer in slices, so either is read from disk once at most.

The database#

chain.redb uses a bounded page cache, 256 MiB unless "db_cache_mib" in config.json says otherwise; a larger cache keeps more of the state tree in memory on a node with memory to spare. Its tables hold:

Hashes, record values and encodings keep their protocol meaning. An empty value is a present record, and absence is encoded separately. A value of up to 1 MiB is kept in its row; a larger one is split into parts of 1 MiB in value_parts, by its table, its row's key and the part's number, and its row holds only its length. No database value therefore nears redb's limit of 3 GiB, and the node stores a record of any size the protocol allows.

The tree is stored as it is defined: one entry for each node and one for each leaf, at the position where it sits, so about two entries per record. A leaf's entry holds its record key and leaf hash, so that moving a record up or down the tree never reads or hashes its value. Merkle paths are recomputed with the exact rules of the state tree.

Execution against the database#

The VM reads either an immutable database view or the view of a private write transaction, through a storage abstraction. Its in-memory overlay and nested-call journal hold only the records and Merkle paths it touches. Values are shared between the overlay and the journal rather than copied, and a length query or an overwrite reads only the record's length.

Validated programs stay decoded in memory across blocks, keyed by CodeHash, in a cache bounded by 256 MiB of decoded programs, counted by the memory each takes decoded, which is many times its code; the program used least recently leaves first. A program that is not cached is read and decoded again when it is called.

A database failure is a node failure. It never turns into a missing record that consensus would accept, nor into a transaction trap with an accepted receipt. No physical I/O, cache miss, reversed change or database operation changes C, S, D or G.

Per-block changes#

For every block it executes, the node keeps the block's undo data: the previous value of each record the block changed, once per key, including the manifests and code records the runtime manages. Changes that a call later reverted, and records that ended where they started, are left out. Undoing a block applies its undo data, and the root it reaches must be its parent's.

When it disconnects a block, the node first keeps the block's redo data, the values that block left, which it reads from the state before undoing it. Connecting that block again applies them without executing it; a block without redo data is executed again, which gives the same result.

Undo data is the largest thing a node keeps, and nothing in the protocol bounds it. What a block writes is paid for by the byte, in compute for each value's leaf hash. What it deletes or overwrites is not: a delete or an overwrite costs two state accesses whatever the size of the record it replaces (state access), and reads only that record's length. One block can therefore replace as many records as half its state-access capacity, about a thousand at the production capacities, each of any size, and its undo data is as large as they were.

The node therefore never holds a block's previous values in memory together. The changes a block's execution reports name each changed record and the value it has now, not the one it had. When the node connects the block it copies each previous value from the record's own rows to the block's undo rows, a part of 1 MiB at a time, before it writes the new one. Undoing a block moves values the same way, from the records to its redo rows and from its undo rows back to the records, and connecting it again moves its redo rows to the records; the leaf of each restored value is hashed as the value moves, and the root is checked once every record is in place, before the commit. A replay's undo data is copied between the two databases a row and a part at a time too. The memory one block takes is then the values it writes, which its compute bounds, one part of a value on its way, and the database's page cache, whatever the size of the records it replaces. The disk space it takes is not bounded that way: connecting a block that deletes a record writes a second copy of the value as undo data before the record's own rows are freed.

A node keeps undo data for a window, by default the last 43,200 canonical blocks, 30 days, and the redo data of disconnected blocks as long (deep reorganizations). "undo_window" in config.json sets the window in blocks; 0 keeps everything.

What a node holds of a block#

• Header. Checked and in the header tree, in memory only (headers first).
• Stored. The body is in a block file and matches its header, but has not been executed. Its location, header and work are in the database. A body is stored when the node asked for it along the best header's branch, or when a peer pushed it and it is executed at once; a pushed block whose branch does not overtake the tip is only held in memory (pushed blocks). Block files are only ever appended to, so what a peer can make a node write is what the node either asked for or executes.
• Valid. Executed on its branch, with its record changes kept. Only valid blocks are ever connected.
• Invalid. Its execution failed, or the operator marked it invalid, or it sits above such a block. It is never connected, and neither is anything built on it. A failed block's status is stored, and so is the operator's mark; what sits above follows from them, so a restart tries none of it again.

The block index#

The node keeps an index of the blocks it knows: for each, its header, its accumulated work, the limits it runs under and what the node holds of it. Choosing, downloading and connecting the best branch work from it. The index holds in memory the most recent canonical blocks, every stored side branch, and the headers whose blocks are not stored yet, which headers sync indexes only for a chain that overtook the best header (headers first). Canonical blocks further down are settled: the index reads them from the database when it needs them, their header and work from headers and their limits from block_limits. Its memory therefore does not grow with the length of the chain.

The index holds at least the last three voting epochs of canonical blocks, whose votes a header check counts, and eleven more, or the undo window if that is longer: 60,491 blocks on the production network, about 42 days. Below that, blocks settle 64 at a time once the tip has moved on, in one database transaction that records their limits and the new settled height. Only an executed, valid canonical block that the operator has not marked, and that the best header's branch also holds, settles.

Settled blocks are canonical by definition. So before anything at or below the settled height changes, the node holds the blocks above the new height in memory again, writing the lower height first. It does so:

• before it follows a branch that forks there;
• before the best header's branch forks there;
• before the operator marks or reconsiders a block there.

A reorganization of any depth therefore still works the same way, with a moment's reading from the database first.

A side branch whose blocks all lie at or below the settled height, some 42 days or more behind the tip on the production network, is forgotten as blocks settle and when the node opens. It leaves the index, and what is stored of its blocks leaves the database: their location and header, the occurrences of their transactions and their events, and any undo and redo data. Their frames stay in the block files, unreferenced, as the files are only ever appended to. Should such a branch ever be extended past the tip, it is fetched again like any other branch.

The node's status, ting status --json or GET /v1/status, shows settled_height, the height at or below which canonical blocks are settled, and held_headers, the index entries in memory.

Connecting the best branch#

Of the branches whose blocks are all stored and none invalid, the node connects the heaviest that either grows its tip with more work or exceeds the tip's work by the fork-choice margin. Most of the time that branch is the tip plus one new block, and connecting it executes that block; a heavier side branch that does not clear the margin never holds the tip back from growing. When it is another branch, the node moves to it:

1. Disconnect the connected blocks back to the last block both branches share, newest first, each by its undo data, keeping its redo data.
2. Connect the other branch's blocks, oldest first: a block with redo data, from an earlier visit to this branch, by applying it; any other by executing it against the state its parent left, checking every commitment, and keeping its undo data.

Every state root reached is checked against its header. The move goes in steps, each one database transaction: disconnecting up to 64 blocks, connecting up to 64 already-executed blocks, or executing one block. A reorganization of any depth therefore never holds more in memory than one block's written values and tree changes, with one part of a value on its way (per-block changes), and the node answers requests between steps. Between two steps the tip may briefly have less work than before; the next step continues toward the heavier branch. Until the move ends, every choice is measured against the tip it began from, not the one it has reached.

A block whose execution fails is marked invalid, with every block above it. The node then connects the heaviest branch that is left, by the same rule. That is the branch it came from unless another clears the margin against it, and going back is cheap, since that branch's blocks have their redo data. A branch that cannot be connected for another reason, such as damaged undo data, leaves the node on its tip with a message on standard error, at startup too.

Each reorganization, once the tip has settled, is reported on standard error with the number of blocks replaced and the height above which they were, and the last one appears in the node's status as reorganization: replaced, fork_height, height, tip, time and deep. One that replaces six blocks or more is a deep one, logged as a warning: it is valid, since reorganizations have no depth limit, but it means a network split healing, a miner with most of the work, or a fault, and a service that waits for confirmations may want to look.

Deep reorganizations#

A branch that forks below the window needs undo data the node no longer has, and so does one that forks below where the node pruned it while its tip stood higher, as before a reorganization: undo data once pruned stays pruned. The node then stays on its tip and, in the background, replays its canonical blocks from genesis into a temporary database next to its own, from block files alone, checking every state root on the way. It keeps the undo data of each block between the fork and the window as the replay passes it, copies it into its database a step at a time, and carries out the reorganization as any other. There is no depth limit: the deeper the fork, the longer the replay, never a refusal. A restart during the replay starts it again.

The temporary database holds a second copy of the state until the replay ends, so a node must have room for it. An archive node, "undo_window": 0, never needs a replay. The node's status shows undo_pruned, the height up to which canonical blocks have no undo data, and replaying while a replay runs.

Warnings#

The node's status lists warnings that need its operator. There is one: a branch that failed validation here carries more than six blocks of work more than the tip. The work counted is that of checked headers, so it cannot be claimed without being done. Either the branch is invalid and most of the work goes to it, or this node follows other rules than much of the network, as after a fault in it or in an update. The warning is also logged once, and goes away when the tip comes within six blocks of the branch's work.

Overriding validity#

An operator can override what the node accepts, for the rare case of a fault in the node itself, through the control channel:

• ting invalidate BLOCK marks a block invalid, by id or by the height of a connected block. The node disconnects it and every block above it if they are connected, then connects the best branch that is left, and never connects the marked block or anything above it. A block not seen yet can be marked too: it is refused when it comes. The mark is kept across restarts. Peers that send a marked block, or one above it, are answered with block marked invalid by this node's operator: a refusal of this node's own, held against no peer.
• ting reconsider BLOCK clears the invalid status of the block, of every block above it and of every block below it, whether the operator marked them or their execution failed, and connects the best branch again. A block executed before keeps its record changes, so a reconsidered branch is reconnected without executing it again; a failed block is executed again, and if it fails again, it is invalid again.

So a node that accepted a block it should not have can be moved off it at once, without waiting for a fixed release; and one that refused a valid block, as before a fix, can take it again without being synchronized from scratch. The genesis block cannot be marked. A change that moves the tip far may take longer than the CLI waits for an answer; the node carries on regardless, and ting status shows where it is.

Commits and durability#

Readers see the old committed view until the new transaction commits, never a mix: REST's reader threads read beside the chain thread this way. A block executes against the last commit, which its own step has not yet changed, through a read view whose tables open once; its changes are then written in the step's transaction. No write by a program commits on its own: its effects reach the database only through the block's transaction.

A block's frame is always forced to disk before its commit. The commit itself is forced to disk only:

• for the first block after the node opens;
• for the first block in each new segment file;
• otherwise, at least every five minutes.

Forcing every commit would take most of the time a block's state writes take. Commits that store no block, those of a reorganization or of the operator, follow the same five minutes, the first after opening forced, and the moves the node makes as it opens are forced before it serves anyone. Commits left unforced are forced once five minutes have passed, whether or not a block arrives. redb's quick repair is enabled.

After a crash, the database holds the last forced commit, and every frame written since lies in that commit's segment or the next one, where recovery finds and replays them (restarts). A clean shutdown forces everything. If a commit fails, the chain thread stops: the node is reopened and recovers, instead of guessing which commit reached stable storage. A failed import that leaves an unindexed frame at the end of the block files also stops the node until it is restarted.

Restarts#

On restart the node opens its stored validated state and loads the metadata of the blocks its index holds in memory: the canonical blocks above the settled height, and the side blocks side_blocks lists. It reads no settled block, no historical block body and no transaction, so a restart takes the same time however long the chain is. A database from before side_blocks fills it once, from every stored header, and one from before the event tables fills them once, from every executed block, a thousand blocks to a commit. One from before every executed block kept its limits stores those of the blocks its index holds, once; settled blocks' are there already.

Before it serves anything it checks that:

• every header it loads hashes to its block id, carries this ChainId and has a block location;
• the stored tip matches the canonical mapping, which has no gaps, and every canonical block it holds has the parent and height that follow from the one before, the first from the highest settled block;
• every block it loads has the height, proof of work and accumulated work that follow from its parent, and every canonical block is valid;
• the stored record count equals the number of records and of record lengths, and the state root matches both the tip's header and the persisted root node.

A stored block whose parent was only a header when the node stopped, as blocks arrive out of order, waits outside the index until that header comes again, and is then indexed with it. One at or below the settled height, with no block above that height waiting on it, is forgotten like a side branch that far behind.

Frames appended after the file cursor, left by a crash before their commits reached the disk, are stored again in the order they were written, without appending another copy. One whose parent comes in a later frame, as blocks arrive out of order, is tried again once the others are in, for as long as that stores more; one still left over, such as one whose parent was only a header, is passed over, and its block is fetched again. Nothing refers to a frame after the cursor, so a damaged one (incomplete, zero-filled as a power loss may leave it, or with a wrong checksum) is truncated with everything after it in its file, and its block is fetched again. A block file that nothing indexed lies in yet may have lost its header the same way, cut short or zero-filled: the node writes it again. A header that is neither this chain's nor empty is refused, as below.

Then the operator's marks are applied, and the node connects the heaviest stored branch, as in connecting the best branch: after a crash in the middle of a reorganization, it simply carries on. Last, what lies below the held heights settles: all of it at the first start of a node whose index held every block.

Indexed frames are checksum-verified when they are read, rather than scanned at every restart; a stored block whose frame no longer reads back is forgotten, its header kept, with its limits and any undo and redo data, and fetched again. A block is read before it is connected, so one that cannot be read is never connected. The database's own checksums are responsible for physical integrity.

A database or block file made for a different storage format or a different genesis is rejected, never reset.

Reading history#

Block bodies are read from disk on demand, or from the blocks kept in memory. Transaction lookups use the occurrence index and keep only occurrences on the canonical chain, so a transaction id that appears in several blocks lists every occurrence, page by page.

A program's events are read the same way: the event tables give, in order of height, the blocks with events of that program, or of one of its topics, and only those that are canonical count; the page then reads each such block once for the events' transactions and payloads. An executed block's rows are written in the commit that stores its execution, and a reorganization changes none of them, so reading events costs in proportion to the events found and the blocks they lie in, never to the length of the chain.

The mempool is not stored; deferred transactions are, in deferred.json, as described in Mining and voting. A deferred.json that does not parse is moved to deferred.json.corrupt, with a message on standard error, and the node starts with an empty set.

Pruning and retention#

The reference node keeps the body and header of every canonical block, and the undo data of the last 30 days (per-block changes); side branches long behind the tip are forgotten (the block index). There are no snapshots and no other historical state pruning.

A node may prune old local data without redefining historical validity, provided it keeps, or can obtain, the state and data needed for the validation it performs. Live-state growth and historical retention are separate questions. There is no state rent or expiry, so the growth capacity miners vote is the only bound on how much state the network commits to keeping.