Decentralized Storage And Content Addressing

Location Versus Content

When you open a web page, you type an address such as https://example.org/report.pdf. That address names a place: a server called example.org and a path on it. It says nothing about what the file contains. The owner can change the file tomorrow, or move it, or delete it, and your link then points to something else or to nothing. This is location addressing.

Content addressing names data by its content. The address is a fingerprint computed from the bytes themselves. If the bytes change, the address changes. Anyone holding a file with that fingerprint holds exactly the same file, no matter which machine it came from.

This turns out to be a strong tool for decentralized systems. In week 3 you saw that BitTorrent checks every piece against a hash. Content addressing takes that habit further: the hash becomes the name. You can fetch data from any stranger, because you can check what you received without trusting the sender.

Hashes As Addresses

A cryptographic hash function takes any data and returns a fixed-length fingerprint, called a digest. SHA-256, a widely used function, always returns 256 bits, written as 64 hexadecimal characters. Good hash functions have properties that make them useful as addresses.

They are deterministic: the same input always gives the same digest. They are fast to compute. A tiny change in the input changes the digest completely. They are one way: you cannot work backward from a digest to the data. And they are collision resistant: finding two different inputs with the same digest is infeasible in practice. Older functions such as MD5 and SHA-1 have known collision attacks, so new designs use SHA-256 or newer functions.

With these properties, a digest identifies data. Publish the digest through a trusted channel, and anyone can fetch the data from an untrusted source and verify it by hashing again. A match proves the data is the original.

• Same data gives the same address, everywhere
• Changed data gives a different address
• Verification needs no trust in the sender

Hash Output Sizes

bits

Longer outputs make accidental or crafted collisions harder. MD5 and SHA-1 are no longer considered safe against deliberate collisions. Source: RFC 1321; NIST FIPS 180-4

Merkle Trees

Ralph Merkle described hash trees in 1979. A Merkle tree hashes data in a hierarchy. Split a file into blocks and hash each block. Then hash the digests of pairs of blocks to make a layer above. Repeat until a single digest remains at the top, called the root.

The root commits to everything below it. Change any block, and its hash changes, and every hash above it changes, including the root. So one short digest vouches for a large amount of data.

A Merkle tree also allows compact proofs. To show that block 3 belongs to a file with a known root, you supply block 3 together with the sibling hashes along the path to the root. The verifier recomputes upward and compares with the root. The proof is small: its size grows with the logarithm of the number of blocks, so even a million blocks need only about twenty hashes. Bitcoin uses a Merkle tree to summarize the transactions in each block, and light clients use proofs to check that a payment is included without downloading the whole block.

From A File To An Address

1. Split the file into chunks
2. Hash each chunk
3. Hash pairs up into a tree
4. The root hash names the file
5. Anyone can verify any chunk

The root is a short fingerprint of the whole file.

Git As A Merkle DAG

If you have used Git, you have used content addressing. Git stores everything as objects, and each object is named by the hash of its content. A blob holds the contents of a file. A tree lists file names and points to blobs and other trees by hash, much like a folder. A commit points to a tree and to its parent commits by hash, and holds the author, date and message.

Because each object contains the hashes of the objects it points to, the whole history forms a Merkle DAG. DAG stands for directed acyclic graph: arrows go one way and never loop back. A commit's hash depends on the whole project snapshot and on all earlier history. Change an old file, and every later commit hash changes. This is why Git can detect corruption and why two people with the same commit hash hold the same history.

Identical content is stored once. If two folders contain the same file, both point to the same blob. Git has long used SHA-1 for its hashes and is moving toward SHA-256.

IPFS

IPFS, the InterPlanetary File System, was designed by Juan Benet and described in a 2014 paper. It applies Git's idea to the open Internet. Data is split into blocks, arranged into a Merkle DAG, and named by a content identifier, or CID. A CID contains the hash and also labels saying which hash function and encoding were used, so the format can evolve.

To fetch data, a peer asks the network who holds the CID. IPFS uses a DHT of the kind studied in week 3 to map CIDs to peers that hold them, then downloads the blocks and checks each against its hash. Peers who fetch a block can serve it onward.

An important point: IPFS is a way to name and move data, and it does not promise to keep it. A peer stores only what it chooses to keep, and unwanted blocks can be dropped. Keeping a CID available means pinning it: asking a node, or a paid pinning service, to hold the data. If nobody pins a file, it may vanish.

IPFS also supports names that can be updated. Since a CID changes when content changes, IPFS offers naming layers such as IPNS, which points a stable name to a changing CID. This is mutable pointer on top of immutable data, a pattern you will see often.

Filecoin And Arweave: Paying For Persistence

Someone must pay for disks and bandwidth. Two networks add economic incentives so that storage lasts. Both rely on the IPFS idea of verifiable data but differ in what they promise.

Filecoin, created by Protocol Labs, is a market for storage contracts. A client and a storage provider agree on a deal: this data, for this period, for this price. The provider locks up collateral in the network's token, FIL, and submits regular cryptographic proofs that the data is still stored. Two proof types matter: proof of replication shows that the provider holds a unique physical copy, and proof of spacetime shows that the copy is held continuously over time. A provider that fails the proofs loses collateral. Filecoin therefore sells rented storage with checkable promises.

Arweave aims at permanent storage with a single upfront payment. The fee is designed to cover the estimated cost of storing the data for a very long time, based on the assumption that storage hardware keeps getting cheaper. Part of the payment goes into an endowment that pays miners over the years. Miners are rewarded for proving they can access stored data, including older data, which pushes the network to keep full copies. Arweave's 'permaweb' stores web pages and files that are meant to stay available.

The honest comparison: Filecoin gives flexible, time-limited deals and a market price. Arweave gives a one-time purchase with permanence as the goal, which depends on its economic assumptions holding over decades. Neither makes data private. By default stored data is public, so private data must be encrypted before upload.

Three Ways To Keep A File Alive

System	Who pays	What it promises
IPFS	Whoever pins the data	Find data by content hash, no payment promise
Filecoin	Renter pays providers	Storage deals checked by proofs
Arweave	Pay once upfront	Permanent storage funded by an endowment

Content addressing says what a file is. Storage markets decide who keeps it. Source: Project documentation of IPFS, Filecoin and Arweave

Limits And Design Lessons

Content addressing has trade-offs worth knowing. Immutable data cannot be edited, so deletion is hard to promise: a copy may sit on any peer who kept it. That conflicts with rules such as the right to erasure, so personal data usually stays off public storage or stays encrypted with keys that can be destroyed.

Fetching data from far-away peers can be slower than from a nearby server of a content delivery network, so many projects combine gateways, caches and IPFS together. Availability still depends on someone keeping the data, and the incentives described above exist for this reason.

Finally, blockchains rarely store large files directly, because every validator would need a copy. A common pattern is to store the file on IPFS, Filecoin or Arweave and write only its CID, a few dozen bytes, on the chain. The chain gives an agreed, tamper-evident record, and the storage layer holds the bytes.

Next week we turn to a related question: how do you name and verify not data but people and organizations, using keys instead of hashes?

Key Terms

Content addressing

Naming data by a fingerprint of its content instead of by its location.

Cryptographic hash

A function that turns any data into a fixed-length fingerprint that is infeasible to forge.

Merkle tree

A tree of hashes in which each parent hashes its children, ending in one root.

Merkle proof

A short list of hashes that shows one block belongs under a known root.

Merkle DAG

A graph of objects that point to each other by hash, with no loops.

CID

A content identifier, the self-describing hash used as an address in IPFS.

Pinning

Asking a node to keep specific data so that it stays available.

Proof of spacetime

In Filecoin, evidence that a provider has stored data continuously over time.

Endowment

In Arweave, part of the upfront fee set aside to pay for future storage.

Exercise: Build A Content-Addressed Store And A Merkle Root

1. Create a file named cas.py and paste the code from the 'Content-addressed store' slide. Run it and note the two addresses printed.
2. Change one letter in the stored text and run again. Compare the two addresses.
3. Paste the code from the 'Merkle root in Python' slide into merkle.py and run it with four blocks. Write down the root.
4. Change the third block and run again. Check that the root changes.
5. Create a new Git repository with git init, commit two files, and run git cat-file -p HEAD followed by git cat-file -p on the tree hash. Identify the commit, the tree and the blobs.
6. Optional: install IPFS Desktop (free), add a text file, and see its CID. Add the same file again and confirm that the CID is identical.

Further Reading

• IPFS: Content Addressed, Versioned, P2P File System, Juan Benet, 2014 (arXiv)
• Pro Git (chapter 10: Git Internals), Scott Chacon and Ben Straub, free online
• A Digital Signature Based on a Conventional Encryption Function, Ralph Merkle, CRYPTO 1987 (hash tree work from 1979)
• Filecoin: A Decentralized Storage Network, Protocol Labs, whitepaper, 2017
• Arweave: A Protocol for Economically Sustainable Information Permanence, Sam Williams and William Jones, Arweave yellow paper