inb4: IPFS doesn’t work, unfortunately as you cannot provide the hash of an arbitrarily large file and retrieve it from the network. IPFS content IDs (CID) are a hash of the tree of chunks. Changes to chunk size can also change the hash!

Basically, I’d like to take the SHA256, SHA3, blake2, md5, of a file and either retrieve it from a network or get a list of sources for that file. Does something like that exist already or will I have to build it?

If I have to build it

it will be a really simple, dumb, HTTP service with

  • GET /uris/:hash:?alg=sha256|md5|blake
  • POST /uri/:hash: with the contents being a URI to the file
    supported URI schemes would probably be HTTP/S and FTP. Maybe P2P protocols like IPFS and if there’s a way to target a specific file in a torrent, maybe magnet links too. But that’s feels like risky territory.

Of course for hashing requests it would have a limited task queue (maybe 5 in parallel?), rate limiting by IP, and a size limit for retrieval (1GB feels like more than enough).

Can’t think of a way to do it with a DHT 🤷

  • AnAmericanPotato@programming.dev
    link
    fedilink
    English
    arrow-up
    0
    ·
    12 days ago

    IPFS content IDs (CID) are a hash of the tree of chunks. Changes to chunk size can also change the hash!

    I don’t understand why this is a deal-breaker. It seems like you could accomplish what you describe within IPFS simply by committing to a fixed chunk size. That’s valid within IPFS, right?

    Is it important to use any specific hashing algorithm(s)? If not, then isn’t an IPFS CID (with a fixed, predetermined chunk size) a stable hash algorithm in and of itself?