Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Years back someone proposed a cute algorithm for erasure codes that depended not on spinning rust but on multipath networking.

I believe they called it network coding and the idea was in a network with multiple routes I might get a file faster by pulling an erasure code that used two parts of the file or even two files from one upstream instead of waiting for the entire file from the primary server.



This has been used in Ceph for a long time:

https://docs.ceph.com/en/latest/rados/operations/erasure-cod...

I would not be surprised if there was a lot of stuff like this behind S3 and other cloud storage systems too, at least in the lower-access tiers of storage, but I have no actual knowledge of AWS or GCP systems.


Yeah, I use it in my homelab and it is really awesome to have "RAID(5/6)" basically work over the network.


Network coding is more than that, participants in the graph can synthesize new parts on the fly from parts they just got without having the whole thing.

FWIW, freenet at least uses fec-coded files so that you can have some flexibility in what parts you get and durability against a file becoming broken just because a single part gets lost.


and usenet binaries with https://en.wikipedia.org/wiki/Parchive and even earlier with RAR's recovery block, and probably even earlier with BBSes, but my memory is failing me what I was using before it in the 1990s.

edit: I see immediately below while I was composing this, someone mentioned pararchive..


I thought about something like that to make video calls more stable. For example I'll get completely different routes (sometimes noticeably lower/more predictable latency) when I use a VPN to connect to some peer in the US (from Europe.) Would be cool to combine different routes.


One difficulty with using multiple routes is you'll probably need to spend a lot of bytes on active probing, because the quality of a connection may change during the call and when the active connection loses quality it's not apparent what the other connections will do, so you need recent probing from them as well to make a smart change.

But in theory, you should have several routing options in a well supported calling service. I'll illustrate one direction, but the same options apply in the other direction, and there's no need for both peers to use the same connection to send (although WebRTC will)

Peer A -> Peer B

Peer A -> Relay near A -> Peer B

Peer A -> Relay near B -> Peer B

Peer A -> Relay near A -> Relay near B -> Peer B

If at least two of the four hosts mentioned have IPv4 and IPv6, you can also add those permutations. It's pretty typical to have different routing for v4 and v6.


It’s called SD-WAN


can you explain? I tried looking it up but I didn't quite understand how it is called SD-WAN.


With something like https://github.com/cberner/raptorq you can do several gbits/s over high latency/lossy UDP.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: