Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I remember talking to my ex's dad about his job, which involved planning refuels of a large nuclear-powered generation station in the Lower Midwest.

The words "it's a miracle it works at all" routinely popped up in those conversations, which is... something you don't want to hear about any sort of power generation - especially not nuclear - but it's true. It's a system basically built to produce "common accidents". It's amazing that it doesn't on a regular basis.




> The words "it's a miracle it works at all" routinely popped up in those conversations, which is... something you don't want to hear about any sort of power generation - especially not nuclear - but it's true.

Funny thing is, those are the exact words I use when talking to people about networking. And realistically anytime I dig deep into the underlying details of any big enough system I walk away with that impression. At scale, I think any system is less “controlled and planned precision” and more “harnessed chaos with a lot of resiliency to the unpredictability of that chaos”


This is one of the key insights in my early SRE career that changed how I viewed software engineering at scale.

Components aren’t reliable. The whole thing might be duct tape and popsicle sticks. But the trick for SRE work is to create stability from unreliable components by isolating and routing around failures.

It’s part of what made chaos engineering so effective. From randomly slowing down disk/network speed to unplugging server racks to making entire datacenters go dark - you intentionally introduce all sorts of crazy failure modes to intentionally break things and make sure the system remains metastable.


Everything is chaos, seek not to control it or you will lose your mind.

Seek only to understand it well enough to harness the chaos for more subtle useful purpose, for from chaos comes all the beauty and life in the universe.


Message on a mug: "if carpenters built houses the way programmers write software, a woodpecker could destroy civilization."

The syncronasation of a power grid ... Wow.


If houses could be torn down and recreated with the press of a key, we probably wouldn't have a housing shortage.


We would instead have HaaS, with monthly subscriptions for a license to use the house. Which can be randomly revoked at any moment if the company doesn't feel like supporting it is profitable enough, or if an AI thinks your electricity usage is suspicious and permabans you from using a home in the entire town.


Isn't that called "renting"?


So, in short, being a tenant in a low-regulatory environment.

Tell me more about this paradise.


Ha. I live in Australia and unfortunately that’s already about the quality level for new builds.


> those are the exact words I use when talking to people about networking

Or the U.S. financial system. Or civilization in general.


It ultimately comes down to shared norms, shared expectations, and trust.


A bit of a tangent, but I don't think this is it. There are plenty of species with plenty of shared norms, expectations, and trust - but no civilization. And, vice versa, many of the greatest societies have been riddled with completely incompatible worldviews yet created amazing civilizations. Consider that Sparta and Athens were separated by only 130 miles, yet couldn't possibly have been further apart!

The reason people work together is fundamentally the same reason you go to work - self interest. You're rarely there because you genuinely believe in the mission or product - mostly you just want to get paid and then go do your own thing. And that's basically the gears of society in a nutshell. But you need the intelligence to understand the bigger picture of things.

For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps. They even wage war over them and in efforts to expand them or control their territory. This [1] war was something that revolutionized our understanding of primates behaviors, which had been excessively idealized beforehand. But they lack the intelligence to understand how to bring their little societies up in scale.

They understand full well how to kill the other tribe and "integrate" their females, but they never think to e.g. enslave the males, let alone higher order forms of expansion with vassalage, negotiated treaties, and so on. All of which over time trend towards where we are today, where it turns out giving somebody a little slice of your pie and letting him otherwise roam free is way more effective than just trying to dominate him.

[1] - https://en.wikipedia.org/wiki/Gombe_Chimpanzee_War


> There are plenty of species with plenty of shared norms, expectations, and trust

Citation needed on that one.

> Consider that Sparta and Athens were separated by only 130 miles, yet couldn't possibly have been further apart!

They spoke the same language, shared the same literature, practiced the same religion, had a long history of diplomatic ties. When the Persians razed Athens, they took refuge with the Spartans.

> For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps.

Again, I don't think this claim stands to evidence. The so called chimp war you mention is about a group of about a dozen and a huge fight that broke out among them. That doesn't support the idea that they are capable of 200-strong 'intricate' groupings.


Not the OP, but:

"They spoke the same language" ... not exactly, the Spartans spoke Doric, while the Athenians Attic. (Interestingly, there is a few Doric speakers left [0].) While those languages were related, their mutual intelligibility was limited. Instead of "Greek" as a single language, you need to treat it as a family of languages, like "Slavic".

"shared the same literature" ... famously, the Spartans weren't much into culture and art, and they left barely any written records of their own. Even the contemporaries commented on just how boring Sparta was in all regards.

If we delve deeper into ideas about how a good citizen looked like, or how law worked, the differences between Sparta and Athens are significant, if not outright massive.

While those two cities weren't entirely alien to each other, had some ties, same gods, and occassionally fought on the same side in a big war, there was indeed a huge political and cultural distance between them. I would compare it to Poland vs. Russia.

[0] https://en.wikipedia.org/wiki/Tsakonian_language


You can split linguistic hairs all you want.

Not "entirely alien, had some ties" is not it. They were part of the same cultural cluster, participated in the same games, traveled to the same sanctuaries, had mutual proxenies. The very fact that we know the opinions of several Athenians about Spartans is telling. We don't know what they thought of inhabitants of Celtic population centers, or Assyrian cities, or Egyptian ones. But we know what they thought of individual Spartans that they mention by name, biographical detail and genealogy.


I stand by my comparison to the Slavic nations of today.

Yeah, we have a lot of opinions of one another, yes we understand basic vocabulary of our cousins, though details in fine speech are another matter, yes, we are technically Christian, but still the political and societal difference between, say, Czechs and Russians is quite big.

As was the difference between the Spartans and the Athenians. Constitutionally, the poleis were all over the map, from outright tyrannies, through oligarchies and theocracies, to somewhat democratic states.


So your argument is: Athens and Sparta had things in common but were different. Like Czechia and Russia. Czechia and Russia are quite different. So were Athens and Sparta?

That's called circular reasoning.


Try to speak holistically. I have no idea what you're trying to argue. I could expand or provide evidence for everything I said, but providing a citation or proving that there are indeed social groups of upwards of 200 chimps, or whatever, isn't going to do much, because you're not really formulating any argument or contrary view yourself.

Put another way, you're arguing against an example and not a fundamental premise. Proving the example is correct doesn't really get us anywhere since presumably you disagree with the fundamental premise.


> Try to speak holistically.

That sounds very much like "Just believe me." or even more "The rules were that you guys weren’t gonna fact-check"

> I have no idea what you're trying to argue.

Presumably you know what you are trying to argue. That is what the questions were about.

> Proving the example is correct doesn't really get us anywhere

You would have solid foundations to build your premise from. That is what it would get us.

First we check the bricks (the individual facts), then we check if they were correctly built into a wall (do the arguments add up? are the conclusions supported by the reasoning and the facts?). And then we marvel at the beautiful edifice you have built from it (the premise). Going the other way around is ass-backwards.

> you're not really formulating any argument or contrary view yourself.

I don't know what viewpoint namaria has. I know that "Sparta and Athens [..] couldn't possibly have been further apart" is ahistorical. They were very similar in many regards. If you think they were that different you have watched too many modern retellings, instead of reading actual history books. That's my contrary view.

> For instance Chimps have intricate little societies that at their peak have reached upwards of 200 chimps.

Here the question is what do we believe to be "societies". The researchers indeed documented hundreds of chimps visiting the same human made feeding station. Is that a society now? I don't think so, but maybe you think otherwise. What makes the Chimps' behaviour a society as opposed to just a bunch of chimps at the same place?


Which is why the long tail impact of current times is frankly terrifying.


Yes. The preppers are starting to look sane.


The preppers can only buy themselves a small amount of time, though—no more than a year or two. Eventually, their stockpiled supplies will run out, or some piece of equipment will need a replacement part.

I'd much rather focus on "prepping" by building social resiliency, instead. The local community I'm plugged into is much stronger together than anything I could possibly build individually.


For me it is the financial system as a whole.

I am an ex-scientist and an engineer and had a look at the books of my son who studies finance in the best finance school in the world (I am saying this to highlight that he will be one of the perpetrators, possibly with influence, of this mess)

The things in there are crazy. There are whole blocks that are obvious but made to sound complicated. I spent some time on a graph just to realize that they ultimately talk about solving a set of two linear equations (midfle school level).

Some pieces were not comprehensible because they did not make sense.

And then bam! A random differential equation and explanation as it was the answer to the universe. With an incorrect interpretation.

And then there are statistics that would make "sociology science" blush. Yes, they are so bad that even the, ahem, experts who do stats in sociology would be ashamed (no hate for sociology, everyone needs to eat, it is just that I was several times reviewer of thesises there and I have trauma afterwards).

The fact that finance works is because we have some kind of magical "local minimum of finance energy" from which the Trumps of this world somehow did not maybe to break from (fingers crossed) by disrupting the world too much.


I did a lot of work for a major airline earlier in my career and came away with same impression. I just couldn’t see how they kept planes in the air based on my experiences through out the organization. I think in a big enough org the sheer momentum keeps things moving despite all the fires happening constantly.


"Funny thing is, those are the exact words I use when talking to people about networking"

Computer networking is not the same. Our networks will not explode. I will grant you that they can be shite if not designed properly but they end up running slowly or not at all, but it will not combust nor explode.

If you get the basics right for ethernet then it works rather well as a massive network. You could describe it as an internetwork.

Basically, keep your layer 1 to around 200 odd maximum devices per VLAN - that works fine for IPv4. You might have to tune MAC tables for IPv6 for obvious reasons.

Your fancier switches will have some funky memory for tables of one address to other address translation eg MAC to IP n VLAN and that. That memory will be shared with other databases too, perhaps iSCSI, so you have to decide how to manage that lot.


You tried to nerdsnipe someone without mentioning L2 is effectively dead within datacenters since VXLAN became hardware accelerated in both Broadcom and "NVIDIA"(Mellanox) gear. And for those that don't need/care about L2 they don't even bother and run L3 all the way.

EVPN uses BGP to advertise MAC addresses in VXLAN networks which solves looping without magic packets, scales better and is easier to introspect.

And we didn't even get into the provider side which has been using MPLS for decades.

A problem with high bandwidth networking over fiber is that since light refracts within the fiber some light will take a longer path than other, if the widow is too short and you have too much scattering you will drop packets.

So hopefully someone doesn't bend your 100G fiber too much, if that isn't finicky idk what is, DAC cables with twinax solve it short-range for cheaper however.


If people knew how crappy, insecure, and unreliable nuclear computer systems were, there'd be a lot more existential dread about cyber security


I built control computers for nuclear reactors. Those machines are not connected to a network and are guarded by multiple stages of men with automatic machine guns. It was designed to flawlessly run 3x boards each with triple-modular-redundant processors in FPGA fabric all nine processors instruction-synced with ECC down to the Registers (including cycling the three areas of programmable fabric on the FPGAs). They cycle and test each board every month.

What’s your source?


Well, the news says that doge randos are potentially exfiltrating the details of systems like that as well as financial details of many Americans, including those who hold machine guns and probably suffer from substandard pay and bad economic prospects/job security as much as anyone else does.

Perhaps the safest assumption is that system reliability ultimately depends on quite a lot of factors that are not purely about careful engineering.


Nothing like a special commando of people doing your more malicious biddings while also being expendable


A bit off topic, but my uncle used to be security at a nuclear plant. Each year the Delta Force (his words) would conduct a surprise pentest. He said that although they were always tipped off, they never stopped them.


How is the software inspected and tested for defects, malicious or accidental? I'm just very curious about how this is done.


Almost all computers are insecure, not just the systems in nuclear stations.

Most operating systems are based on ambient authority, which is just a disaster waiting to happen.


What's the alternative?


I guess the biggest security advantage of any of these old critical systems is fact that they are not connected to the internet. At least I hope they are not.


My definition of technology is, “something built by humans that barely works”


Modern aircraft? Those are excellent and work well. I am thinking of a B787 and A350. More: How about medical implants, like a heart pacemaker?


The regulations around parts sourcing, required maintenance, and training has more to do with how well/safe modern aviation is than anything else. If those aren’t done properly, all sorts of weird things start happening. Pretty much the only reason aerospace safety records aren’t worse in third world countries is because of how obviously bad the consequences are quickly - and even then….




Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: