> First put all your services in a monorepo have it all build as one under CI. That’s a static check across the entire system.
That helps but is insufficient, since the set of concurrently deployed artifact versions can be different than any set of artifact versions seen by CI -- most obviously when two versions of the same artifact are actively deployed at the same time. It also appears to rule out the possibility of ever integrating with other systems (e.g., now you need to build your own Stripe-equivalent in a subdir of your monorepo).
> Second don’t use databases.
So you want to reimplement your own PostgreSQL-equivalent in another monorepo subdir too? I don't understand how opting not to use modern RDBMSes is practical. IIUC you're proposing implementing a DB using compiled queries that use the same types as the consuming application -- I can see the type-safety benefits, but this immediately restricts consumers to the same language or environment (e.g., JVM) that the DB was implemented in.
>That helps but is insufficient, since the set of concurrently deployed artifact versions can be different than any set of artifact versions seen by CI
Simple, although I only mentioned repos should be mono, I should've also said deployment should be mono as well. I thought that was a given.
>So you want to reimplement your own PostgreSQL-equivalent in another monorepo subdir too?
I'm too lazy to do this but in general I want an artifact that is built. All sql queries need to be compiled and built and the database runs that artifact. And of course all of this is part of a monorepo that is monodeployed.
>I don't understand how opting not to use modern RDBMSes is practical.
It's not practical. My point is there's some really stupid obvious flaws we live with because it's the only practical solution.
> Simple, although I only mentioned repos should be mono, I should've also said deployment should be mono as well. I thought that was a given.
Deploying your service graph as one atomic unit is not a given, and not necessarily even the best idea - you need to be able to roll back an individual service unless you have very small changes between versions, which means that even if they were rolled out atomically, you still run the risk of mixed versions sets.
>Deploying your service graph as one atomic unit is not a given,
It's not a given because you didn't make it a given.
>and not necessarily even the best idea - you need to be able to roll back an individual service unless you have very small changes between versions, which means that even if they were rolled out atomically, you still run the risk of mixed versions sets.
It is the best idea. This should be the standard. And nothing prevents you from rolling back an individual service. You can still do that. And you can still do individual deploys too. But these are just for patch ups.
When you roll back an individual service your entire system is no longer in a valid state. It's in an interim state of repair. You need to fix your changes in the monorepo and monodeploy again. A successful monodeploy ensures that the finished deploy is devoid of a common class of errors.
Monodeploy should be the gold standard, and individual deploys and roll backs are reserved for emergencies.
> It is the best idea. This should be the standard. And nothing prevents you from rolling back an individual service. You can still do that. And you can still do individual deploys too. But these are just for patch ups.
There are a ton of reasons it's not the best idea. This flies in the face of a lot of _better_ ideas.
Keeping changesets small so that it's easier to debug when something goes wrong? Blown out of the water by deploying everything at once.
Bringing every service up at once is a great way to create the coldest version of your entire product.
Requiring a monodeployment turns canarying or A/B testing entire classes of changes into a blocking rollout where any other feature work has to move at the pace of the slowest change.
> When you roll back an individual service your entire system is no longer in a valid state. It's in an interim state of repair.
The gold standard is that each version of your service can work with each other version of your service, because in The Real World your service will spend time in those states.
> Monodeploy should be the gold standard, and individual deploys and roll backs are reserved for emergencies.
No, because if it's still possible to mix versions in your services, then a monodeploy doesn't actually solve any issues.
I actually am a big fan of fewer services and deploying bigger artifacts, but if you have multiple services, you have to act like you have multiple services.
Outstanding piece. It articulates the actually fundamental correctness problem of most software development:
> “given the actual set of deployments that exist right now, is it safe to add this one?”
This problem has been right in front of our faces for decades, but I haven't ever seen it laid out like this, or fully realised it myself. Which is surprising, given the obvious utility of types for program-level correctness.
I think the next big leap forward in software reliability will be the development/integration of both the tooling to accomplish this, and the cultural norms to use them. Automating this (as far as possible -- they do note that semantic drift is immune to automatic detection) will be as big a deal as the concept of version control for code, or CI/CD.
There's lots here that I need to dig into further.
I don't think that's happening in practice, but 1) it may not be specified and 2) What you say could well be the persisted state after a machine crash or power loss. In particular if those files live in different directories.
You can remedy 2) by doing fsync() on the parent directory in between. I just asked ChatGPT which directory you need to fsync. It says it's both, the source and the target directory. Which "makes sense" and simplifies implementations, but it means the rename operation is atomic only at runtime, not if there's a crash in between. It think you might end up with 0 or 2 entries after a crash if you're unlucky.
If that's true, then for safety maybe one should never rename across directories, but instead do a coordinated link(source, target), fsync(target_dir), unlink(source), fsync(source_dir)
Depending on metadata cache behavior configuration, if the system is powered off immediately after the first command, then that could indeed happen I think.
As to whether it’s technically possible for it to happen on a system that stays on, I’m not sure, but it’s certainly vanishingly rare and likely requires very specific circumstances—not just a random race condition.
Uhh, if the system powers off immediately after the first command (mv a b), the second command (mv c d) would never run. So where would d come from if the command that created it never executed?
Sure, but splitting "atomic" operations across a reboot is an interesting design choice. Surely upon reboot you would re-try the first `mv a b` before doing other things.
All you need for this to occur is the window where both renames occurs overlap. A system polling to check if a, b, c, and d exist while the renames are happening might find all four of them.
Assuming that the two `mv` commands are run in sequence, there shouldn't be any possibility for a and d to be observed "at once" (i.e. first d and then afterwards still a, by a single process).
With regards to Linux kernel implementation, I could map you out a list of sequence points caused in the kernel by above sequential "shell script", proving that no other process could observe d and then a using two subsequent "stat" calls. I could imagine that a single directory listing (getdents) could happen to first visit a and then d, though -- I don't know the internals of that.
With regards to POSIX specification, I don't know the details but anything less than that guarantee would be very a huge robustness problem.
I'm almost certain what the OP meant was if the commands were run synchronously (ie: from 2 different shells or as `mv a b &; mv c d`) yes there is a possibility that a and d exist (eg: On a busy system where neither of the 2 commands can be immediately scheduled and eventually the second one ends up being scheduled before the first)
Or to go a level deeper, if you have 2 occurrences of rename(2) from the stdlibc ...
rename('a', 'b');
rename('c', 'd');
...and the compiler decides on out of order execution or optimizing by scheduling on different cpus, you can get a and d existing at the same time.
The reason it won't happen in the example you posted is the shell ensures the atomicity (by not forking the second mv until the wait() on the first returns)
nitpick, it should be `touch a c & mv a b & mv c d` as `&;` returns `bash: syntax error near unexpected token `;'`. I always find this oddly weird, but that would not be the first pattern in BASH that is.
`inotifywait` actually sees them in order, but nothing ensure that it's that way.
$ inotifywait -m /tmp
/tmp/ MOVED_FROM a
/tmp/ MOVED_TO b
/tmp/ MOVED_FROM c
/tmp/ MOVED_TO d
`stat` tells us that the timestamps are equal as well.
$ stat b d | grep '^Change'
Change: 2026-02-06 12:22:55.394932841 +0100
Change: 2026-02-06 12:22:55.394932841 +0100
However, speeding things up changes it a bit.
Given
$ (
set -eo pipefail
for i in {1..10000}
do
printf '%d ' "$i"
touch a c
mv a b &
mv c d &
wait
rm b d
done
)
1 2 3 4 5 6 .....
And with `inotifywait` I saw this when running it for a while.
$ inotifywait -m -e MOVED_FROM,MOVED_TO /tmp > /tmp/output
cat /tmp/output | xargs -l4 | sort | uniq -c
9104 /tmp/ MOVED_FROM a /tmp/ MOVED_TO b /tmp/ MOVED_FROM c /tmp/ MOVED_TO d
896 /tmp/ MOVED_FROM c /tmp/ MOVED_TO d /tmp/ MOVED_FROM a /tmp/ MOVED_TO b
The Poisson distribution is well approximated by the binomial distribution when n is high and p is low, which is exactly the case here. Despite the high variance in the sample mean, we can still make high-confidence statements about what range of incident rates are likely -- basically, dramatically higher rates are extremely unlikely. (Not sure, but I think it will turn out that confidence in statements about the true incident rate being lower than observed will be much lower.)
More data would certainly be better, but it's not as bad as you suggest -- the large number of miles driven till first incident does tell us something statistically meaningful about the incident rate per mile driven. If we view the data as a large sample of miles driven, each with some observed number of incidents, then what we have is "merely" an extremely skewed distribution. I can confidently say that, if you pick any sane family of distributions to model this, then after fitting just this "single" data point, the model will report that P(MTTF < one hundredth of the observed number of miles driven so far) is negligible. This would hold even if there were zero incidents so far.
The Kalman filter examples I've seen always involve estimating a very simple quantity, like the location of a single 3D point, from noisy sensors. It's clear how multiple estimates can be combined into a new estimate.
I'd guess that cameras on a self-driving car are trying to estimate something much more complex, something like 3D surfaces labeled with categories ("person", "traffic light", etc.). It's not obvious to me how estimates of such things from multiple sensors and predictions can be sensibly and efficiently combined to produce a better estimate. For example, what if there is a near red object in front of a distant red background, so that the camera estimates just a single object, but the lidar sees two?
1. make prediction on the next state change of some measurable n dimentional quantity, and estimate the covariance matrix across those n dimentions, which describe essentially a probability that the i-th dimention is going to increase (or decrease) with j-th dimention, where i and j are between 0 and n (indices of the vector)
2. Gather sensor data (that can be noisy), and reconcile the predicted measurement with the measured to get the best guess. The covariance matrix acts as a kind of weight for each of the elements
3. Update the covariance matrix based on the measurements in previous step.
You can do this for any vector of numbers. For example, instead of tracking individual objects, you can have a grid where each element represents a physical object that the car should not drive into, with a value representing certainty of that object being there. Then when you combine sensor reading, you still can use your vision model but that model would be enhanced by what lidar detects, both in terms of seeing things that camera doesn't pick up and rejecting things that aren't there.
And the concept is generic enough to where you can set up a system to be able to plug in any additional sensor with its own noise, and it all works out in the end. This is used all the You can even extend the concept past Gaussian noise and linearity, there are a number of other filters that deal with that, broadly under the umbrella of sensor fusion.
The problem is that Karpathy is more of a computer scientist, so he is on his Code 2.0 train of having ML models do everything. I dunno if he is like that himself or Musks "im smarter than everyone else that came before me" rubbed off.
And of course when you think like that, its going to be difficult to integrate lidar into the model. But the problem with that thinking is that forward inference LLM is not AI, and it will never ever be able to drive a car well compared to a true "reasoning" AI with feedback loops.
You're a machine. You're literally a wet, analog device converting some forms of energy into other forms just like any other machine as you work, rest, type out HN comments, etc. There is nothing special about the carbon atoms in your body -- there's no metadata attached to them marking them out as belonging to a Living Person. Other living-person-machines treat "you" differently than other clusters of atoms only because evolution has taught us that doing so is a mutually beneficial social convention.
So, since you're just a machine, any text you generate should be uninteresting to me -- correct?
Alternatively, could it be that a sufficiently complex and intricate machine can be interesting to observe in its own right?
If humans are machines, they are still a subset of machines and they (among other animals) are the only ones who can be demotivated and so it is still a mistake to assume an entirely different kind of machine would have those properties.
>Other living-person-machines treat "you" differently than other clusters of atoms only because evolution has taught us that doing so is a mutually beneficial social convention
Evolution doesn't "teach" anything. It's just an emergent property of the fact that life reproduces (and sometimes doesn't). If you're going to have this radically reductionist view of humanity, you can't also treat evolution as having any kind of agency.
"If humans are machines, they are still a subset of machines and they (among other animals) are the only ones who can be demotivated and so it is still a mistake to assume an entirely different kind of machine would have those properties."
Wrong level of abstraction. And not the definition of machine.
I might feel awe or amazement at what human-made machines can do -- the reason I got into programming. But I don't attribute human qualities to computers or software, a category error. No computer ever looked at me as interesting or tenacious.
reply