> often a benefit to having a human have an understanding of the concrete details of the system
Further elaborating from my experience.
1. I think we're in the early stages, where agents are useful because we still know enough to coach well - knowledge inertia.
2. I routinely make the mistake of allowing too much autonomy, and will have to spend time cleaning up poor design choices that were either inserted by the agent, or were forced upon it because I had lost lock on the implementation details (usually both in a causal loop!)
I just have a policy of moving slowly and carefully now through the critical code, vs letting the agent steer. They have overindexed on passing tests and "clean code", producing things that cause subtle errors time and time again in a large codebase.
> burn the time to understand it.
It seems to me to be self-evident that writing produces better understanding than reading. In fact, when I would try to understand a difficult codebase, it often meant that probing+rewriting produced a better understanding than reading, even if those changes were never kept.
It's hard for me to argue with these few direct sentences.
They delegitimize knowledge, inhibit cognitive development, short circuit
decision-making processes, and isolate humans by displacing or degrading human connection.
The result is that deploying AI systems within institutions
immediately gives that institution a half-life.
... even if we don't have a ton of "historical" evidence for AI doing this, the initial statement rings true.
e.g., an LLM-equipped novice becomes just enough of an expert to tromp around knocking down chesterton's fences in an established system of any kind. "First principles" reasoning combined with a surface understanding of a system (stated vs actual purpose/methods), is particularly dangerous for deep understanding and collaboration. Everyone has an LLM on their shoulder now.
It's obviously not always true, but without discipline, what they state does seem inevitable.
The statement that AI is tearing down institutions might be right, but certainly institutions face a ton of threats.
The examples that the paper cites that are historical are not compelling, in my opinion.
The authors use Elon Musk's DOGE as an example of how AI is destructive, but I would point out that that instance was an anomaly, historically, and that the use of AI was the least notable thing about it. It's much more notable that the richest man in the world curried favor by donating tens of millions of dollars to a sitting US president and then was given unrestricted access to the government as a result. AI doesn't even really enter the conversation.
The other example they give is of the FDA, but they barely have researched it and their citations are pop news articles, rather than any sort of deeper analysis. Those articles are based on anonymous sources that are no longer at the agency and directly conflict with other information I could find about the use of that AI at the FDA. The particular AI they mention is used for product recalls and they present no evidence that it has somehow destroyed the FDA.
In other words, while the premise of the paper may seem intellectually attractive, the more I have tried to validate their reasoning and methodology, the more I've come up empty.
The issue is the disconnect between professed principles and action. And the fact that nowadays there are not many ways to pick and choose principles except two big preset options.
That's the actual point. Everyone else is there to make money gambling, but the whole premise is to incentivize people with secret information to share it anonymously with the public, and take a reward for doing it.
All without traceability or secret drops or whatever.
Oh, I suppose one could interpret it as literally as possible, and arrive at that essay's conclusion.
I have always considered the following to be basically synonymous:
* In the absence of info, consider the intended output of the system to be what it is measured to be
* The output of the system is best determined using observation vs reasoning
Most of the examples there are moreso about two systems colliding. Yes, the purpose of the military is to disable the enemy and by god they are disabling a lot of each other so much so that they don't seem to be doing much else.
Except the bus system, in which the purpose is indeed to turn fuel into exhaust, because the busses move whether they are full or not. The purpose of busses is to drive around, and it so happens people like to use them. If the purpose of busses was to shuttle people around, it could be done several other better ways.
If the purpose is to gamble, it can be done many other ways. This system seems purpose-designed (or purpose-emergent) to coax out secret information in the form of large bets.
That is the "Single Cause" Fallacy or Causal Reductionism. That is the tendency to oversimplify a complex system by highlighting one "root cause".
Selecting a single measurement is the same as selecting a purpose.
Either way, we see people can choose a nonsense root cause, to argue something specious by defining a nonsense POSIWID.
There's also a crossover with the human tendency to try and attribute causes to the bosses of a system. Sometimes systems are emergent and are not designed/run by any specific person. Especially when hallucinating benefits to specific people e.g. "follow the money". I'd like to define this as "agentic reification of emergent systems" however unfortunately modern times are creating noise around each of those words.
In general I find it interesting to see how people argue about systems. From what I see, very few people understand the systems they comment upon - instead most people rely on political memes and shallow analysis instead of any deeper rational thinking. I've been decomposing my own thinking about mortgages for a while and I'm still terribly ignorant about that system!
The market can only resolve based on public information, so it could only incentivize revealing information that is already destined to be imminently revealed. Furthermore, it doesn't incentivize sharing that information with enough lead time to actually take action based on that information; the opposite is actually true, insiders are incentivized to wait until just before the event to make their trade, meaning that the public gets no actionable information in practice. And that's assuming that you can distinguish an insider from someone lying for the sake of market manipulation.
> [...] insiders are incentivized to wait until just before the event to make their trade, [...]
What are you basing that one? And how is this supposed to work?
If you are an insider the incentive is to trade as soon as possible, lest some other insider beats you to the punch, or some conventional leak (or investigative journalist) spoils your party.
This is easiest to see, when there are multiple unconnected insiders: the first to trade wins. But even if you merely suspect another insider might exist, you have an incentive to trade first.
> And that's assuming that you can distinguish an insider from someone lying for the sake of market manipulation.
That's exactly the same as any other noise trader in financial markets, yes. Nothing specific about insider information.
Without additional signals, you can just as well use it to manipulate markets.
E.g. there's a 1-to-1000 bet for $1m today on Trump falling down the staircase. So markets read this and go crazy, buying up the stock. The next day, nothing happens and the markets go down. But somebody could have made billions betting on that.
More seriously, I think GP was commenting on the stereotypical response that when a "wife" brings up a "problem" they often want "emotional support" rather than "discussion of solutions".
Not that wife literally gets mad when he solves a problem, unless his problem solving is yet another "engineer system solution" (vs manual labor) which I know from experience, try the patience of everyone involved in family life :D
I'm not looking to get into a whole thing here, but I disagree. There are such things as people that will simply need emotional supportz and then there are those that cannot disagree and commit.
Thank you for replying. You're providing an excellent example of someone that can't disagree and commit and must have emotional support to validate their feelings.
To me it's "search" like a missile does "flight". It's got a target and a closed loop guidance, and is mostly fire and forget (for search). At that, it excels.
I think the closed loop+great summary is the key to all the magic.
Which is kind of funny because my standard quip is that AI research, beginning in the 1950s/1960s, and indeed much of late 20th century computer tech especially along the Boston/SV axis, was funded by the government so that "the missile could know where it is". The DoD wanted smarter ICBMs that could autonomously identify and steer toward enemy targets, and smarter defense networks that could discern a genuine missile strike from, say, 99 red balloons going by.
It's a prediction algorithm that walks a high-dimensional manifold, in that sense all application of knowledge it just "search", so yes, you're fundamentally correct but still fundamentally wrong since you think this foundational truth is the end and beginning of what LLMs do, and thus your mental model does not adequately describe what these tools are capable of.
Me? My mental model?
I gave an analogy for Claude not a explanation for LLMs.
But you know what? I was mentally thinking of both deep think / research and Claude code, both of which are literally closed loop. I see this is slightly off topic b/c others are talking about the LLM only.
Sorry, I should have said "analogy" and not "mental model", that was presumptuous. Maybe I also should have replied to the GP comment instead.
Anyway, since we're here, I personally think giving LLMs agency helps unlock this latent knowledge, as it provides the agent more mobility when walking the manifold. It has a better chance at avoiding or leaving local minima/maxima, among other things. So I don't know if agentic loops are entirely off-topic when discussing the latent power of LLMs.
I dont see why phones can't come with a browser that does this. Parents could curate a whitelist like people curate playlists, and share it, and the browser would honor that.
Combined with some blacklisted apps (e.g., all other browsers), this would be a passable opt-in solution. I'm sure there's either a subscription or a small incentive for someone to build this that hopefully isn't "Scam children".
It's not like kids are using PCs, and if they use someone else's phone, that's at least a severely limiting factor.
They do, don’t they? Apple devices have had a robust whitelisting/blacklisting feature for at least a couple of years. I use it to block websites and apps to lessen my phone addiction. I’m sure Android offers similar features
Whoa whoa whoa let's not bring the accountants in!
Code isn't a liability b/c it costs money (though it does). Code is a liability like an unsafe / unproven bridge is a liability. It works fine until it doesn't - and at that point you're in trouble. Just b/c you can build lots of bridges now, doesn't mean each new bridge isn't also a risk. But if you gotta get somewhere now, conjuring bridges might be the way to go. Doesn't make each bridge not a liability (risky thing to rely on) or an asset (thing you can sell, use to build value)
Even proven code is a liability. The point of it being a liability is that it costs time and effort to maintain and update.
The same with the bridge. Even the best built and most useful bridge requires maintenance. Assuming changing traffic patterns, it might equally require upgrades and changes.
The problem with this whole “code is a liability” thing is that it’s vacuous. Your house is a liability. The bridge that gets you to work as a liability. Everything that requires any sort of maintenance or effort or upkeep or other future cost is ina sense a liability. This isn’t some deep insight though. This is like saying your bones could break so they are liability. OK, but their value drastically outweighs any liability they impose.
reply