I have been working on something like this that fits better into my existing bash/zsh workflow. So I landed on using ollama and a small widget that tries to mirror the same behavior of ctrl+r aka reverse-i-search. Barely functional, but I have found value from it.
My personal problem with this is that I have no confidence in the answer provided by an AI. If I search the internet for help with some command line operation, I can try to find a reputable source that provides not only the command I'm looking for, but an explanation of the syntax that a) gives me confidence that it will do what I expect it to and b) allows me to learn how to construct that command so I can do it myself next time. This is, of course, less true today than it used to be, now that Google is likely to show me some convincing AI slop at the top of my results.
This isn’t the use case for the majority of people. Most are amateur developers who copy/paste the first thing they see on StackOverflow without understanding what they’re doing or reading the explanation because they just want to move forward on whatever project they’re actually working on (ex: building a REST API but they can’t install the necessary Python packages the YouTube tutorial they’re following uses). LLM-powered guidance here is tremendously useful, and if the person is curious, they can ask the LLM to explain the command, which it’s actually very good at based on my experience with GPT-4 and GPT-4o.
I think saying there’s “no confidence in the answer provided by an AI” is an overstatement and underestimates the value AI can have for the majority of users because that statement overestimates the value of “a reputable source that provides not only the command I’m looking for, but an explanation of the syntax” for the majority of users. Reputable sources and thorough explanations are great in theory, but very few have the patience to go through all that for a single CLI command when they want to make visible progress on the bigger picture project they actually care about.
> I think saying there’s “no confidence in the answer provided by an AI” is an overstatement and underestimates the value AI can have for the majority of users because that statement overestimates the value of “a reputable source that provides not only the command I’m looking for, but an explanation of the syntax” for the majority of users. Reputable sources and thorough explanations are great in theory, but very few have the patience to go through all that for a single CLI command when they want to make visible progress on the bigger picture project they actually care about.
These are exactly the people who shouldn't be running code written by a random number generator.
> These are exactly the people who shouldn't be running code written by a random number generator.
The beauty of technology and the internet is that there's near-zero gatekeeping. Anyone with enough interest and time can learn to build anything without seriously harming anyone. Dismissing LLMs as a random number generator is very clearly an overstatement for the value they're already able to provide. Ideally, how would you suggest a new developer learn to build something such as a REST API in Python?
That's why I was explicit in calling it "my personal problem" in the very beginning of my comment, and specified that I don't have confidence in the answer returned by AI. I apologize if I inadvertently appeared to speak for anybody but myself.
No worries—and it's clear that it's your personal problem. I think it's always valuable to present a counterpoint in these comment threads so that less-informed readers know the issue isn't totally clear and that the truth likely lies somewhere between your point and mine. I apologize if I came across as argumentative—my goal is to expand the scope of the discussion.
> I have no confidence in the answer provided by an AI
You don’t need to have confidence, these are shell one liners, you can generate it, run it, and verify the results in the time it would take you to open a browser window or scroll halfway down a man page.
And if it doesn’t work, then you can still fallback on those.
I verify the results by looking at the output and seeing if it did what I wanted it to do.
I don’t mean to be dismissive or reductive. If I need to do something more complex then I’m probably not trying to do it in a shell one-liner.
I used this feature earlier today and prompted it with something like “given piped input, use awk to filter just the 2nd and 5th columns, and truncate the 5th column to 25 chars”. I don’t use awk often but I know that it exists, and I could have figured out how to do this in about 20-40 seconds with a google search or man page, but this was 10x faster.
Is this a real opinion people have, or am I misunderstanding...
It sounds like you're recommending people to not understand the shell one-liners they're pasting into their CLI and instead just verify that the outcome was observed after the fact?
That's pretty much the opposite of what I've understood to be "good terminal practice" for my entire life, which can be summed up as "don't execute anything in the terminal that you don't understand".
Is "just paste it and hope you don't lost data/just installed a virus" the new modus-oprandi?
> It sounds like you're recommending people to not understand the shell one-liners they're pasting into their CLI and instead just verify that the outcome was observed after the fact?
Yes
> Is "just paste it and hope you don't lost data/just installed a virus" the new modus-oprandi?
No
Somewhere in between those two extremes is knowing enough to know that the command you’re pasting in isn’t going to install a virus, even though you don’t fully understand what the -g flag is doing
I do have this confidence if by AI we mean GPT4. I already routinely use it to generate small scripts and complex command line incantations. My memory is better spent elsewhere imo
Gen AI and its potential impact has been covered by Stephenson multiple times.
I wrote a short post on some of the references in Anathem.
It's also a large plot point in Fall or Dodge in Hell. One of the characters open sources some generative ai to fill the internet with slightly untrue spam.
When I first read Quicksilver I assumed that Newton as Warden of the Coin was one of the speculative fiction parts of the story. Nope, he was indeed involved heavily in improving the state of the art in authentic money.
I'm surprised he didn't get more negative attention from counterfeiters. But then once you get one hanged, and drawn and quartered, maybe they realize this cat has claws and they should leave it alone.
I agree with the author. The idea of "collateral knowledge" or "chain of learning" is definitely important for personal understanding and learning. However, I think depending on the person, ChatGPT actually helps the chain of learning rather than hinders.
I would love if someone forks this and turns it into an AI competition. Like that social network only for bots, this would be a runescape server only for bots.
There is a long term vision of supporting fine-tuning through an existing evaluation flow. We originally created this because we were worried about how to evaluate ‘what changed’ between a fine-tuned LLM and its base model. I wonder if Vertex AI has an API that we could plug-in, though, or if it’s limited to the UI.
I am not a physicist, but the "no probabilities" problem could just be solved with a constant offset right? Or just say that whichever direction the marble rolls is the zero degree mark.
I don't see how "no cause" problem is fully true either. The system has potential energy due to the marble being places at the top of the dome, so that was the cause.
I have been working on something like this that fits better into my existing bash/zsh workflow. So I landed on using ollama and a small widget that tries to mirror the same behavior of ctrl+r aka reverse-i-search. Barely functional, but I have found value from it.