Instead of using AWS another approach involves self hosting the hardware as well. Even after factoring in energy, this does dramatically lower the price.
Assuming we want to mirror our setup in AWS, we’d need 4x NVidia Tesla T4s. You can buy them for about $700 on eBay.
Add in $1,000 to setup the rest of the rig and you have a final price of around:
$2,800 + $1,000 = $3,800
This whole exercise assumes that you're using the Llama 3 8b model. At full fp16 precision that will fit in one 3090 or 4090 GPU (the int8 version will too, and run faster, with very little degradation.) Especially if you're willing to buy GPU hardware from eBay, that will cost significantly less.
I have my home workstation with a 4090 exposed as a vLLM service to an AWS environment where I access it via reverse SSH tunnel.
Why did this only occur to me recently? You can selfhost a k8s cluster and expose the services using a $5 digital ocean droplet. The droplet and k8s services are point-to-point connected using tailscale. Performance is perfectly fine, keeps your skillset sharp, and you’re self-hosting!
You can also just directly connect to containers using Tailscale if it's just for internal use. That is, having an internally addressable `https://container_name` on your tailnet per-container if you want. This way I can setup Immich for example and it's just on my tailnet at `https://immich` without the need for a reverse proxy, etc...
Whoa, so you have code running in AWS making use of your local hardware via what is called a reverse SSH tunnel? I will have to look into how that works, that's pretty powerful if so. I have a mac mini that I use for builds and deploys via FTP/SFTP and was going to look into setting up "messaging" via that pipeline to access local hardware compute through file messages lol, but reverse SSH tunnel sounds like it'll be way better for directly calling executables rather than needing to parse messages from files first.
Look into Nebula (or Tailscale if you trust third parties). I have all my workstations and servers on a mesh network that appears as a single /24 that is end to end encrypted, mutually authenticated and works through/behind NAT. I can spawn a vhost on any server that reverse proxies an API to any port on any machine.
Well, one example, depending on your threat model—their privacy policy states that they retain info and comply with subpoenas.
There's also potential for malicious updates to compromise a network (as there is with most software unless you're auditing the source for each update).
E2EE is only as meaningful as where the keys reside, and how easily those keys are abused.
The idea of “user’s permission” is determined by tailscale and/or the oidc provider. I don’t know anything about “tail lock”, perhaps it is a new mitigation for this issue?
I didn’t start with tailscale because the only way you could log into it was with Google or GitHub or something. I don’t trust Microsoft or Google with auth for my internal network. I thought about running Headscale but Nebula was faster/easier for me.
the docs are good. when creating the initial CA make absolutely sure you set the CA expiration to 10-30 years, the default is 1 which means your whole setup explodes in a year without warning.
I feel like wire guard definitely does nat punching unless I misunderstand you, I've been doing this sort of thing to have my phone and desktop on the same "LAN" all the time so I can moonlight in from anywhere (among other things) and they're definitely natted.
Unfortunately my mac mini isn't beefy enough to run ollama, it's the base model m1 from a couple years ago lol. But it's very powerful for builds, deploys, and some computation via scripts. Now I'm curious to check out how much memory the newest ones support for potentially using ollama on it haha. Thanks!
(It may need to be Q4 or Q3 instead of Q5 depending on how the RAM shakes out. But the Q5_K_M quantization (k-quantization is the term) is generally the best balance of size vs performance vs intelligence if you can run it, followed by Q4_K_M. Running Q6, Q8, or fp16 is of course even better but you’re nowhere near fitting that on 8gb.)
Dolphin-llama3 is generally more compliant and I’d recommend that over just the base model. It's been fine-tuned to filter out the dumb "sorry I can't do that" battle, and it turns out this also increases the quality of the results (by limiting the space you're generating, you also limit the quality of the results).
Most of the time you will want to look for an "instruct" model, if it doesn't have the instruct suffix it'll normally be a "fill in the blank" model that finishes what it thinks is the pattern in the input, rather than generate a textual answer to a question. But ollama typically pulls the instruct models into their repos.
(sometimes you will see this even with instruct models, especially if they're misconfigured. When llama3 non-dolphin first came out I played with it and I'd get answers that looked like stackoverflow format or quora format responses with ""scores"" etc, either as the full output or mixed in. Presumably a misconfigured model, or they pulled in a non-instruct model, or something.)
Dolphin-mixtral:8x7b-v2.7 is where things get really interesting imo. I have 64gb and 32gb machines and so far the Q6 and q4-k_m are the best options for those machines. dolphin-llama3 is reasonable but dolphin-mixtral is a richer better response.
I’m told there’s better stuff available now, but not sure what a good choice would be for for 64gb and 32gb if not mixtral.
Also, just keep an eye on r/LocalLLaMA in general, that's where all the enthusiasts hang out.
using Tailscale can make the networking setup much easier, really like their service for things like this (or curling another dev's local running server)
I don't know enough about networking and that level of configuration to do it confidently and safely yet. I'd rather rely on using credentials where the only real access is some limited command line executables or file transfer, rather than exposing more of the hardware directly to the network for a direct connection. I do have interest in learning this much, but I find that my current FTP/SFTP approach has more guard rails than a direct connection. Do you agree with this or am I just not understanding enough about ipv6 and direct connections home?
You get the equivalent setup in ipv6 by having your home modem or router deny inbound connections.
I think port forwarding configuration is a pain that does not offer value over just poking a hole in your firewall to do an authenticated connection over ssh.
Read the other comment and also immediately thought of Cloudflare tunnel instead - is there a reason you chose that? Wondering if I should do the same with my old Titan XP (probably slower than your 3070 but it does have 12gb of vram)
No reason other than that cloudflare is quite a nice reverse proxy and I already use it for managing my DNS's. Thus far, I've not noticed any issues with it, other than when my home wifi goes down.
1B of tokens for Gemini Flash (which is on par with llama3-70b in my experience or even better sometimes) with 2:1 input-output would cost ~600 bucks (ignoring the fact they offer 1M tokens a day for free now). Ignoring electricity you'd break even in >8 years. You can find llama3-70b for ~same prices if you're interested in the specific model.
I answered the financial thinking in another reply, but another factor is I need to know if the model today is exactly the same as tomorrow for reliable scientific benchmarking.
I need to tell if I change I made was impactful, but if the model just magically gets smarter or dumber at my tasks with no warning then I can’t tell if I made an improvement or a regression.
Whereas the model on my GPU doesn’t change unless I change it. So it’s one less variable and LLM are black box to start with.
I may be wrong for Gemini, but my impression is all the companies are constantly tweaking the big models. I know GPT on Monday is not always the same GPT on Thursday for example.
I've worked professionally over the last 12 months hosting quite a few foundation models and fine tuned LLMs on our own hardware, aws + azure vms and also a variety of newer "inference serving" type services that are popping up everywhere.
I don't do any work with the output, I'm just the MLOps guy (ahem, DevOps).
You mention expense but on a purely financial basis I find any of these hosted solutions really hard to justify against GPT 3.5 turbo prices, including building your own rig. $5k + electricity is loads of 3.5 Turbo tokens.
Of course none of the data scientists or researchers I work with want to use that though - it's not their job to host these things or worry about the costs.
So my main motivation is not so much to have the lowest cost, but to have the most predictable cost.
Knowing up front this is my fixed ML budget gives me peace of mind and gives me room to try stupid ideas without worrying about it.
Whereas doing it in the cloud you can a) get slammed with some crazy bill by accident, b) have to think more about what resources testing an idea will take, or conversely c) getting GPU FOMO and thinking “if just upgrade a level all my problems will be solved”.
It works for me, everybody mileage varies but personally I like to budget; spend; and then totally focus on my goals and not my cloud spend.
I’m also from the pre-cloud era, so doing stuff on my own bare metal is second nature.
I’m doing non-interactive tasks, but in terms of the A6000 running llama3 70b in chat mode it’s as usable as any of the commercial offerings in terms of speed. I read quickly and it’s faster than I read.
Yea, for any hobbyist, indie developer, etc. I think it'd be ridiculous to not first try running one of these smaller (but decently powerful) open source models on your own hardware at home.
Ollama makes it dead simple just to try it out. I was pleasantly surprised by the tokens/sec I could get with Llama 3 8B on a 2021 M1 MBP. Now need to try on my gaming PC I never use. Would be super cool to just have a LLM server on my local network for me and the fam. Exciting times.
I assume the characteristics would be pretty different, since your local hardware can keep the context loaded in memory, unlike APIs which I'm guessing have to re-load it for each query/generation?
Yeah but this article is terrible. First it talks about naively copy-pasting code to get “a seeming 10x speed-up” and then “This ended up being incorrect way of calculating the tokens used.”
I would not bank on anything in this article. It might as well have been written by a tiny Llama model.
This is great advice. I used to run my dev stuff on AWS, then built a small 6 server proxmox cluster in my basement, 300 cores, 1tb memory, 12tb ssd storage for about 3k usd. I don’t even want to know what it would cost to run a similar config on AWS. You can get cheap ddr4 servers on eBay all day.
Furthermore, the AWS estimates are also really poorly done. Using EKS this way is really inefficient, and a better comparison would be AWS Bedrock Haiku which averages $0.75/M tokens: https://aws.amazon.com/bedrock/pricing/
This whole post makes OpenAI look like a better deal than it actually is.
I was getting that sense too. It would not be difficult to build a desktop machine with a 4090 for around $2500. I run Llama-3 8b on my 4090, and it runs well. Plus side is I can play games with the machine too :)
Don't listen to this person. They have no idea what they're talking about.
No one cares about this TOS provision. I know both startups and large businesses that violate it as well as industry datacenters and academic clusters. There are companies that explicitly sell you hardware to violate it. Heck, Nvidia will even give you a discount when you buy the hardware to violate it in large enough volume!
In a previous AI wave hosters like OVH and Hetzner started offering servers with GTX 1080 at prices other hosters with datacenter-grade GPUs couldn't possibly compete with - and VRAM wasn't as big of a deal back then. That's who this clause targets.
If you don't rent our servers or VMs Nvidia doesn't care. They aren't Oracle.
There's no further elaboration on what "datacenter" means here, and it's a fair argument to say that a closet with one consumer-GPU-enriched PC is not a "datacenter deployment". The odds that Nvidia would pursue a claim against an individual or small business who used it that way is infinitesimal.
So both the ethical issue (it's a fair-if-debatable read of the clause) and the practical legal issue (Nvidia wouldn't bother to argue either way) seem to say one needn't worry about.
The clause is there to deter at-scale commercial service providers from buying up the consumer card market.
Llama-3 is one of the models provided by AWS Bedrock which offers pay as you go pricing. I'm curious how it would break down on that.
LLAMA 8B on Bedrock is $0.40 per 1M input tokens and $0.60 per 1M output tokens which is a lot cheaper than OpenAI models.
Edit: to add to that, as technical people we tend to discount the value of our own time. Bedrock and the OpenAI are both very easy to integrate with and get started. How long did this server take to build? How much time does it take to maintain and make sure all the security patches are applied each month? How often does it crash and how much time will be needed to recover it? Do you keep spare parts on hand / how much is the cost of downtime if you have to wait to get a replacement part in the mail? That's got to be part of the break-even equation.
About 3 days [from 0 and iterating multiple times to the final solution]
> How much time does it take to maintain and make sure all the security patches are applied each month?
A lot
> How often does it crash and how much time will be needed to recover it? Do you keep spare parts on hand / how much is the cost of downtime if you have to wait to get a replacement part in the mail?
All really good points, the exercise to self host is really just to see what is possible but completely agree that self hosting makes little to no sense unless you have a business case that can justify it.
Not to mention if you sign customers with SLAs and then end up having downtime would put even more pressure on your self hosted hardware
Just to bounce off this a little. If you are looking to fine-tune using an on demand service it seems Amazon Sagemaker can do it at seemingly decent prices:
I'd love to hear someones experience using this as I want to make an RPG rules bot tied to a specific ruleset as a project but I fear AWS as it might bankrupt me!
In my experience SageMaker was relatively straightforward for fine-tuning models that could fit on a single instance, but distributed training still requires a good bit of detailed understanding of how things work under the covers. SageMaker Jumpstart includes some pretty easy out-of-the-box configurations for fine-tuning models that are a good starting point. They will incorporate some basic quantization and other cost-savings techniques to help reduce the total compute time.
To help control costs, you can choose pretty conservative settings in terms of how long you want to let the model train for. Once that iteration is done and you have a model artifact saved, you can always pick back up and perform more rounds of training using the previous checkpoint as a starting point.
The option to pay is still listed as coming soon, but I also see pricing information in the settings page, so maybe it actually is coming somewhat sooner. I’m seeing $0.05/1M input and $0.10/1M output for llama3 8B, which is not exactly identical to what the previous person quoted.
Either way, I wish Groq would offer a real service to people willing to pay.
tl;dr: no-ish, it's getting better but still not there.
I don't really get it, only thing I can surmise is it'd be such a no-brainer in various cases, that if they tried supporting it as a service, they'd have to cut users. I've seen multiple big media company employees begging for some sort of response on their discord.
> How long did this server take to build? How much time does it take to maintain and make sure all the security patches are applied each month? How often does it crash and how much time will be needed to recover it? Do you keep spare parts on hand / how much is the cost of downtime if you have to wait to get a replacement part in the mail? That's got to be part of the break-even equation.
All of these are the kinds of things that people say to non-technical people to try to sell cloud. It's all fluff.
Do you really think that cloud computing doesn't have security issues, or crashes, or data loss, or that it doesn't involve lots of administration? Thinking that we don't know any better is both disingenuous and a bit insulting.
I've managed fleets on cloud providers with over 100k instances, even with all the excellent features through APIs, managing instances can quickly get tricky.
Tbh, your comment is kind of insulting and belittles how far we've come ahead in infrastructure management.
The cloud is probably more secure than a set of janky servers that you have running in your basement. You can totally automate away 0-days, cves and get access to better security primitives.
> The cloud is probably more secure than a set of janky servers that you have running in your basement.
Apples/Oranges. Your janky cloud[0] is less secure than the servers in my basement, because I'm a mostly competent sysadmin. Cloud lets you trade some operational concerns for higher costs, but not all of them.
[0] If you can assume servers run by somebody who doesn't know how to do it properly, obviously I can assume the same about cloud configuration. Have fun with your leaked API keys.
If my comment is insulting, I apologize. That was not my intention. My intention was to say that writing sales speak in a technical discussion is insulting to those of us who know better.
However, you've now gone out of your way to try to be insulting. You know nothing about me, yet you want to suggest that the cloud is more secure than my servers, and that my servers are "janky"?
Please try a little harder to engage in reasonable discourse.
I've managed both data centers and cloud and IMHO, no, it is not fluff. To take it in order:
> doesn't have security issues
It sure does, but the matrix of responsibility is very different when it is a hosted service. Note: I am making these comments about Bedrock, which is serverless not in relation to EC2.
> It crashes
Absolutely, but the recovery profile is not even close to the same. Unless you have a full time person with physical access to your server who can go press buttons.
> data loss
I'm going to shift this one a tiny bit. What about hardware loss? You need backups regardless. On the cloud when a HDD dies you provision a new one. On premise you need to have the replacement there and ready to swap out (unless you want to wait for shipping). Same with all the other components. So you basically need to buy two of everything. If you have a fleet of servers that's not too bad since presumably they aren't going to all fail on the same component at the same time. But for a single server it is literally double the cost.
> doesn't involve lots of administration
Again, this is relation to Bedrock with is a managed serverless environment. So there is litterally no administration aside from provisioning and securing access to the resource. You'd have a point if this was running on EC2 or EKS but that's not what my post was about.
> Thinking that we don't know any better is both disingenuous and a bit insulting.
I'm not saying cloud is perfect in any way, like all things it requires tradeoff, but quite frankly I find you dismissing my 25 years of experience, 1/3 of that has been working in real data centers (including a top-50 internet company at the time) as "fluff" as "disingenuous and a bit insulting".
> Unless you have a full time person with physical access to your server who can go press buttons.
Every colo facility I’ve used offers “remote hands”. If you need a button pressed or a disk swapped, they will do it, with a fee structure and response time that varies depending on one’s arrangement with the operator. But it’s generally both inexpensive and fast.
> What about hardware loss? You need backups regardless. On the cloud when a HDD dies you provision a new one. On premise you need to have the replacement there and ready to swap out (unless you want to wait for shipping).
Two of everything may still be cheaper than expensive cloud services. But there’s an obvious middle ground: a service contract that guarantees you spare parts and a technician with a designated amount of notice. This service is widely available and reasonably priced. (Don’t believe the listed total prices on the web sites of big name server vendors — they negotiate substantial discounts, even in small quantities.)
I guess inexpensive is relative. I've been on cloud for a while so I'm not sure what the going rates are for "remote hands" and most of my experience is with on-premise vs co-lo.
> Two of everything may still be cheaper than expensive cloud services.
That is true. Everything has tradeoffs. Though in the OPs case I think the math is relatively clear. With Open AIs pricing he calculated the break even at 5 years just for the hardware and electricity. Assuming that calculation is right, two of everything would up that to 7+ years, at which point... a lot can happen in 7 years.
> I guess inexpensive is relative. I've been on cloud for a while so I'm not sure what the going rates are for "remote hands" and most of my experience is with on-premise vs co-lo.
At a low end facility, I’ve usually paid between $0 and $50 per remote hands incident. The staff was friendly and competent, and I had no complaints. The price list goes a bit higher, but I haven’t needed those services at that facility.
You could have gotten rid of the middle paragraph. It’s not fluff. These are valid technical points. Issues most companies would rather (reasonably) pay to not have to deal with.
And do you really think you can offer better security and uptime than AWS? Not impossible but very expensive if you’re managing everything from your own hardware. You clearly vastly underestimate all that AWS is taking care of.
Self hosting means hosting it yourself, not running it on Amazon. I think the distinction the author intends to make is between running something that can't be hosted elsewhere, like ChatGPT, versus running Llama-3 yourself.
Overlooking that, the rest of the article feels a bit strange. Would we really have a use case where we can make use of those 157 million tokens a month? Would we really round $50 of energy cost to $100 a month? (Granted, the author didn't include power for the computer) If we buy our own system to run, why would we need to "scale your own hardware"?
I get that this is just to give us an idea of what running something yourself would cost when comparing with services like ChatGPT, but if so, we wouldn't be making most of the choices made here such as getting four NVIDIA Tesla T4 cards.
Memory is cheap, so running Llama-3 entirely on CPU is also an option. It's slower, of course, but it's infinitely more flexible. If I really wanted to spend a lot of time tinkering with LLMs, I'd definitely do this to figure out what I want to run before deciding on GPU hardware, then I'd get GPU hardware that best matches that, instead of the other way around.
I would say that is "cloud hosted", which is obviously very expensive compared to running on hardware you own (assuming you own a computer and a GPU). That was the comparison I was interested in, the fact that renting a computer is more expensive than the OpenAI API is not a surprising result.
The very first definition from the link you provide is:
> Self-hosting is the practice of running and maintaining a website or service using a private web server, instead of using a service outside of someone's own control.
Hosting anything on Amazon is not "using a private web server" and is the very definition of using "a service outside of someone's own control".
The fact that the rest of the article talks about "enabled users to run their own servers on remote hardware or virtual machines" is just wrong. It's not "their own servers", and we don't have "more control over their data, privacy" when it's literally in the possession of others.
> The practice of self-hosting web services became more feasible with the development of cloud computing and virtualization technologies, which enabled users to run their own servers on remote hardware or virtual machines. The first public cloud service, Amazon Web Services (AWS), was launched in 2006, offering Simple Storage Service (S3) and Elastic Compute Cloud (EC2) as its initial products.[3]
I hate when terms get diluted like this. "self hosted", to me, means you own the physical machine. This reminds of how "air-gapped server" now means a route configuration vs an actual gap of air, no physical connection, between two networks. It really confuses things.
I wonder how long NVIDIA can justify its current market cap once people realize just how cheap it is to run inference on these models given that LLM performance is plateauing, LLM's as a whole are becoming commoditized, and compute demand for training will drop off a cliff sooner than people expect.
As someone else points out, training is slightly more involved, but I also find that these smaller models are next to worthless compared to the larger ones.
There are probably some situations where it suffices to use a small model, but for most purposes, I'd prefer to use the state of the art, and I'm eager for that state to progress a little more.
> but for most purposes, I'd prefer to use the state of the art
I'm guessing in the future, we'll see a lot more automatic inference "on our behalf", and you won't care or notice if its using Llama5:3b or whatever comes out then.
I'm betting that in a few years we'll see LLMs baked into a ton of stuff - lots of "simple" things mostly, like email summary/rewording, and similar "light touch" use cases. Maybe light multi-modal work like photo labeling. Probably expanded to be running in every other SaaS applications doing god knows what. For those, the small models would be more than enough, and much cheaper to run in large volumes.
I'm guessing "chat with a bot to ask questions" will be a small amount of the inference that happens on our behalf, but will use the valuable SOTA model use case.
I am a partial believer that the real race for many tech players is actually AGI and ASI later and till the problem is solved the hardware arm race will keep being part of it.
Not only big tech is part of it but billion dollars startups are popping everywhere from China to US and Middle East.
Unless we hit another AI winter. We might get to the point where the hardware just can't give better returns and have to wait another 20 years for the next leap forward. We're still orders of magnitude away from the human brain.
Nvidia doesn’t obviously have a strong inference play right now for a widely-deployed small model. For a model that really needs a 4090, maybe. But for a model that can run on a Coral chip or an M1/M2/M3 or whatever Intel or AMD’s latest little AI engines can do? This market has plenty of players, and Nvidia doesn’t seem to be anywhere near the lead except insofar as it’s a little bit easier to run the software on CUDA.
Well every other tech company is writing checks to prove they have the chops to make an LLM. From IBM to Databricks to the big guys like Google. Tons of companies made one just to show off to investors or their CEO or just because they wanted skin in the game. But that probably won't continue forever. We've already seen certain orgs that seem to outshine others, and if they can't catch up, they may just accept to using the open-access models or APIs instead.
At some point everyone will realize that it is becoming a commodity, and it is very expensive to train, then only those with wither a structural advantage to lower price (eg Google) or a true goal of being on the high-end/SOTA of the market (OpenAI, Anthropic) will keep going.
You're talking about training the foundation models. But what about all the fine tuning on non-public business data that will be necessary to make gen AI useful for actual business processes?
I'm finding it difficult to estimate the size of this workload compared to continued training of foundation models. Perhaps it depends on whether there are new architectural breakthroughs that require retraining of foundation models.
And what about non-language tasks such as interpreting video and 3D sensory data? This is potentially huge, but between huge peaks there is often a valley of unknowable depth and breadth.
I have yet to hear a use-case for “fine tuning from business data” that relied on large models to succeed. Once again, I’m skeptical the average business will need this.
Yea yes video probably requires a lot of GPUs to train. And a lot of source material to train against. And a use case. Which again, most companies don’t have.
Model development is clearly here to stay, and clearly valuable. Models from every other company, either foundation or fine tuned - I’m not sure that emperor is wearing many clothes any time soon.
> I wonder how long NVIDIA can justify its current market cap once people realize just how cheap it is to run inference on these models given that LLM performance is plateauing
next wave driving demand can be actual new products developed on LLMs. There are very few usecases currently well developed besides chatbots, but potential is very large.
Agreed with the sentiments here that this article gets a lot of the facts wrong, and I'll add one: the cost for electricity when self-hosting is dramatically lower than the article says. The math assumes that each of the Tesla T4s will be using their full TDP (70W each) 24 hours a day, 7 days a week. In reality, GPUs throttle down to a low power state when not in use. So unless you're conversing with your LLM literally 24 hours a day, it will be using dramatically less power. Even when actively doing inference, my GPU doesn't quite max out its power usage.
Your self-hosted LLM box is going to use maybe 20-30% of the power this article suggests it will.
Source: I run LLMs at home on a machine I built myself.
Surprised no comments are pointing out that the analysis is pretty far off simply due to the fact that the author runs with batch size of 1. The cost being 100x - 1000x what API providers are charging should be a hint that something is seriously off, even if you expect some of these APIs to be subsidized.
AWS's pricing is just ridiculous. Their 1-year reserve pricing for an 8x H100 or A100 instance (p4/p5) costs just as much as buying the machine outright with tens of thousands left over for the NVIDIA enterprise license and someone to manage them (per instance!). Their on demand pricing is even more insane - they're charging $3.x/hr for six year old cards.
That's why I said "and someone to manage them". The difference is in the tens of thousands of dollars per instance. The savings from even a dozen instances is enough to pay for someone to manage them full time, and that's just for the first year. Year 2 and 3 you're saving six figures per instance so you'd be able to afford one person per machine to hand massage them like some fancy kobe beef.
A100 TDP is 400W so assuming 4kW for the whole machine, that's a little more than $5k/year at $0.15/kWh. Again, the difference is in the tens of thousands per instance. Even at 50% utilization over three years, if you need more than a dozen machines it's much cheaper to buy them outright, especially on credit.
I thought it was generally known they were operating at a loss?
even with the subs and api charges, they still let people use chatGPT for free with no monetization options. Sure they are collecting the data for training, but that's hard to quantify the value of.
I mean, no, I came to scan the comments quick after reading because there's a lot of bad info you can walk away with from the post, it's sort of starting from scratch on hosting LLMs
If you keep reading past there, they get it down significantly. The 8 tkn/s number AWS was evaluated on is really funny, that's about what you'd get on last year's iphone and it's not because apples special, it's because theres barely any reasonable optimization being done here. No batching, float32 weights (8 bit is guaranteed indistinguishable from 32 bit, 5 bit tests as definitely indistinguishable in blind tests, 4 bit arguably is indistinguishable)
You're right. In fact, using EKS at all is silly when AWS offers their Bedrock service with Claude Haiku (rated #19 on Chat Arena vs. ChatGPT3.5-Turbo at #37) for a much lower cost of $0.75/M tokens (averaging input and output like OP does)[0].
So in reality AWS is cheaper for a much better model if you don't go with a wildly suboptimal setup.
yeah what, a GV100 is $1600 on eBay and you can run like a 3.8bpw quant of 70b llama at some decent number of tk/s, if you buy two you can run a 70b 5bpw quant with 32k context no problem
A single synchronous request is not a good way to understand cost here unless your workload is truly singular tiny requests. Chatgpt handles many requests in parallel and this article's 4 GPU setup certainly can handle more too.
It is miraculous that the cost comparison isn't worse given how adversarial this test is.
Larger requests, concurrent requests, and request queueing will drastically reduce cost here.
Great mix of napkin math and proper analysis, but what strikes me most is how cheap LLM access is. For it being relatively bleeding edge, us splitting hairs on < $20/M tokens is remarkable itself, and something tech people should be thrilled about.
Smacks of the "starving kids in Africa" fallacy, you could make the same argument that tech people should be thrilled for current thing being available at $X for X = $2/$20/$200/$2000...
Llama 8B is like 17ish gigs. You can throw that onto a single 3090 off ebay. 700 for the card and another 500 for some 2nd hand basic gaming rig.
Plus you don't need a 4 slot PCIE mobo. Plus it's a gen4 pcie card (vs gen3). Plus skipping the complexity of multi-GPU. And wouldn't be surprised if it ends up faster too (everything in one GPU tends to be much faster in my experience, plus 3090 is just organically faster 1:1)
Or if you're feeling extra spicy you can do same on a 7900XTX (inference works fine on those & it's likely that there will be big optimisation gains in next months).
> Llama 8B is like 17ish gigs. You can throw that onto a single 3090 off ebay
Someone correct me if I'm wrong, but I've always thought you needed enough VRAM to have at least double the model size so that the GPU has enough VRAM for the calculated values from the model. So that 17 GB model requires 34 GB of RAM.
Though you can quantize to fp8/int8 with surprisingly little negative effect and then run that 17 GB model with 17 GB of VRAM.
These costs don't line up with my own experiments using vLLM on EKS for hosting small to medium sized models. For small (under 10B parameters) models on g5 instances, with prefix caching and an agent style workload with only 1 or a small number of turns per request, I saw on the order of tens of thousands of tokens/second of prefill (due to my common system prompts) and around 900 tokens/second of output.
I think this worked out to around $1/million tokens of output and orders of magnitude less for input tokens, and before reserved instances or other providers were considered.
Interesting, I think how the model runs makes a big difference and I plan to re-run this experiment with different models and different ways of running the model.
Does anyone know the impact of the prompt size in terms of throughput? If I'm only generating 10 tokens, does it matter if my initial prompt is 10 tokens or 8000 tokens? How much does it matter?
I just bought a $1099 MacBook Air M3, I get about 10 tokens/s for a q5 quant. Doesn't even get hot, and I can take it with me on the plane. It's really easy to install ollama.
Until January this year I mostly used Google Colab for both LLMs and deep learning projects. In January I spent about $1800 getting Apple Silicon M2Pro 32G. When I first got it, I was only so-so happy with the models I could run. Now I am ecstatically happy with the quality of the models I can run on this hardware.
I sometimes use Groq Llama3 APIs (so fast!) or OpenAI APIs, but I mostly use my 32G M2 system.
The article calculates cost of self-hosting, but I think it is also good taking into account how happy I am self hosting on my own hardware.
I own an 8 GPU cluster that I built for super cheap < $4,000. 180gb vram, 7 24gb + 1 24gb. There are tons of models that I run that's not hosted by any provider. The only way to run it is to host myself. Furthermore, the author has 39 tokens in 6 seconds. For llama3-8b, I get almost 80 tk/s and if parallel, can easily get up to 800 tk/s. Most users at home infer only one at a time because they are doing chat or role play. If you are doing more serious work, you will most likely have multiple inference running at once. When working with smaller models, it's not unusual to have 4-5 models loaded at once with multiple inference going. I have about 2tb of models downloaded, I don't have to shuffle data back and forth to the cloud, etc. To each their own, the author's argument is made today by many on why you should host in the cloud. Yet if you are not flush with cash and a little creative, it's far cheaper to run your own server than in the cloud.
To run llama-3 8b. A new $300 3060 12gb will do, it will load fine in Q8 gguf. If you must load in fp16 and cash is a problem a $160 P40 will do. If performance is desired a used 3090 for ~$650 will do.
I am looking into renting Hetzner GEX44 dedicated server to run a couple models on with Ollama. I haven't done the arithmetics yet but I wouldn't be surprised to see a 100x cost-decrease compared to OpenAI APIs (granted the models I'll run on the GEX44 machines will be less powerful)
What kind of setup were you able to do for so cheap? I'd love to be able to do more locally. I have access to a single RTX A5000 at work, but it is often not enough for what I'm wanting to do, and I end up renting cloud GPU.
I agree with most of the criticisms here, and will add on one more: while it is generally true that you can’t beat “serverless” inference pricing for LLMs, production deployments often depend on fine-tuned models, for which these providers typically charge much more to host. That’s where the cost (and security, etc.) advantage for running on dedicated hardware comes in.
Except that the article assumes that the GPUs would be using their max TDP all the time, which is incorrect. GPUs will throttle down to 5-20w (depending on the specific GPU). So your actual power consumption is going to be much, much lower, unless you’re literally using your LLM 24/7.
I checked some logs from my past experiments, the decoding went for about 400 tps over a ~3k token query, so about 7 seconds to process it, and then the generation speed was about 28 tokens.
does aws not have lower vcpu and memory instances with multiple T4s? because with 192gbs of memory and 24 cores, you're paying for a ton of resources you won't be using if you're only running inference.
This is a good way to do math. But honestly, how many products actually have 100% utilisation. I did some math a few months ago but mostly on the basis of active users, on what would be the % difference if you have 1k to 10K users/mo. You can run this as low as $0.3K/mo on Serverless GPUs and $0.7K/mo on EC2.
There's also the option of platforms such as BentoML (I have no affiliation) that offer usage-based pricing so you can at least take the 100% utilization assumption off the table. I'm not sure how the price compares to EKS.
This is another one of those "I used this for 5 minutes and found this out" naive posts which add nothing useful.
Check out the host LLM's at home crowd. One app to look at is llama.cpp. Model compression is one of the first techniques to successfully run models on low capacity hardware.
Yup. True. Should say "will" - currently free but heavily rate-limited. Together AI looks to be about $0.30 / 1M tokens, as another price comparison. Which you can pay for.
I’ve used llama3 on my work laptop with ollama. It wrote an amazing pop song about k-nearest neighbours in the style of PJ and Duncan’s ‘Let’s Get Ready to Rhumble’ called ‘Let’s Get Ready to Classify’
For everything else it’s next to useless.
With dstack you can either utilize multiple affordable cloud GPU providers at once to get the cheapest GPU offer or also use an own cluster of on-prem servers. Dstack supports both altogether.
Disclaimer: I’m a core contributor to dstack
Up until not too long ago I assumed that self-hosting an llm would come at an outrageous cost. I have a bunch of problems with LLM's in general. The major one is that all LLMs(even openAI) will produce output which will give anyone a great sense of confidence, only to be later slapped across the face with the harsh reality-for anything involving serious reasoning, chances are the response you got was at large bullshit. The second one is that I do not entirely trust those companies with my data, be it OpenAI, Microsoft or Github or any other.
That said, a while ago there was this[1] thread on here which helped me snatch a brand new, unboxed p40 for peanuts. Really, the cost was 2 or 3 jars of good quality peanut butter. Sadly it's still collecting dust since although my workstation can accommodate it, cooling is a bit of an issue - I 3D printed a bunch of hacky vents but I haven't had the time to put it all together.
The reason why I went this road was phi-3, which blew me away by how powerful, yet compact it is. Again, I would not trust it with anything big, but I have been using it for sifting through a bunch of raw, unstructured text and extract data from it and it's honestly done wonders. Overall, depending on your budget and your goal, running an llm in your home lab is a very appealing idea.
Hetzner GPU servers at $200/month for an RTX 4000 with 20GB seem competitive. Anyone have experience with what kind of token throughput you could get with that?
Yeah, or you can get a gpu server with 20GB VRAM on hetzner for ~200 EUR per month.
Runpod and DigitalOcean are also quite competitive on prices if you need a different GPU.
Kind of a ridiculous approach, especially for this model. Use together.ai, fireworks.ai, RunPod serverless, any serverless. Or use ollama with the default quantization, will work on many home computers, including my gaming laptop which is about 5 years old.
Assuming we want to mirror our setup in AWS, we’d need 4x NVidia Tesla T4s. You can buy them for about $700 on eBay.
Add in $1,000 to setup the rest of the rig and you have a final price of around:
$2,800 + $1,000 = $3,800
This whole exercise assumes that you're using the Llama 3 8b model. At full fp16 precision that will fit in one 3090 or 4090 GPU (the int8 version will too, and run faster, with very little degradation.) Especially if you're willing to buy GPU hardware from eBay, that will cost significantly less.
I have my home workstation with a 4090 exposed as a vLLM service to an AWS environment where I access it via reverse SSH tunnel.