OpenpPipe - https://openpipe.ai/ - is probably the service that most closely resembles what you’re asking for, but I found the evals weren’t really what I wanted — i.e. following my custom evaluation criteria — so you probably will end up having to do that yourself anyway. But for the finetuning, they’re all somewhat the same. Predibase and OpenPipe are two good options for that. Predibase has more base models for you to finetune, but it’s a bit more unwieldy to work with. I wrote about that in a previous post here -- https://mlops.systems/posts/2024-06-17-one-click-finetuning.....
(Disclaimer: founder of OpenPipe). Thanks for the shout-out. Note that we're actively working on improved evaluations that will let you add more specific criteria as well as more evaluation types, like comparing field values to that of a golden dataset. This is definitely something that customers are asking for!
Wild to see them advertising collecting GPT4 responses for training other models. That’s definitely not allowed by TOS. I suspect many do, but front page advertising is another thing entirely.
Predibase ( http://predibase.com ), also referred in the article, is a platform specifically designed for exactly that. It also has "repos" for finetuning multiple models and comapre their performance and keeping things organzie. It also allow you to query any of the finetuned models on the fly from a single GPU with multi-lora serving. (Predibase founder here)
