Practical Bayesian Optimisation of Machine Learning Hyperparameters [1] was a great recent paper on this topic that looked to have very nice results in practice across a range of models. Is there any interest in implementing that algorithm in hyperopt?
Hey, hyperopt author here. I started something of a spearmint binding [1] but I didn't get very far yet. The main trouble is that spearmint currently does not have proper support for parameters that apply "sometimes". There's a paper at the upcoming NIPS workshop on Bayesian optimization with an interesting scheme for doing that, but still very much a research question.
From the documentation: "Hyperopt has been designed to accommodate Bayesian optimization algorithms based on Gaussian processes and regression trees, but these are not currently implemented." [1] So it sounds like they've at least thought about it.
For those that don't know: a Bayesian optimization approach based on Gaussian processes has been implemented in Python by the author of that paper [0].
Here's a nice blog post with some information on some problems it can run into in practice [1].
The purpose of a hyperparameter opt algo is definitely to have fewer than the thing it is configuring, and definitely for it to be faster to get a good model by just running the default search policy than by trying to rig up your own meta-algorithm on top of it.
Well, there is such a thing as hyperheuristics, which are related to what you're saying (perhaps the name metaheuristic would've been used for this, but it was already taken). While theoretically possible, I don't think it makes much practical sense.
Different configuration search strategies will perform differently on different fitness landscapes, and fitness landscapes depend on both data and learning algorithm. An online bandit-style selection of what seems to be working could be a good idea. Portfolio-management algorithms do this sort of thing. Hyperopt can accommodate this type of algorithm. File a ticket on the issue tracker if you'd like to work on this and I'll help you get started.
It would be nice if something like the Efficient Global Optimization Algorithm were implemented. In my opinion this would be the "automagical" way of optimizing hyper parameters.
My hunch is that this algorithm, like GP-based strategies will excel in small numbers of dimensions, but struggle in high numbers of dimensions, especially with conditional parameters. But it's a classic algorithm, and a great fit for some problems, so it would be a nice addition to the lib.
Thanks for the HN post @chewxy. I just wanted to mention a bit of progress status. The version on pypi has been a workhorse.
We're currently working on a not-so-minor update that will include a major upgrade in code quality, better documentation, probably include two new optimization algorithms, and come with a wrapper around scikit-learn. All that should be happening by February.
This looks nice, though I feel like this is one of those cases where it's easier to code it yourself than to learn and use the library API. At least until the BOA stuff is implemented, and perhaps even then.
[1]: http://arxiv.org/pdf/1206.2944.pdf