> I'll admit that to some degree RethinkDB is responding to what we see as the direction the market is moving.
Is this really true? I mean, it is clear the majority of stuff you read about is from people who don't like types, but honestly as time has marched on development has been "democratized" to a large extent... yet, if I think about "people with both a lot of experience and a lot money who are willing to both pay for things to use and people to maintain those things" (what I would call "the market" for a database solution) I see lots and lots of rigid types.
I mostly see "schemaless" from highly-popular (to the masses) database system where I don't really see much money being made or even moved (on database software). It could be, however, that I'm just totally misunderstanding how well some of these companies are doing. However, when I see a company evaluating things to plunk money down on, they seem to be 1) clustering support for PostgreSQL, 2) hosted management of MySQL, and 3) system administrators and licenses for Oracle.
In contrast, people providing NoSQL "schemaless" systems seem mosty to be living in a world where the developers expect everything to be free; they sell support contracts, but I never hear of anyone actually buying them... do a lot of companies do this? It also always sounds like a magical accomplishment when you hear one being used a large company with a lot of money (such as the developers at UbiSoft giving a talk on how they are using CouchBase at re:Invent)... I haven't heard RDBMS systems have to justify "look, this company used us at scale and we worked fine" in years.
Note: to be clear, I do hear of companies with "NoSQL" systems who have a lot of money, but they are all using Casaandra, HBase, or SOLR. SOLR let's you make define dynamic columns, but everything is still rigidly typed. HBase and Cassandra both have schemas with fixed columns; sure, Cassandra allowed you to largely ignore the schema, but if you've seen CQL3 they are actually nigh-into turning into an RDBMS: it is hilarious, and I think awesome. Even then, it isn't clear to me how much these companies pay money for their databases.
> Changing around static types in your code isn't anywhere near as hard as changing around a static structure in your database because you need to modify all the old data to make it meet the new structure.
This is not actually true, as the structure can be metadata stored describing the data. When I add or delete fields in PostgreSQL, the operation is instantaneous, as is renaming existing fields. AFAIK, only MySQL sucks at this. The only time I need to do a table rewrite is if I change the datatype of an existing field, and I am pretty certain that you can also handle the majority of those cases without a rewrite. Even with the rewrite, you shouldn't have to block incoming queries.
It isn't like not having a schema fixes this, as there is a schema after all... my code certainly has an assumption about how the data is stoted. It just means I either a) can't change the schema at all, b) can change the schema, but have to do so by doing the table rewrite manually, or c) can manually implement the metadata behavior (but then why not just add this to the database server). Given how the patterns one uses to solve these problems don't really rely on the data being stored, it simply makes sense to standardize them in the database server.
slava @ rethink here. DB innovation moves slowly. It takes time for people to adjust to new technology, for new technology to adjust to market demands (DBs are fundamentally more difficult to develop than most other software, so it takes more time), and for existing large companies to adopt new technology. As a result, the NoSQL field is still evolving, and there hasn't been a product yet that really nails it. We hope Rethink will be it, but it will take a little bit of time for that to happen.
> I haven't heard RDBMS systems have to justify "look, this company used us at scale and we worked fine" in years.
The main reason you've not heard this recently is because RDBMS have been around for 30+years. If we could go back to those times we'd probably see exactly the same things. The Oracle Wikipedia page actually lists the first customer of their solution [1]
Also if you take a look at the analytic databases (which are built on top of the relational model, but more recent), you'd notice exactly the same behavior.
IMO this has nothing to do with the maturity or approach of the products, but rather with the adoption cycle [2].
Well, that is obvious: I certainly knew that while writing this, which is why I provided the comparison between schemaless and with-schema newer database systems and the trajectories they seem to have (such as Cassandra becoming more rigid as time moves on), and looked at the kind of people who are interested in both. Database systems that involve schemas seem to be mostly on a downturn in money spent on them because free alternatives are improving in quality and are becoming easier to support: not because their demand has slowed.
However, it isn't clear to me that companies making schemaless systems are really making any real money at all, and they seem to be marketing themselves towards people who don't even have budgets for database tools, and probably have no people dedicated to database administration. (Does UbiSoft even pay for CouchBase support? It didn't sound like it ;P.) Can you demonstrate in these other cases that Oracle, for example, was first used by people who thought it was "cool" and insisted on it being free, and worked their way up to "people should pay $150k a year on this"?
FWIW, I really am curious about this, and would be perfectly happy being wrong. It just seems like a weird decision, and it isn't clear that it is base on "the needs of people who have budgets for this sort of thing is or will change" (which is what I'd argue was the case for "eventually you will outgrow your filing cabinet and even your spreadsheets and yes, even your mainframe"; the business case for these earlier shifts was clear: "large companies spend $Xm per year on record retention departments, those could be replaced by computers!"), as much as "people who don't spend money on database systems at all find this more exciting".
Looking into it, it seems I had forgotten you are the new MemBase, so a lot of companies are using you as a best-of-breed cache (supporting automatic and effificient repartitioning as well as on-disk persistence) , where the schema you want is just blob->blob anyway. My argument then either a) then treats you as "a database with a fixed schema that happens to be exactly the one people want or b) doesn't work on you at all, as the market you are in is more directly competing with Oracle Coherence, not Oracle RAC (and apparently you eat Oracle Coherence's lunch... apparently I should be evaluating you guys as my cache ;P).
> Can you demonstrate in these other cases that Oracle, for example, was first used by people who thought it was "cool" and insisted on it being free, and worked their way up to "people should pay $150k a year on this"?
No, I cannot do that :-). What I think you are noticing is a change in the business and adoption models. While not being an economist, my hypothesis is that open source was one of the main causes of these changes.
Going back to your question, the closest things that come to mind in the data space that could give us some more info are:
1. Neo4j and InfiniteGraph (graph databases)
2. Cloudera and MapR (Hadoopy)
As far as I know (unfortunately without any real data though) is that both Neo4j and Cloudera are doing quite well financially. On the other hand that's not to say that the model chosen by InfiniteGraph and MapR isn't working.
Is this really true? I mean, it is clear the majority of stuff you read about is from people who don't like types, but honestly as time has marched on development has been "democratized" to a large extent... yet, if I think about "people with both a lot of experience and a lot money who are willing to both pay for things to use and people to maintain those things" (what I would call "the market" for a database solution) I see lots and lots of rigid types.
I mostly see "schemaless" from highly-popular (to the masses) database system where I don't really see much money being made or even moved (on database software). It could be, however, that I'm just totally misunderstanding how well some of these companies are doing. However, when I see a company evaluating things to plunk money down on, they seem to be 1) clustering support for PostgreSQL, 2) hosted management of MySQL, and 3) system administrators and licenses for Oracle.
In contrast, people providing NoSQL "schemaless" systems seem mosty to be living in a world where the developers expect everything to be free; they sell support contracts, but I never hear of anyone actually buying them... do a lot of companies do this? It also always sounds like a magical accomplishment when you hear one being used a large company with a lot of money (such as the developers at UbiSoft giving a talk on how they are using CouchBase at re:Invent)... I haven't heard RDBMS systems have to justify "look, this company used us at scale and we worked fine" in years.
Note: to be clear, I do hear of companies with "NoSQL" systems who have a lot of money, but they are all using Casaandra, HBase, or SOLR. SOLR let's you make define dynamic columns, but everything is still rigidly typed. HBase and Cassandra both have schemas with fixed columns; sure, Cassandra allowed you to largely ignore the schema, but if you've seen CQL3 they are actually nigh-into turning into an RDBMS: it is hilarious, and I think awesome. Even then, it isn't clear to me how much these companies pay money for their databases.
> Changing around static types in your code isn't anywhere near as hard as changing around a static structure in your database because you need to modify all the old data to make it meet the new structure.
This is not actually true, as the structure can be metadata stored describing the data. When I add or delete fields in PostgreSQL, the operation is instantaneous, as is renaming existing fields. AFAIK, only MySQL sucks at this. The only time I need to do a table rewrite is if I change the datatype of an existing field, and I am pretty certain that you can also handle the majority of those cases without a rewrite. Even with the rewrite, you shouldn't have to block incoming queries.
It isn't like not having a schema fixes this, as there is a schema after all... my code certainly has an assumption about how the data is stoted. It just means I either a) can't change the schema at all, b) can change the schema, but have to do so by doing the table rewrite manually, or c) can manually implement the metadata behavior (but then why not just add this to the database server). Given how the patterns one uses to solve these problems don't really rely on the data being stored, it simply makes sense to standardize them in the database server.