Database 2.0 – Part III
The next database won’t require a schema. It won’t require special tools to manage the tablespace. It won’t require a JDBC connection and it won’t require an ORM framework. You’ll give it an object and it will store it. You’ll ask for an object and you’ll get one back. Most importantly, it will be intrinsically aware of transactions. You’ll be able to query transactions, audit them and report on them, construct statistical charts: without a single line of code.
Let’s go back a step. Why do we need a schema, and why do we need a O-R mapping? Because databases unwrap objects into flat tables and we need a way to repackage them on retrieval. Eliminate the packaging problem and you’ve solved about 50% of the impedance mismatch problems that exists today. This is nothing new – it has already been done with native object databases.
Why do we need a connection to the database? Because a database resides on a separate machine or, at least, executes in a separate process. Why? Because databases are too slow to run on the same machine as the application server in production. If they would, the whole system would just become bogged down with serving users and storing data. Solve this problem and you’ve eliminated another 25% of the issues. Keep the connection option open for clustering and scalability purposes, but for medium to small scale installations there should be no compelling reason to isolate the database from the rest of the application.
The remaining 25% is tied up in transactions. Databases are aware of transactions insofar as to avail recoverability. Once a transaction completes the data is guaranteed durability, but all records of the transaction are lost. Who instigated the change, when, why, what sort of data was changed? These questions can only be answered if we had made extra provision for the transitional data, by keeping it in a separate set of tables, a flat file, on tape or otherwise.
Speed and transaction-awareness is where the emphasis is at. And this is exactly what DTS/S1 ‘Pitch Black’ is all about. Pitch Black represents all the knowledge about relational and object databases that I have acquired in over 5 years. Its a transaction-oriented database. This piece of software has kept all of the ‘good’ in databases (hardly anything) and has changed all that is ‘bad’ (pretty much everything).
I’ve referred to this Gen 2.0 product a ‘transaction-oriented database’. Unlike everything else out there, everything that Pitch Black does is auditable. Like all other persistent entities, these audits are available to the application as plain old Java objects (POJOs). This changes the playing field completely. You don’t have to buy a transaction processor with mediocre database search and retrieve capability. Nor do you have to buy a mainstream RDBMS or ODBMS and then hand-code the transactional parts. There is now a single product that can do both these things, and can do them well.
We didn’t just go and build Pitch Black to prove a point. Or even to earn some money. I’m the managing director of Obsidian Dynamics and our sister company ventured into financial markets and set out to build a next generation trading platform for Web 2.0. In case you haven’t noticed, 99% Web 2.0 sites aren’t commercial. The whole Web 2.0 umbrella is about openness, building communities, peer interaction. But commercial trade has been a foundation for building communities for centuries. So we decided to amalgamate start-up venture capital and general principles of a securities exchange with the Web 2.0 community. The traditional VC model, just like about anything else, has flaws. In light of the recent economic downturn these flaws are becoming more apparent. The core problem is that a small handfull of people decide whether a new startup should be granted funds. If they’re wrong in their decision, the outcome will amount to either a bad investment and massive financial write off, or a missed opportunity. One solution to this problem is to bring a startup company to the collective attention of millions of Internet users, and let the people decide in a startup’s future by investing their own money.
We have researched this model thoroughly from both a financial and a technical perspective. We came up with the following conclusion. When building a Web 2.0 business, in most cases its fine to slap some PHP on the front, ram a MySQL through the back, and Bob’s your uncle. For a majority of Web 2.0 auditability is a non-requirement. But when you’re taking money from people and issuing them with securities, conventional tooling is insufficient, and you would otherwise be forced into writing a transaction processor on top of a relational database. As you know by now, I’ve had it with databases and I simply wouldn’t stand for it.
Right from the word go, we’ve designed Pitch Black to unify all enterprise data storage requirements into a single, cohesive solution. We have achieved this by closely mimicking the heap allocation model of memory-managed development environments such Java, and depicting similar object-lifecycle semantics in a persistent storage model. In plain terms, what this amounts to is the ability for application developers define classes as they normally would, instantiate those into objects, manipulate various fields and have those changes transparently committed to stable storage. Transactions here are critical. Pitch Black doesn’t simply use the concept of a transaction to populate its REDO log, it keeps transactions indefinitely and exposes them to the application layer when required.
The strength of what we’ve built is that nothing what exists was put in as an afterthought. Every single feature of Pitch Black existed in our minds and our hearts before it was implemented. The resulting product is precisely what we thought it would be and fit with our business model perfectly. But at the end of the day, the trading platform is only half the reason why we built it. Its is a fully featured commercial turnkey solution that anyone can use right off the shelf.
At this point in time its difficult for me to speculate on the future success of Pitch Black. We’re using it internally and we are loving it. But one thing I do know for sure is that we were the first ones there: the first turn a “Database 2.0″ concept into a reality, and no-one’s going to take that away. Being the principal developer of DTS/S1, what thrills me is that if anyone else comes up with an implementation, they will be motivated partially by us and the precedent that we’ve set. We’ve made something that will work for years to come, and this makes me feel liberated. Finally I don’t have to do what I’ve been doing all these years, which is what I’ve HAD to do. Now, I can do what I WANT to do. I can concentrate on building a business app by thinking about it in terms of objects, components, models – going forward and seeing things work rather than worrying about SQL, database licensing, fixing bottlenecks, tuning queries, ORMs and all that’s evil in this world. A transaction-oriented database is here. And we’re going to fight hard to make it stay.
Emil
http://obsidiandynamics.com/
