Comments on: InnoDB is dead. Long live InnoDB!

By: Alaric Snell-Pym

Alaric Snell-Pym — Mon, 30 Nov 2009 18:24:29 +0000

Synchronous replication isn’t the only way to go… sure, it’s bearable on small clusters, but when you have tens of slaves, it gets painful.

Here at GenieDB, we’ve done some tricks with asynchronous replication combined with synchronously updating a chosen-per-record-with-a-hash-of-the-PK ‘consistency buffer’ server that stores the record in-memory for long enough for the asynch replication to happen, thereby getting around the problem with a small latency increase (the consistency buffer is highly optimised for low latency; we use memcache!) that doesn’t grow as the number of replicas does.

By: Vasil Dimov

Vasil Dimov — Fri, 11 Sep 2009 16:11:01 +0000

Mark,

I agree with you 🙂

By: Mark Callaghan

Mark Callaghan — Fri, 11 Sep 2009 15:33:15 +0000

Vasil,
When the server is on a 3 to 4 year release cycle and the current release does not scale on commodity HW, then you should make these changes to the GA release or users will go elsewhere (PostgreSQL, XtraDB). The InnoDB plugin has made 5.1 much more attractive as an upgrade target.

By: Vasil Dimov

Vasil Dimov — Fri, 11 Sep 2009 15:25:59 +0000

Hi,

You can’t expect new features or significant performance improvements in a GA branch. This is what GA is by definition. Hot features and stability are adverse.

By: shlomi

shlomi — Fri, 11 Sep 2009 09:29:44 +0000

Hi Vasil,

Thanks. I’m not referring to the fact that InnoDB in 5.1 & 5.0 is now frozen to new features.
I’m not referring to bug fixes, as well.

I’m wondering if any further development is going to take place on builtin InnoDB. My impression is that all further development, performance improvements etc. goes to InnoDB plugin, and I suspect that the Plugin will become that main InnoDB distribution from now on.
For example: will the builtin InnoDB code get the group-commit fix? Is it planned to get this fix in future versions?

My impression is that it will not. I may be completely wrong; but it appears this way. If you know this isn’t the case at all, please say so. But I believe I’m not the only one who got the impression I expressed here. Therefore, Innobase/Oracle may wish to speak openly about InnoDB development plans.

Regards,
Shlomi

By: Vasil Dimov

Vasil Dimov — Fri, 11 Sep 2009 06:37:26 +0000

“I only saw (or was exposed to, at least) progress on the plugin. The way I understand it, the plugin is the main (and only?) focus of development. And this is the significant thing to consider: if you’re keeping to “old InnoDB”, fine – but it won’t get you much farther; you’re unlikely to see great performance improvements”

Hi, the builtin InnoDB in MySQL 5.1 and the whole MySQL 5.1 is GA and frozen for new features and big/risky changes. This is a common sense – you don’t risk the stability of the stable branch with new features. Of course 5.0 and 5.1 are getting all the bugfixes, so they are nowhere near “stalled”.

By: Andy Grove

Andy Grove — Thu, 10 Sep 2009 20:38:17 +0000

@Mark – What I can say is that we co-ordinate with the database transaction to ensure that we record the order that the commits happen and then make sure we apply the transactions in the same order on the slave.

@Shlomi – Yes, we do have higher concurrency on the slave when using dbShards. Transactions are effectively written to the master and slave machines in parallel with the same concurrency (although it should be noted that we do not write to the slave DB at this time, just to slave memory or log). We know that the dbShards log reliable replication is much faster than the master DB, so this is how we have such a minimal effect on performance. The main evil of replication lag is what happens if the master fails before transactions have been communicated to the slave. With our approach we do not lose any transactions if this happens.

By: Mark Callaghan

Mark Callaghan — Thu, 10 Sep 2009 18:42:06 +0000

Andy,

OK, and I will add dbShards to the list of topics I won’t discuss. I don’t want to interfere with your business, but I also don’t want to waste my time doing marketing for it.

By: shlomi

shlomi — Thu, 10 Sep 2009 18:37:24 +0000

@Mark – thanks

@Andy – to fight replication lag you would need more than slowing down the master. Does your solution make for higher concurrency on the slave? You say you do not serialize the commits. Does it follow that you are able to pass them on synchronously to the slave with original concurrency?

By: Andy Grove

Andy Grove — Thu, 10 Sep 2009 18:31:19 +0000

That was the hard part in building the product. It’s actually part of our patent-pending technology so unfortunately I can’t go into the detail of how we implemented but we do not serialize the commits (we tried that approach early on and performance was terrible).