The sessions are expected to be technical, and I'm happy the proposals follow this guideline. I use Giuseppe's and Baron's general guidelines for submitting a proposal. I wish to humbly add a couple notes myself.

Be fairly brief

Explain your session/tutorial as clearly as you can. The reader should be able to get the general impression of the session from two or three paragraphs. Some can make a point in two sentences; for most it would take somewhat more.

If you're going to talk about some database feature, for example, please do not write the manual for that feature. That's for the session itself. Just explain how you're going to discuss the feature, and why it should be of interest (e.g. what the benefits of this feature are, the risks or pitfalls, the ingenious C code behind it or the quirks of the operating system involved).

Clarify

It's important for me to understand two things when reading a proposal, which establish the grounds for better evaluating the proposal:

• Who the target audience is (newbies, developers, DBAs, Linux internal experts etc.)
• To what depth are you going to deliver the content you describe.

That is not to say you should explicitly state "This session is for MySQL DBAs", but the attendee should be able to easily decide whether your session appeals to his type of work or expertise. I, myself, have happened upon sessions that were completely different from what I expected. To illustrate, I give two examples, while not disclosing the exact details:

• A session which was about locking in database. I got the impression it was about ways to avoid locking, issues with mutexes etc. It turned out to be a discussion between the presenter and a few member of the audience about the specific code internals, lines 667-684 in the lock_module.cc file, and the recently reported bug. To me it was more like the weekly rnd meeting of some company. I couldn't understand anything of the entire talk.
• A session promising insight on working out great scale-out with some product: I was expecting to hear of the "DOs and DON'Ts", or of great configuration and implementation tricks on the subject. However, it turned out to be more of a general talk on "how we used the product in our company and found it to work great".

The two sessions above were perfectly valid, and had their place in the conference. But were poorly described in the two respects I mentioned.

A great submission, in my opinion, is one where attendees get what the expect, and don't shyly leave the conference room 15 minutes into the talk.

Submit a proposal here.

Thursday

• OQGRAPH: Dealing with Graphs and Hierarchies in Plain SQL / Antony Curtis (Blizzard Entertainment)

I opened this day's sessions with a smile.

Antony Curtis and Arjen Lentz have authored the OQGraph storage engine a while back. I had every intention to try it out, but never got round to it. Which is why I was happy to find this session in the conference. OQGraph is a specialized engine, and comes to solve hierarchal or otherwise graph-related queries, such as: "who are the descendants of a given node", "find a route from a to b", etc. MySQL does not support the RECURSIVE syntax as Oracle does, and does not provide out-of-the-box solution for that.

The syntax is just excellent: you just write something like SELECT * FROM my_graph WHERE from_node = 'empusa fasciata' AND to_node = 'homo sapiens' to find a route. Otherwise just use WHERE from_node = 'Heathrow Central' to find all outgoing links. So this is just plain old SQL, no new syntax involved.

I rounded corners. It is also possible (and required) to specify an algorithm. Do you want Djekstra? BFS? You specify it in the query. The result of a route query is a rowset, where each row is a step in the route, along with its sequence within the route. So you can do your ORDER BY, LIMIT etc. I find that syntax-wise, OQGraph is very intuitive!

Implementation-wise, there are two versions: 2 (stable) and 3 (testing), which are very different. In version 2 you must create and populate an OQGraph table. You must populate it with from-to;weight values. The entire table is stored in memory a-la MEMORY engine. In version 3 the OQGraph is not directly populated. Instead, you use the CONNECTION table property to identify a real table where the data resides, along with the names of the relevant columns. So your data can reside within your normal InnoDB table, and your queries will follow your normal isolation-level rules.

Graph search implementation is done via the boost() library. This means that information must be loaded in memory. Tests show that up to a million or a few million edges are as much as OQGraph can take. So right now it may not fit your requirements is you need larger datasets.

Antony seemed like he was enjoying every minute of writing OQGraph, was very aware of current limitations, and offered great ideas on improvement. The version 3 design is a huge improvement over version 2, and I can't wait to see how it evolves!

• Error Detection and Correction with MySQL Replication / Daniel Peek (Facebook)

An interesting view into Facebook's attempt to locate & define data drifts within MySQL replication. Apparently, Facebook guys realized there were some inconsistencies between masters and slaves. They embarked on a mission to find out what exactly was inconsistent, and why.

Daniel Peek described their method of detecting changes. In similar manner to Maatkit's mk-table-checksum, they use hash codes over chunks of rows. To be more specific: in order to detect the differences (if any) between master & slaves on a given table, they iterate said table in chunks, such that first line of any chunk overlaps with last line of previous chunk. This is easily done when there's unique key, less so when there isn't. They copy rows to a utility table a chunk at a time (e.g. 1,000 rows at a time), and take checksum. Their script then compared checksum on master & slaves. If it equals, they conclude (at very high probability) that tested chunk is identical. Otherwise they take action to detect the exact row that is changed.

When the two chunks do not have the same number of rows, then there's at the very least a missing or extra row in either master or slave. Otherwise, there's a change in column's data. Apparently, nearly all data drifts are changes in column data, not missing or extra rows.

To make a long story short, they realized most of their data drift related to TIMESTAMPs. After realizing some timezone settings were incorrect, they were still left with 0.00056% (if my memory serves me right) worth of data drift. How often do they do this test? Once, is the reply. What is the meaning of 0.00056%? We don't know, is the answer.

And, although we are left with unanswered questions: Why does that happen? How can we detect such changes in reasonable time? What should we do once we realize there is a drift? Whom shall we trust? We have gone to length to recognize a way to detect such drifts.

• Summary Tables, Aggregate Tables and Materialized Views Using Flexviews / Justin Swanhart (Percona)

The conference had to end with a bang. In the last three years (hopefully with some sleep), Justin Swanhart has been working on what seems to me like an utterly (positively) crazy project: Flexviews: a materialized views solution for MySQL.

MySQL's native views are immaterialized. There is no data in them. When you access the view, you directly query the tables beneath. Justin's code allows you to create a materialized view (implemented as an actual MySQL table), which can utilize almost any type of SELECT you would want to use. It can only SELECT tables (including other materialized views).

How does this work? You don't directly create your materialized view. You get a set of stored routines with which you create the view. Each provides with a simple step in the creation of the view (e.g. JOIN to this table, return this column, etc.). You don't have to worry too much about this, since Justin provides with a PHP script which translates a SQL query to the required invocation sequence of said routines.

OK, view is created. What then? Flexviews reads binary logs (must be in ROW binlog format), and decides whether logged action manipulates tables used by any materialized table. If so, it logs that action into a special log table.

What have we got so far? SQL parser which generates sequence of stored routines; sequence of stored routines which creates a table, while storing structure; Binlog reader, capable of recognizing relevant events, associating them with relevant materialized views; and, most importantly, some brains which can figure out just how said event affects data. Think about it: say our query did some aggregation, with COUNT(*) or SUM(column). To be able to update the aggregated result based on the fact a few rows have been added/deleted/updated sounds to me like very hard work.

Flexviews do not get updated immediately, but rather on demand. It's all in the logs, it's just a matter of when to apply the logs. We had a very nice & simple demonstration of this.

I don't mean to be too superlative, but this was a very impressive session, and myself, as well as other people, all had an occasional "wow" slip during and concluding this talk.

Throughout the week

• Planet MySQL

During conference, Planet MySQL was swamped with various announcements. Little were technical announcements, like "X.X has been released", and most were purely marketing announcements. This is to be expected; everyone wants to make a living. I think it is quite all right that for 4 days a year, the planet is mostly marketing stuff. I personally mostly ignore the planet during these days.

Wednesday

• BoF: Replication, Lars Thalmann, Mat Keep (Oracle)

Lars Thallman presented the MySQL 5.5 replication features, as well as the expected 5.6 features. Among other features, one could notice parallel replication, binlog checksums, sub-second resolution and more. There was an open discussion about these features, asking for comments; looking for new ideas and suggestion from the audience.

I can't possibly cover it all. I'll note two discussion I participated in, and which have interested me. This also serves for noting down to myself my ideas and thoughts.

Parallel replication

As the trend goes, parallel replication will be implemented by having a thread per schema. My own contribution to the discussion is that with such paradigm, the slave may be at any state inconsistent with the master. That is, at any point in time it can be in such state that has never existed in any point in time on the master. This is because it is possible that one or more of the replication threads executes much faster than one or more of the others.

We have discussed the meaning of STOP SLAVE. I have suggested that STOP SLAVE works in such way that it stops in a consistent slave. Lars suggested there could be normal STOP SLAVE and STOP SLAVE CONSISTENT.

It has been noted that we may not care about the slave being inconsistent, as the prerequisite for parallel replication is that the distinct schemata are independent of one another. After some thought, I disagree. Today parallel replication may be implemented on different schemata; tomorrow - based on finer parameters. I think it's best to prepare for the inevitable future. Besides, STOP SLAVE INCONSISTENT is such an easy feature to implement!

Slave analysis

I also had a long time feature request. While writing management tools, I realized it is difficult to learn the topology of a replication environment by simple interrogation. If you connect to a master, you can either issue SHOW PROCESSLIST to detect the slaves, or SHOW SLAVE HOSTS.

The former does not tell you the port on which the slaves are listening. The latter requires that you configure report_host and report_port properly.

What I would like to see is a SHOW SLAVES STATUS command (or similar INFORMATION_SCHEMA table) on the master. Such that it lists connected slaves, one per row, with the following details:

• Slave_host (this is easy, already exists in SHOW PROCESSLIST)
• Slave_port (slave should send this kind of information to master upon first connect)
• Slave_master_log_file (last log file requested by slave)
• Slave_master_log_pos (last log pos requested by slave)
• Slave_seconds_behind_master (slave should report this upon each request)
• Slave_server_id (slave should send this upon connect)

By reading Slave_host, Slave_port & Slave_server_id, it makes it easy for us to build replication topology. It's a single operation to detect a master's slaves. It takes recursive steps to build entire topology.

By reading the Slave_master_log_file, Slave_master_log_pos, we can easily determine that binary logs up to that point have already been written to the slave's relay log. This makes it easier to decide how to purge master logs. Slave_seconds_behind_master completes the replication picture.

Wednesday

• Replication for Availability & Durability with MySQL and Amazon RDS / Grant McAlister (Amazon.com)

Grant McAlister described the Amazon RDS offer, which provides with a pre-installed MySQL servers, and supports auto management of replication and high availability. He desribed asynchronous vs. synchronous replication, logical (i.e log shipping & replaying) vs. physical replication.

Amazon implement physical replication by shipping data pages to a secondary, standby server, located at a different availability zone. A transaction does not complete before pages are shipped to, and acknowledged by standby machine. The standby machine writes data pages in parallel. This is similar in concept to DRBD. RDS uses InnoDB, which promises data integrity in case of power/network failure.

The fail over process, in case active master has crashed, involves blocking access to the active master, starting MySQL on standby master (promoted to be active), while changing elastic IP for master to point to promoted master. McAlister said this process takes a few minutes. Live demo resulted at about 4 minutes.

Actually, the recovery time is affected by the InnoDB transaction log. In replying my question, McAlister explained that the InnoDB trasaction log is not under the user's control. It is managed by Amazon. You can't just set it to 512MB.

Otherwise RDS also offers normal, asynchronous replication, so as to allow for read scale out.

I later on spotted Grant McAlister in the expo hall, and had the chance to ask a few more questions. Here is a very brief summary of a few Q&A items:

Q: Today, EBS performance is unexpected, since other users may be using the disk. Will you allow RDS customers to have dedicated EBS volumes?

A: We're working towards making this work.

Q: In many cases I've seen, EBS is very slow storage. People do software RAID over several EBS volumes to boost performance, but then loose the ability to do consistent snapshots. Any solution for that?

A: We get so much of that! We're working to change this as well, in time.

Q: Does an RDS user get access to the DB machine?

A: No. Though you do get the API for monitoring it.

Q: How much control do I get over the MySQL configuration file?

A: You have control over most of the settings; a few are under Amazon's control (e.g. control over innodb transaction log file size)

Q: How far have you went into changing MySQL code?

A: A little. Not far enough that compatibility breaks. People need compatibility. We also need to be quickly able to provide with new versions.

Q: If I need to upgrade MySQL version, does RDS provide with the mechanism to first upgrade slave, then do fail-over?

A: No. We just upgrade a server (i.e. master) in place.

Q: Any integration of collateral tools?

A: No. You have access to the database; you are free to run your own tools.

I hope my memory serves me correctly in recalling the above conversation.

• The Aspersa System Administrator's Toolkit / Baron Schwartz (Percona Inc.)

Baron presented with some of Aspersa's tools: ioprofile, diskstats, summary, mysql-summary, collect, stalk & sift.

The presentation was accompanied with demos, wither using live data or prepared, real world data. Not surprisingly, Baron came up with real customer cases, where the "standard" tools gave little help in way of analyzing the problem (e.g. iostat not separating read and write counters). Apparently this is a bash/sec/awk based toolkit, with bash unit tests. Cool!

Some of the tools are self explanatory; others require deeper understanding of either OS internals or of the patterns one should be looking for. Lot's to study.

Baron writes tools based on extensive experience of consulting & performance optimization work; not only of his own, but of Percona's entire team. When he says "this tool saves a lot of analysis time" or "with this tool we were able to analyze a problem we were unable to, or found it very hard to, solve with existing tools", you'd better check it out. In fact, I noticed the audience laptop usage during this talk was very low. There was a lot of attention in the crowd.

• Automatic Failover: Design and Reality / Ian Gulliver (Google), Eric Rollins (Google)

This session described the replication topology and failover design on one (undisclosed) of Google's products, using MySQL. Using XXX shards of MySQL, that is, each with XX servers in replication topology (we are asked to do the math ourselves).

Ian & Eric described the logic behind their sharding, scale out & failover process, and the sad reality. Apparently shards are implemented on many levels: multiple MySQL instances on single machine, multiples schemata in each MySQL instances, sharded data within schema. For scale-out, they prefer adding new machines, then distributing MySQL instances between them. There is nothing too complicated about this, once we are past syncing masters & slaves.

Speaking of which, their preferred way of taking snapshots is MySQL shutdown, then full file system copy. They do not want to bother with licenses. With XX servers, it should be fairly easy to take one down for backup/copy.

Otherwise they may be splitting schemata between servers, or, at the worst, managing the shards. An interesting note was that Google is known to use crappy hardware, which makes for a lot of hardware failures. Apparently, their solution needs to work a lot.

A very nice notion they presented is the empty shard. They have an empty shard, with complete replication topology, on which version upgrades or otherwise tests are done. Very nice! (Forgive me if this is plain obvious to you).

The logic part detailed the expected "which slave to promote", "how to modify the hierarchy" etc. procedures.

Besides being very interesting in itself, the session also posed the interesting notion, that most everyone (and in particular the BIG companies), use plain old MySQL replication for both scale out & high availability. I think this is an important notion.

• No Silver Bullet - The Slightly Less Painful Way to Sharding / Gwen Shapira (Pythian)

I've heard a lot about Gwen Shapira before, and this is the first time I heard her speak. I enjoyed her session. Gwen presented a vendor-neutral session on sharding. There were a lot of "how not to shard" notes. These were based either on her very early experience, before the word "sharding" came to be known, as well as recent experiences. The stories and examples were instructive and amusing (lot's of "war stories"). She presented with general guidelines on proper sharding, but, as the title suggests, presented no silver bullet. Actually, she presented with counter cases where a non-desirable sharding solution for one company, may yet be viable for another.

• openark-kit: MySQL utilities for everyday use / myself

I went to see my own talk that day . I was honored to have good audience and I'm thankful for their attention. I gained good participation, and a couple feature requests/bug reports.

The people

To begin with, this conference was a big success for me, in many respects. The sessions were great (more on that later), but of course, meeting with new people and with familiar people, was the more important part.

I live in Israel, which makes travel to the US very long and expensive. Apparently not many MySQL community members in my neighborhood, so I don't ever get to meet the faces. The conference makes that possible. I did not participate in all community events, as I had scheduled calls with little girls who miss their father. And I was very much under jet lag. And I have more excuses on demand.

But I did get to meet known faces; people I only knew by name; unfamiliar people who were familiar with my work (fun!); and otherwise just (ex-)strangers.

The sessions

There was a variety of sessions to choose from. Many times, I had to pick one out of two or three sessions I was interested in, running at the same time. Not all sessions appeal to one in the same way, but looking back, I find there were a lot of GOOD sessions I attended. I mostly like sessions that are very technical; preferably drilling into details of algorithms & implementation.

I wish to review some of the sessions I attended. I can't possibly review them all, so my apologies to those I left out.

Monday

• Learn how to cure MySQL replication deprivation with Tungsten! / Robert Hodges (Continuent.com), Edward Archibald (Continuent)

This is actually the first time I learned about the internal details of Tungsten. Robert, CEO of Continuent & original developer of Tungsten, did an excellent job in reviewing the various aspects of Tungsten. But rather than being a promotional talk, this was a purely technical tutorial.

So it was interesting to understand the difficulties in building Tungsten, at least on the MySQL part. For example, the Tungsten replicator must do its own filtering of events based on server id, since, although a replicating slave knows how to ignore circular queries issues by itself, there is no client API to do so; you can't "send over" the server id along with a query and expect the server to ignore it.

As seems to be the current and emerging paradigm, parallel replication is possible by paralleling queries from different schemata. But apparently Continuent seem to want to change that. That is, there's nothing that strictly forces that this must be the paradigm; it's just the current implementation.

Giuseppe was in charge of live demos, and has shown stuff like slave disconnect, fail over, multi-master replication etc. Edward Archibald later presented a cluster-like solution based on Tungsten and a replication aware connection proxy.

There was plenty of time for asking questions. Interesting technical issues were discussed and answered. It was also stated very clearly what Tungsten does not do for you, or when it may actually be the gun with which you shoot yourself in the foot. Good talk!

Tuesday

• MQL-to-SQL: a JSON-based Puery Language for RDBMS Access from AJAX Applications / Roland Bouman (XCDSQL Solutions / Strukton Rail)

Roland is now passionate about finding an easier query language for web applications to use. Rather than SQL, he suggests using MQL (created by Freebase). MQL uses JSON (-like) syntax to describe both queries (requests) and result sets (responses). I won't go into the details since Roland himself laid them out on his blog.

As usual with Roland, his session is well thought of. He describes an elegant, appealing syntax. In my opinion, MQL is an intuitive language. It is easily comprehensible by both DBAs and developers. Possibly by my mother.

However Roland goes further and offers his MQL-to-SQL, an open source implementation of a translation tool between the two languages. In his presentation, he demonstrated some simple and some not-so-simple queries, generated by MSQL, translated to SQL, the results of which translated back to MQL.

MQL is not an all-purpose solution. You don't do analytics with MQL, but it does seem to answer some problems that other ORMs (e.g. Django) are, to the best of my knowledge, unable to perform in convenient manner (that is, without reverting to some SQL-like syntax).

Special thanks

Thanks to Giuseppe Maxia, when on MySQL Replication Advanced Techniques he presented some of the openark kit tools, and, having located me in the crowd, invited me to speak.

To be continued

I need to prepare for my flight. Will probably publish next part(s) once back home. Hope to see you all next year!

The slides are uploaded to the O'Reilly site, and I'm attaching them here as well. Feel free to download the PDF: openark-kit-mysqlconf11.pdf

I wish to thank all who attended my talk!

However I've got something up my sleeve: a benefit many can enjoy, that'll make me a respectful, popular and sought after speaker. While others may try and lure you with such earthly temptations as a 20% off discount, I am in a position to offer you a more spiritual gift: my friendship!

See, if you become my friend, I can offer you a 25% discount on the MySQL conference. Yes, that's 5% more than my competitors! The only thing I ask in return is that you be my friend (hey, it's called "friends of speaker"). Not like a FB virtual friend, but a real friendship! One where you can buy me beer or dinner!

If you agree to such humane terms, I will be in the position to let you know that all you have to do is fill in mys11fsd in your registration form.

No, wait! I let it slip! Rewrite: You should fill in mys11fsd [will only tell you this password after your commitment to a beer] in your registration form.

Oh no, not again!

Don't use mys11fsd without talking to me first... You're not supposed to... Oh, my beer!

Argghhh!

I will present a 45 minute session titled openark-kit: MySQL utilities for everyday use.

In this session, I will present some of the tools in the openark kit. We'll discuss some limitations of the MySQL server, and how openark kit tools overcome those limitations and provide with solutions to common maintenance and audit problems.

This will be a technical session and will discuss various topics of the MySQL server: security, execution plans, replication, triggers and more. I do not intend to discuss all tools, nor to cover the various options. Instead, I'll present the "behind the scenes", show why the tools work, present common problems and typical use case.

This will be the first time I present at the MySQL Conference (or any conference outside Israel, for that matter). I hope to have a good session. As extra measure of safety, I'll bring along a couple basketballs; if the sun shines, we can all go outside and have a good time!

The idea to submit this talk (credit Roland Bouman) has given me the inspiration to put effort in making a new release with new and updated tools. So this talk is already a success as far as I'm concerned.

Hope to see you there!

[PS shameless plug: openark kit.]

I'm under the assumption that no shocking news are delivered. That is, that for the near future, it's business as usual for MySQL.

Last year's great message, delivered by Karen Padir, was "more community". More community participation, more community patches. Looking back, I'm not sure I saw that coming true. The 5.4 version was announced at that same conference, and was criticized for being community-oriented yet community-hidden. The latest 5.5 milestones announcement took everyone by surprise again. Ideas from Google patches were incorporated into 5.5M2. but, to the best of my understanding, no community patch was delievered.

I have both congratulated and expressed my desire that community took greater part in this.

So what am I looking forward to hear?

1. Like everyone else, the general plans Oracle holds for MySQL. Again, I'm not expecting shocking news here.
2. The expected roadmap for MySQL, technically speaking. I don't actually know if there is a roadmap right now.
3. The intended role for the MySQL community. Frankly, it would be just fine with me if Oracle were to say: "we will not accept community patches", and that would be the end of it. That's fine, because it's their right, and it would be an honest announcement. Naturally, I'll be much happier to hear "we will incorporate the best 20 community patches withing the next three days". Somewhere in between, I'll be really satisfied with a clear explanation of how Oracle sees the community, and how it would like to cooperate with it. Will it share the development plan with the community? Will it allow the community to have a say about what goes in or not?

I realize this must all be very pressing, what with the acquisition; big new company; new rules; new bosses; new things to learn. But I do believe it's in Oracle's best interest (and obligation) to speak up their mind on MySQL in relation to the community.

The situation where the users (not the community) don't know what's planned for MySQL on the technical level, what's the next milestone, when the 5.4/5.5 versions are scheduled to be released, what will happen with all those features which were supposed to be incorporated into 5.2/6.0 -- had better be done with quickly.

My congratulations to the MySQL team on the finalization of the acquisition, and best wishes to a smooth and successful merge into Oracle.

But what I've just recalled was a discussion (was it previous year, or the one before that?) comparing the "PostgreSQL Conference" and the "MySQL Users Conference", as it was named back then. In that discussion, the PostgreSQL people were bashing MySQL, saying that the "PostgreSQL Conference" was all about the database and whatever was around it, whilst the "MySQL Users Conference" clearly stated that the attendees were "just users", not like real participants or members.

As I recall, it was a heat debate (and I apologize for not linking). I thought it was just semantics.

Anyway, here we are at 2009, there is no "Users", it's just like in PostgreSQL, and I still think it's just semantics.

I think a name can mean whatever you want it to mean. It's best to judge by actions, which is an altogether different thing.

I do not hold a position this way or the other.

This concludes my postings with regard to the MySQL (Users?) Conference & Expo 2009, which were non-technical. I'll keep to technical stuff in the future, I promise. I enjoyed the conference very much, a big applaude to the presenters and the organizers, and to the many friendly people who attended!