MySQL Community Contributor of the Year 2013

First of all, thank you to everyone who nominated me, voted for me, and to those of you who shared kind words with me and congratulated me. It’s humbling to have been awarded one of the “MySQL Community Contributor of the Year” awards for 2013. Many people have asked or wondered without asking why I do what I do, and how I got here. Given the occasion, I thought I would share some thoughts on that.

Early days as a user in web development

I started working with MySQL (and before that, mSQL) back in 1998 while working with a web development company. MySQL worked quite well (and I pretty quickly forgot about mSQL), and I started to learn more and more about it. As many new users at the time, I hit a few bugs or quirks, and poked at the code from time to time to understand what was going on. I continued just being a user of MySQL into 1999, and started to build more and more complex applications.

A first bug report

In October 1999, I encountered a crashing bug (a floating point exception which wasn’t caught) with SELECT FLOOR(POW(2,63)) on MySQL 3.22 and FreeBSD 3.3, and I made my first MySQL bug report by emailing the mailing list. After a short discussion with Sinisa Milivojevic and Monty Widenius, Monty agreed to fix the bug. Of course I watched with bright eyes, I read the code for the fix, and I worked to understand it.

The mailing list and IRC drew me in

I was hooked. I found an actual problem, as a 17 year old hacker sitting in Kansas, and I worked with these nice folks who I’ve never met, halfway around the world in Cyprus and Finland, and they agreed to do work for me to fix it, and they didn’t even complain. They were genuinely happy to help me.

I joined the mailing list to report that bug, but I stayed subscribed to it and read every mail. I browsed the archives and learned how the (tiny) community worked at the time. I joined the #mysql IRC channel on EFnet and started listening there as well.

Helping out on the mailing list and on IRC

While lurking on the mailing list and on IRC, I quickly realized that there were a lot of people with problems and questions that I could help out with. I knew some of the answers! I answered things where I knew the answer, and I worked to answer questions that I didn’t know. Through experimentation and reading the MySQL documentation and source code to solve other people’s problems, I learned an amazing amount.

Improving the documentation

In the process of doing web development work, and of helping out answering other people’s questions, I found that the MySQL manual was moderately technically complete, but very messy, sometimes buggy, and strangely worded. I poked around until I could figure out how the manual itself worked. I learned about this weird Texinfo format it was written in. Once I got things to build, I undertook an initial editing of the MySQL manual by reading through the entire Texinfo file, fixing typos and rewording things. I checked examples in the manual against an actual server and cleaned up broken examples and incomplete documentation.

Hey, this is more fun than my real job

I was then working at a web development company in Nashville, and realized that I wasn’t very happy doing that work. At the same time, the company started to melt down, and I began interviewing elsewhere. I spent more and more time doing work on MySQL (sometimes instead of work I should’ve been doing). Contributing to MySQL and working with the MySQL community made me much happier than any other work I had done so far.

Monty??? Hire me!

I don’t actually remember how I initially contacted Monty about this (although he probably still has the email archives), but he and I exchanged emails. He offered that I should come to an upcoming developer meeting in Monterey, California in July of 2000, coinciding with OSCON 2000. I jumped at the chance. I mentioned the invitation to Indrek Siitan on IRC, and he invited me to join a planned road trip to Monterey with some of the earliest MySQL employees: himself, Matt Wagner, and Tim Smith.

No interstates, no hotels, nothing but love

Although I wasn’t an employee yet, and had never met any of them in person, Matt Wagner drove from Minnesota and Tim Smith drove from North Carolina to my house in Tennessee. We piled in Matt’s pickup truck and drove from there down to Louisiana to pick up Indrek. The four of us drove in two cars from New Orleans to Monterey for about 10 days, with a plan to use no interstates—only highways—and camp each night.

I was an almost completely broke and unemployed kid, and they paid for almost everything and took me along—as a friend—across the entire country. I got to know my first few MySQL employees through those many hours in the car talking about life, technology, MySQL, and anything that came up. We had a lot of fun and they showed literally nothing but love. We all became fast friends and they accepted me without hesitation. This became my canonical example of the MySQL community, and still is, even to this day.

Meeting the team

We arrived in Monterey and I (a random non-employee) got to sit in all of the internal company meetings and technical discussions. I got to have a say in how MySQL was being made, and I got to argue with the very creators of MySQL. They not only listened, but respected me and valued my opinion. I mostly just listened through these meetings and got to know everyone, but this was an amazing experience.

At some point later in the meeting, Monty and I met, and he offered me a job at MySQL. I accepted it without hesitation and jumped into my official MySQL career head-first. My first paycheck was wired directly from Monty’s personal bank account in Finland, because there was some trouble setting up payroll for me, and Monty was concerned about making sure I got paid quickly.

Documenting MySQL, and a foray into Swedish and Swenglish

My first tasks were all about making the MySQL documentation better. I made several complete passes through the manual, reading and correcting it. I did some fairly major restructuring of the order of the sections, and normalized the structure as much as possible. (I also got quite good at reading Texinfo documents unformatted and visualizing the formatting.)

I started studying Swedish in order to understand all of the source code comments, variable and function names, and the Swedish error messages. I translated many of these remnants of Swedish and Swenglish as some of my first contributions to the actual codebase, and I did a lot of other easy formatting and fixing work while learning how the code worked. I figured out where all the functions and syntax were defined in order to make sure all elements of the syntax were documented.

A new life as a MySQLer

While at MySQL, I initially worked on documentation and helped out with support, and when customers needed help in person, I flew around and consulted with them. Kaj Arnö’s company Polycon’s training group was acquired by MySQL, and I started helping out with that training. They needed someone to teach training classes, so I started doing that too, eventually managing the whole group.

Ever present in the MySQL world

Since then I have had the opportunity to be a part of a lot of amazing things, and have made sure that every new opportunity and every new job undertaken gives me ample opportunity and motivation to continue being part of the MySQL community. Why? It’s just a part of who I am. I have some gifts for communication, making dense material understandable, understanding the needs of database users, and building scalable and manageable database systems. I want to share with others and give back to the community to give them the same or better opportunities as I was given.

Thanks to you all

Where I am in the MySQL community, and where I am in my life and career would not be possible without amazing examples given to me by a bunch of amazing people. There’s not any one mentor who was my sole example, but rather a community of dozens of individuals, each of whom I admire and have aspired to learn various things from. I’d like to offer special thanks and acknowledgement to the following folks though:

  • Monty Widenius — Of course, Monty was the father of it all, but he has also acted as a father to me personally, taken care of me, and invited me into his home and his family. He has a huge heart and is both a personal and technical mentor to me.
  • Matt Wagner, Indrek Siitan, and Tim Smith — Matt, Indrek, and Tim offered a great example of how a team can be a family, and welcomed me into the community, into their lives, and into the company in an amazing fashion. In addition, they were also great technical mentors and taught me a lot about MySQL.
  • Sinisa Milivojevic, Sasha Pachev, Jani Tolonen, Miguel Solórzano, Tõnu Samuel, Sergei Golubchik, Paul DuBois, Kaj Arnö, Arjen Lentz, Mårten Mickos, Carsten Pedersen, Zak Greant, David Axmark, Brian Aker — These folks are a mix of developers, executives, peers and community, of all backgrounds and experiences. One thing they all have in common is that they helped me to learn what it takes to build software, to run a company, and to be a community. While we haven’t always gotten along or agreed on everything, I have always respected every one of them and keep track of as many of them as I can.
  • Countless others in the community — Others on the mailing lists and IRC, customers, partners, and peers. Thanks for all being here and being awesome!

On the award

In Henrik Ingo’s words:

Several people nominated Jeremy and indeed he has a long history with MySQL, pretty much back to the first release.

For example, people mentioned Jeremy’s insights shared on his blog, on issues such as Linux NUMA memory management. His recent work on innodb_ruby has been widely appreciated both for it’s educational value and perhaps even some potential usefullness.

Most of us will have used the SHOW PROFILE(S) commands created by Jeremy – and for a long time this was the only community contribution that actually made it into MySQL Server!

His consulting company Proven Scaling used to mirror the MySQL Enterprise binaries that were GPL but not otherwise publicly available. This grew into a historical archive of (almost) all MySQL binaries ever released. Related to his issues with the MySQL Enterprise process, and poor handling of community contributions, Proven Scaling was actually the first company to create a community fork of MySQL known as Dorsal Source.

You might also remember in 2008 Jeremy took a public stand against MySQL’s plans to offer new backup functionality only as closed source. This resulted in public outcry on Slashdot and elsewhere, and Sun eventually commanded MySQL executives to give up on those plans.

So any way we look at it, over the years he has really contributed a lot and always had the interests of the MySQL Community close to his heart.

Onwards!

I look forward to continuing to contribute my efforts and my skills to MySQL, and always making my work available to the community. There’s a lot of work left to do, and I hope my efforts in that will be useful to many.

Julian Cash “white background” community photos at Percona Live MySQL Conference

You might have noticed from my profile picture on this blog, as well as on Facebook, Twitter, etc., that I am a fan of Julian Cash‘s photography.

If you’re in the MySQL community, you almost certainly know his photography, both the “white background” style and “light painting” style. Julian took a bunch of photos of the MySQL community at conferences a few years ago, but the community has changed tremendously since then, and it’s time for a whole lot of new ones! I’ve asked Julian to come to the conference and take a bunch more photos of the MySQL community in his iconic “white background” style at the Percona Live MySQL Conference and Expo next week.

In order to be as inclusive as possible, we wanted this to be free for everyone getting their picture taken (come one, come all!) — however, to make this a success…

We need your help to fund the project on Indiegogo!

If you have the means to fund it1, it will certainly help; any amount helps! If you don’t, that’s fine as well, and you can absolutely come get your photo taken regardless. If you’re coming to the conference and want to get your photo taken with Julian, join the MySQL Studio Photos @ Percona Live MySQL event on Facebook so you can get updates about the location, schedule, and any changes.

See you there!

1 If you’re a company and want to do something more exotic than what Indiegogo has listed, feel free to send me an email and I’ll put you in touch with Julian to do that!

Skipping the MySQL Conference and Expo 2009

I have been silent on the topic of this year’s MySQL conference, and really, silent in nearly all ways anyway. Today, as the MySQL Users Conference Conference and Expo 2009 starts up, some people will be wondering where I am, so I ought to at least answer that: I am skipping the conference this year.

As my wife can tell you, this is not a decision I’ve taken lightly, and I’ve gone back and forth on it for months and then down to the final days leading up to now. I’ve decided after much internal and external debate just to skip it all this year, including the side conferences and other stuff. Why? The reasons are basically:

  • It’s my opinion that the “state of the art” with MySQL has not changed, so I am not really missing much technical content to further myself. There’s a lot of interesting stuff going on with Percona and Google’s work, but I largely follow that through blogs and mailing lists and personal contacts already.
  • I strongly disagree with the speaker selection process used in this and the previous couple of years’ conferences. I feel that they actively discourage any speakers who refuse to stay on script—those willing to tell users the reality of the situation. I can’t and/or won’t do that, so I didn’t submit anything this year—my initial effort, months ago, to disengage from the conference. I watched from the sidelines this year and listened to everyone else’s complaints about not being selected. I don’t care to hear about the process, the details, the counter-arguments, etc., but really, any selection process which ends up leaving out Percona, for any reason, is just broken. I know they got some consolation slots in the end, but that doesn’t change my opinion.
  • The previous years’ conferences, while they have been fun in various ways, have been declining in actual technical content and increasing in politics, marketing hype, PR stuff, and whatnot. This does not have value to me (see above).
  • I am tired of explaining MySQL Enterprise vs. MySQL Community, InnoDB/Oracle vs. MySQL Inc., where the patches have gone, why the foreign key implementation in MySQL sucks, why the triggers implementation sucks, the problems with replication, etc., ad infinitum. Nothing has changed, it’s all basically as broken as it was last year, the year before that, etc.

OK, many people know the above about me already. So why not go to the Percona Performance Conference or hang out and chat with people, or…

  • The most important reason: I am making a concerted effort to stay out of the politics of the whole conference this year. While I feel that I add nominally value to the conference by analyzing and reporting on the happenings, and asking uncomfortable questions, I just can’t do it this year. I need a break.

I debated potentially taking this entire week as vacation and really disappearing for the entire week, disconnected, but I really value the friendship of all of my friends that come out to California from the far reaches of the Earth for the conference. I will be in the Bay Area (as usual every week), and I would very much enjoy having a politics-free drama-free dinner if you’re up for it. Contact me!

Administrative Note: I have disabled comments on this entry, as I don’t want the debate and politics to shift over here. I apologize in advance for my anti-blog-like behavior in that regard. I welcome any and all notes and comments at jeremy@jcole.us.

Now available: Proven Scaling MySQL yum repository

Yum is an extremely popular system to download, install, and update RPM-based packages from multiple repositories. Proven Scaling has launched a set of repositories to augment the existing central distributions’ repositories with packages our customers need for deploying MySQL-based systems. We’ve been working on it for a while, and have had many people making use of it. We are providing:

  • RPMs of community and enterprise releases of MySQL for RHEL/CentOS, as built by MySQL and distributed on MySQL.com
  • RPMs of community tools such as maatkit and innotop and their dependencies.
  • Proven Scaling-created tools such as mysql_snapshot (an LVM snapshot-based backup utility).
  • Difficult to find RPMs of Perl libraries (dependencies for other scripts, such as innotop).

Here are the yum repositories we are providing:

To install these repositories, grab the .repo file and place it in /etc/yum.repos.d/. You should then be able to install packages using e.g. yum install maatkit. Here are the .repo files:

We hope you like them and find them useful! Let me know if there are any additional packages you think we should add.

Bravo Oracle: InnoDB Plugin 1.0 released

Yesterday, Oracle‘s Ken Jacobs and Heikki Tuuri, creator of InnoDB, have announced the immediate release of InnoDB Plugin 1.0 for MySQL 5.1. I’ve already downloaded it and played around with it a bit. I haven’t had time to do any performance benchmarks or similar just yet. Those will come in due time.

This release is the beginning of two exciting things: InnoDB is now officially decoupled from MySQL release-wise, and lots of new features have been added to this new release. I will come to what the decoupling means in a moment, but first, the major new features in this release of InnoDB (from my perspective, and with my commentary):

  • Fast in-place index management — The ability to add and drop indexes without rebuilding the entire table in place. This isn’t a complete implementation of the long-awaited “online ALTER TABLE“, as that is mostly a MySQL problem rather than an InnoDB problem.
  • Compression of data and indexes — This should allow data size on disk to be reduced substantially.
  • Storage of entire BLOB, TEXT, and VARCHAR data off-page — This can allow more efficient PRIMARY KEY indexes (where the data is stored in InnoDB because of the index clustering) on tables with BLOB, TEXT, or large VARCHAR columns
  • New information_schema tables — InnoDB is now providing a lot more visibility into what is going on internally. I’m hopeful to extend this even further.
  • InnoDB-specific “strict mode” — Don’t allow InnoDB to fudge things internally, forcing it to error in circumstances it can’t handle, rather than just give a warning or silently continue.

All of the above features look excellent, but one of the more interesting aspects of this announcement is the fact that MySQL and InnoDB are now decoupled for release. That is, Oracle can make a release of InnoDB without having to wait for MySQL to make their own release. While this will make it slightly more difficult to describe what version of MySQL/InnoDB you’re using (especially without a way to find that out from MySQL), it has the potential to make the release process much quicker and more efficient.

I’m quite hopeful, and from what I have seen in the recent past with the collaboration between Heikki, Ken, and Yasufumi (an outside contributor) on fixing InnoDB performance bugs, quite positive, that Oracle will be a lot more accepting of outside patches and code contributions to the InnoDB codebase than MySQL has been recently. I’ve got a lot of ideas for new features targeted at manageability that I’d like to get implemented. I’d love to hear Oracle’s comments on how they will accept patches to InnoDB now, and what they see the community interaction looking like in the future. Ken, any comment?

The only negative aspects I can see with this announcement are:

  • Oracle spent a couple of years working on this in silence, away from the community. Most everyone was surprised by this release, as we haven’t seen anything from Oracle about InnoDB in a long time, and to some extent we were sitting in the corner hopeful that Oracle really doesn’t plan on killing the project. While I understand that perfectly well from a business standpoint, I am hopeful that working for long periods away from the community can be minimized in the future, so that we can all stay more involved.
  • It is now a bit harder to get a new MySQL/InnoDB installation up and running, as the newest InnoDB is not part of the MySQL packages anymore. I’m sure this can be cleaned up with some smart RPM (etc.) packaging.

Overall, I am excited about this announcement, and quite happy that Oracle is making some serious contributions and commitments to maintaining InnoDB. Thanks for all your hard work, Ken and Heikki and the rest of the InnoDB team! Let me know if there’s anything I can do for you!

Just announced: MySQL to launch new features only in MySQL Enterprise

Just announced at the MySQL Partner meeting as part of the MySQL Conference and Expo in Santa Clara, CA:

MySQL will start offering some features (specifically ones related to online backups) only in MySQL Enterprise. This represents a substantive change to their development model — previously they have been developing features in both MySQL Community and MySQL Enterprise. However, with a shift to offering some features only in MySQL Enterprise, this means a shift to development of those features occurring (and thus code being tested) only in MySQL Enterprise.

As I’ve discussed before, the size of the user base for MySQL Enterprise is much smaller than for MySQL Community. That means these critical features will be tested by only a few of their customers. So, in effect, they will be giving their paying customers real, true, untested code. How is this supposed to work? In addition, this means that they are changing their internal development model, splitting the relationship between the two trees, and overall going even further down the path of getting the RHEL/Fedora model backwards.

What do you think about this? Leave a comment, I’m really curious as to everyone’s feeling on this.

UPDATE: Marten Mickos has just acknowledge that I understood the slide quite correctly, and they will indeed develop new features in MySQL Enterprise (in 6.0), without making them available in MySQL Community. Hmm!