Want to host the MySQL mirror/archive?

For a few years now Proven Scaling has been hosting the MySQL mirror/archive at mirror.provenscaling.com, which provides mainly older MySQL releases (which have been removed from MySQL’s mirror long ago) and MySQL Enterprise releases for the last few years.

Since Proven Scaling has been winding down (and we’ve started paying the bills personally), it doesn’t really make sense for us to maintain as large of a colocation footprint as we have. We’re looking to shift things around, and the mirror is something that’s fairly painful for us to host with a small footprint. We’re hoping another MySQL community member with an already-large datacenter/bandwidth footprint will want to pick it up, where it won’t affect their bottom line so much.

I think there’s still some value in having these files be available, so I’m reluctant to just shut it down (but if no one is interested, that may happen).

Some basic stats about the mirror (for July, 2010):

  • Data size: ~305G
  • Hits: 232,242
  • Transfer/mo: ~703G (this is pretty variable, and was ~1.23TB in May)
  • Transfer/day: ~22G avg, ~71G max

I’d be happy to provide file listings and Webalizer stats to interested parties. I will also note that a lot of the data transfer seems to be junk traffic from Asia, although I’m not sure what can be done to minimize this. Ideally we would arrange to rsync the full data set to a server somewhere (or provide ssh access for you to do the same), and you’d maintain the mirror as necessary going forward. We’re happy also to point mirror.provenscaling.com at the new home.

If you’re interested, email me at jeremy@jcole.us.

Now available: Proven Scaling MySQL yum repository

Yum is an extremely popular system to download, install, and update RPM-based packages from multiple repositories. Proven Scaling has launched a set of repositories to augment the existing central distributions’ repositories with packages our customers need for deploying MySQL-based systems. We’ve been working on it for a while, and have had many people making use of it. We are providing:

  • RPMs of community and enterprise releases of MySQL for RHEL/CentOS, as built by MySQL and distributed on MySQL.com
  • RPMs of community tools such as maatkit and innotop and their dependencies.
  • Proven Scaling-created tools such as mysql_snapshot (an LVM snapshot-based backup utility).
  • Difficult to find RPMs of Perl libraries (dependencies for other scripts, such as innotop).

Here are the yum repositories we are providing:

To install these repositories, grab the .repo file and place it in /etc/yum.repos.d/. You should then be able to install packages using e.g. yum install maatkit. Here are the .repo files:

We hope you like them and find them useful! Let me know if there are any additional packages you think we should add.

Proven Scaling goes global

A bit of exciting news… Proven Scaling has officially gone global with the addition of a new MySQL Geek, Mike Griffiths of London, England to our team. We are now capable of easily and efficiently (timezone-wise and travel-wise) handling your on-site MySQL consulting needs in the London area, the UK, and Europe at large. (As well as making remote work in the middle of the night for our US customers a fair bit easier.)

Mike comes to us from Yahoo! Europe where he worked for a number of years specializing in MySQL operations, performance and optimization, replication and high availability, and scalability. He has been a personal friend for several years and I have been looking forward to him joining the Proven Scaling team since we started the company.

If you’re looking for MySQL consulting in Europe (or anywhere, for that matter), let us know!

On Hiring a MySQL DBA/Architect

These days everyone is looking for a MySQL DBA or MySQL Architect. I am regularly contacted by recruiters, Proven Scaling customers, and other contacts, and they all have the same question: “Where do we find MySQL people to hire?” Most of them have had requisitions open for 6+ months (I know of a few in the 12+ month range), they haven’t found anyone, and they’re feeling desperate now. Since I get this question so often, I thought I’d consolidate my advice on the subject and post it.

They don’t exist on the market today.

Currently there are many more job openings for MySQL people than there are qualified people to fill them. Many of you reading this and trying to hire someone are working for startups and are probably relatively “unknown”, perhaps you don’t have a lot to offer. This makes it even harder for you, as you must compete with the likes of Google, Facebook, and even MySQL itself. As soon as a qualified person starts looking, they are snatched up by someone. It is very unlikely that you will just happen upon a MySQL Architect with 5+ years experience etc., etc., that is on the market. Stop dreaming.

What can you do about it?

The lack of available qualified people to hire doesn’t mean you don’t have MySQL problems that need solving. As far as I’m concerned there are a few possible solutions:

  • Use consultants — Many times you can get by in the short term by using consultants to do some DBA-like tasks, and especially architect tasks. A consultant may also be able to help answer questions that a DBA would normally answer for your developers. Obviously this is somewhat self-serving, since this is the business I’m in.
  • Internal transfer — Transfer someone internally to fill the position, and train them into it. This is often the best option, if you have a large enough team. If you’re a small startup, though, you probably don’t have enough staff to make this work.
  • Hire a non-MySQL DBA — Hire someone who has a solid background in databases, but may not be a MySQL expert, and train them up on MySQL.
  • Hire a MySQL non-expert — Hire someone who is technically strong, knows some MySQL, but isn’t the expert you’re looking for, and train them into the position.

If you’re hiring someone new or transferring someone internally, you may want to consider enlisting some outside help in interviewing them to make sure they are a good fit for the position and have fairly high confidence that they will be capable of growing into the position. Proven Scaling offers interview assistance for exactly this purpose.

Okay, we’ve got someone, what now?

After you’ve hired someone from one of the above suggestions, you’ve got a warm body in a seat, but they are not a MySQL expert, so you’ll need to immediately get started on training them into the position. Here’s the basic general training plan I would suggest:

  • Books — Buy them all the books they could possibly want. I would suggest, at a minimum (depending on what you’re asking of them): MySQL, Pro MySQL, High Performance MySQL1, Understanding MySQL Internals, and Understanding the Linux Kernel. None of them are really meant to be read cover-to-cover, but they are good for understanding specific problems.
  • Training — Probably the best way to get them up to speed on a broad range of topics, would be to send them to MySQL’s formal training classes. I would recommend at least: MySQL for DBAs and MySQL 5.0 Performance Tuning. In addition, Proven Scaling can offer customized and specific training classes on certain topics, such as replication, partitioning, and scalability.
  • Consulting and/or Support — Hands-on work with a consultant is a great way to get specific questions answered and address any doubts or fears on an ongoing basis. Using a consultant for hand-holding during any potentially dangerous operations, migrations, installations, etc., is also a good way to ensure that nothing goes terribly wrong. My company, Proven Scaling, does this as well as Percona, and MySQL itself. You may also want to consider an ongoing support relationship with one of those companies as well.
  • Conferences — You shouldn’t hire for a MySQL position without planning on sending them to the MySQL Conference and Expo every year.
  • Networking — Send them to MySQL Meetups, user groups, networking events, etc. to learn from others and perhaps most importantly, learn what they are missing.
  • Give them time — It will take some time for them to get up to speed and feel comfortable in their new position. Give them plenty of time and space to learn what they need to learn. This is especially difficult with internal transfers, as they may be trying to train their replacement in their old job.

Doesn’t sound good to you? Dead-set on finding an expert?

If you’re dead-set on finding and hiring a MySQL expert, and you’re not willing to follow one of the alternate approaches I’ve suggested above, here are some tips:

  • Don’t be anonymous — People interested in and qualified for a MySQL DBA or MySQL Architect job are in the position to choose which employer they want to work for. If they don’t know who you are because you’ve posted your ad as “a hot new startup”, they will skip over you.
  • Don’t waste their time — Show them they are loved, don’t waste their time with too much unnecessary back and forth. Google their name, find them on LinkedIn, do your own research on their background, and contact them only once you’re sure that not only does their experience meet your needs, but that the job you’re asking them about has a chance of being interesting to them. No email interviews; they make you look silly. Keep the stupid questions to a minimum. If they’re from out of town, and you want an on-site interview, pay for travel upfront, and put them in a nice hotel. Engineers are inherently lazy, and reimbursement sucks.
  • Have perks — Free drinks and snacks, commute assistance, relocation, bonuses, top of the line hardware, decent office space, bike parking, showers, decent car parking are all standard perks. Make your company appealing to prospective employees.
  • Pay well — If you’re not willing to pay well, forget it. Make sure your pay scale matches what you’re asking of them. Want 24/7 pager duty? It will cost you.
  • Pay referrals well — The market for referrals is steep as well, and an external referral is going for anywhere from $5k-$10k today. Advertise prominently whatever you’re willing to pay for a referral. Pay on hire, no strings attached. Consider a referral gift on interview (iPod or similar value) regardless of hire. This ensures that “the network” remembers you’re hiring when they run into someone that’s looking.

All of the above advice works when hiring anyone, but it’s especially important when trying to hire for a position where you, as the employer, are at a disadvantage.

Good luck!

I hope this advice has been helpful. Have any more tips, advice, comments? Think I’m wrong? Please leave a comment!

1 High Performance MySQL is somewhat outdated at this point, but a lot of the advice in it is still valid. Take its advice on 5.0 with a grain of salt. I am eagerly awaiting the 2nd Edition. :)

Going to MySQL Camp II, Brooklyn, NY

In case you live in the dark ages (that is, before RSS) and haven’t heard, MySQL Camp II is next week at Polytechnic University in Brooklyn, NY. Sign up and head over there, slackers!

I will be there to talk about Proven Scaling, HiveDB, DorsalSource, and much more! Send me a note if you’d like to meet up or talk about something specific. I will also have ample Proven Scaling bottle openers (photo thanks to Colin Charles) to be distributed!

On serving two markets and mistakes

Zack Urlocker wrote an article today on InfoWorld titled Serving Two Markets where he comments on Matt Asay’s The open-source community’s double standard on MySQL (which is a piece of work itself) and says:

Part of the issue is that often discussions about the business of open source is seen as a “zero sum game” between community users and paying customers, meaning that in order for one group to benefit, the other group must lose. To me this polarizes the discussion in an unhealthy way.

I have to admit, I haven’t seen it that way at all. And I don’t see why anyone would. When RedHat split into Fedora and RHEL, I evaluated the design of the split and said “OK”. It made sense to me. These days, I use Fedora quite a bit for personal projects, desktop machines, toying around, and testing the newest things. On the business side, I often recommend RHEL1 to customers because it “just works” and has proven itself to be quite stable. RedHat produced not one, but two products I found to be useful for different purposes.

With RedHat, there was absolutely some discontent initially—largely from the casual users who didn’t understand the RHEL issues anyway, whining because something was being “taken away” from them. Nonetheless, the split has been by all accounts highly successful. Nothing was in fact taken away from the community, actually I think Fedora is much stronger and more promising now than RedHat ever was. At the same time, they’ve also managed to appease the commercial side of things with RHEL being a fantastic and very stable server OS.

When MySQL originally discussed the split into Community and Enterprise with me, I told them all of my concerns, which apparently were in large part identical to the concerns of several other key MySQL players that they asked. Nonetheless, they went ahead with things exactly as they had discussed, with none of our concerns addressed or even acknowledged. One of the key concerns we all had was actually not that they were taking anything away from Community. Rather it was that the release structure between Community and Enterprise made no sense for Enterprise. That is, we were (and still are) concerned that MySQL is spinning its wheels and not creating a useful product we would buy or recommend our customers to buy. And I, at least, told them as much.

Jump forward to the most recent announcements. Once again we got an early look at the changes, and once again, we voiced our concerns. This time it basically amounted to “Is taking away the Enterprise source supposed to convince people to buy Enterprise?” Their answer was “Yep”. Our only response could be “Uh, good luck with that.” Once again, our concerns mostly centered around whether the Enterprise product made sense, and once again we said that it didn’t. We told them flat out that a single person mirroring the code would nullify all the “force people to buy” effects of their removal of the source, while nullifying none of the good will they lose by hiding it.

The issues around Community this time around were basically moot because nothing had really changed. We get a similar number of actual Community builds as before, and new stuff gets pushed into a far future version, when basically nothing new was being accepted anyway.

I find it ironic that Zack ends his article with:

At this stage, I think we’re all exploring different approaches to building open source businesses and communities. But the good news is, if we make mistakes along the way, folks will tell us.

Yes, we’ll certainly tell you when you make mistakes. That doesn’t mean you’ll listen or try to correct them. Horses and water and all that.

1 More accurately, I generally recommend RHEL if you’re interested in the support part of the equation, and CentOS if you’re not. Quite a lot of our customers choose RHEL, if for nothing else than to give back to RedHat for creating a great product. Let’s not talk about up2date for now. ;)