Want to host the MySQL mirror/archive?

For a few years now Proven Scaling has been hosting the MySQL mirror/archive at mirror.provenscaling.com, which provides mainly older MySQL releases (which have been removed from MySQL’s mirror long ago) and MySQL Enterprise releases for the last few years.

Since Proven Scaling has been winding down (and we’ve started paying the bills personally), it doesn’t really make sense for us to maintain as large of a colocation footprint as we have. We’re looking to shift things around, and the mirror is something that’s fairly painful for us to host with a small footprint. We’re hoping another MySQL community member with an already-large datacenter/bandwidth footprint will want to pick it up, where it won’t affect their bottom line so much.

I think there’s still some value in having these files be available, so I’m reluctant to just shut it down (but if no one is interested, that may happen).

Some basic stats about the mirror (for July, 2010):

  • Data size: ~305G
  • Hits: 232,242
  • Transfer/mo: ~703G (this is pretty variable, and was ~1.23TB in May)
  • Transfer/day: ~22G avg, ~71G max

I’d be happy to provide file listings and Webalizer stats to interested parties. I will also note that a lot of the data transfer seems to be junk traffic from Asia, although I’m not sure what can be done to minimize this. Ideally we would arrange to rsync the full data set to a server somewhere (or provide ssh access for you to do the same), and you’d maintain the mirror as necessary going forward. We’re happy also to point mirror.provenscaling.com at the new home.

If you’re interested, email me at jeremy@jcole.us.

Now available: Proven Scaling MySQL yum repository

Yum is an extremely popular system to download, install, and update RPM-based packages from multiple repositories. Proven Scaling has launched a set of repositories to augment the existing central distributions’ repositories with packages our customers need for deploying MySQL-based systems. We’ve been working on it for a while, and have had many people making use of it. We are providing:

  • RPMs of community and enterprise releases of MySQL for RHEL/CentOS, as built by MySQL and distributed on MySQL.com
  • RPMs of community tools such as maatkit and innotop and their dependencies.
  • Proven Scaling-created tools such as mysql_snapshot (an LVM snapshot-based backup utility).
  • Difficult to find RPMs of Perl libraries (dependencies for other scripts, such as innotop).

Here are the yum repositories we are providing:

To install these repositories, grab the .repo file and place it in /etc/yum.repos.d/. You should then be able to install packages using e.g. yum install maatkit. Here are the .repo files:

We hope you like them and find them useful! Let me know if there are any additional packages you think we should add.

Proven Scaling goes global

A bit of exciting news… Proven Scaling has officially gone global with the addition of a new MySQL Geek, Mike Griffiths of London, England to our team. We are now capable of easily and efficiently (timezone-wise and travel-wise) handling your on-site MySQL consulting needs in the London area, the UK, and Europe at large. (As well as making remote work in the middle of the night for our US customers a fair bit easier.)

Mike comes to us from Yahoo! Europe where he worked for a number of years specializing in MySQL operations, performance and optimization, replication and high availability, and scalability. He has been a personal friend for several years and I have been looking forward to him joining the Proven Scaling team since we started the company.

If you’re looking for MySQL consulting in Europe (or anywhere, for that matter), let us know!

On Hiring a MySQL DBA/Architect

These days everyone is looking for a MySQL DBA or MySQL Architect. I am regularly contacted by recruiters, Proven Scaling customers, and other contacts, and they all have the same question: “Where do we find MySQL people to hire?” Most of them have had requisitions open for 6+ months (I know of a few in the 12+ month range), they haven’t found anyone, and they’re feeling desperate now. Since I get this question so often, I thought I’d consolidate my advice on the subject and post it.

They don’t exist on the market today.

Currently there are many more job openings for MySQL people than there are qualified people to fill them. Many of you reading this and trying to hire someone are working for startups and are probably relatively “unknown”, perhaps you don’t have a lot to offer. This makes it even harder for you, as you must compete with the likes of Google, Facebook, and even MySQL itself. As soon as a qualified person starts looking, they are snatched up by someone. It is very unlikely that you will just happen upon a MySQL Architect with 5+ years experience etc., etc., that is on the market. Stop dreaming.

What can you do about it?

The lack of available qualified people to hire doesn’t mean you don’t have MySQL problems that need solving. As far as I’m concerned there are a few possible solutions:

  • Use consultants — Many times you can get by in the short term by using consultants to do some DBA-like tasks, and especially architect tasks. A consultant may also be able to help answer questions that a DBA would normally answer for your developers. Obviously this is somewhat self-serving, since this is the business I’m in.
  • Internal transfer — Transfer someone internally to fill the position, and train them into it. This is often the best option, if you have a large enough team. If you’re a small startup, though, you probably don’t have enough staff to make this work.
  • Hire a non-MySQL DBA — Hire someone who has a solid background in databases, but may not be a MySQL expert, and train them up on MySQL.
  • Hire a MySQL non-expert — Hire someone who is technically strong, knows some MySQL, but isn’t the expert you’re looking for, and train them into the position.

If you’re hiring someone new or transferring someone internally, you may want to consider enlisting some outside help in interviewing them to make sure they are a good fit for the position and have fairly high confidence that they will be capable of growing into the position. Proven Scaling offers interview assistance for exactly this purpose.

Okay, we’ve got someone, what now?

After you’ve hired someone from one of the above suggestions, you’ve got a warm body in a seat, but they are not a MySQL expert, so you’ll need to immediately get started on training them into the position. Here’s the basic general training plan I would suggest:

  • Books — Buy them all the books they could possibly want. I would suggest, at a minimum (depending on what you’re asking of them): MySQL, Pro MySQL, High Performance MySQL1, Understanding MySQL Internals, and Understanding the Linux Kernel. None of them are really meant to be read cover-to-cover, but they are good for understanding specific problems.
  • Training — Probably the best way to get them up to speed on a broad range of topics, would be to send them to MySQL’s formal training classes. I would recommend at least: MySQL for DBAs and MySQL 5.0 Performance Tuning. In addition, Proven Scaling can offer customized and specific training classes on certain topics, such as replication, partitioning, and scalability.
  • Consulting and/or Support — Hands-on work with a consultant is a great way to get specific questions answered and address any doubts or fears on an ongoing basis. Using a consultant for hand-holding during any potentially dangerous operations, migrations, installations, etc., is also a good way to ensure that nothing goes terribly wrong. My company, Proven Scaling, does this as well as Percona, and MySQL itself. You may also want to consider an ongoing support relationship with one of those companies as well.
  • Conferences — You shouldn’t hire for a MySQL position without planning on sending them to the MySQL Conference and Expo every year.
  • Networking — Send them to MySQL Meetups, user groups, networking events, etc. to learn from others and perhaps most importantly, learn what they are missing.
  • Give them time — It will take some time for them to get up to speed and feel comfortable in their new position. Give them plenty of time and space to learn what they need to learn. This is especially difficult with internal transfers, as they may be trying to train their replacement in their old job.

Doesn’t sound good to you? Dead-set on finding an expert?

If you’re dead-set on finding and hiring a MySQL expert, and you’re not willing to follow one of the alternate approaches I’ve suggested above, here are some tips:

  • Don’t be anonymous — People interested in and qualified for a MySQL DBA or MySQL Architect job are in the position to choose which employer they want to work for. If they don’t know who you are because you’ve posted your ad as “a hot new startup”, they will skip over you.
  • Don’t waste their time — Show them they are loved, don’t waste their time with too much unnecessary back and forth. Google their name, find them on LinkedIn, do your own research on their background, and contact them only once you’re sure that not only does their experience meet your needs, but that the job you’re asking them about has a chance of being interesting to them. No email interviews; they make you look silly. Keep the stupid questions to a minimum. If they’re from out of town, and you want an on-site interview, pay for travel upfront, and put them in a nice hotel. Engineers are inherently lazy, and reimbursement sucks.
  • Have perks — Free drinks and snacks, commute assistance, relocation, bonuses, top of the line hardware, decent office space, bike parking, showers, decent car parking are all standard perks. Make your company appealing to prospective employees.
  • Pay well — If you’re not willing to pay well, forget it. Make sure your pay scale matches what you’re asking of them. Want 24/7 pager duty? It will cost you.
  • Pay referrals well — The market for referrals is steep as well, and an external referral is going for anywhere from $5k-$10k today. Advertise prominently whatever you’re willing to pay for a referral. Pay on hire, no strings attached. Consider a referral gift on interview (iPod or similar value) regardless of hire. This ensures that “the network” remembers you’re hiring when they run into someone that’s looking.

All of the above advice works when hiring anyone, but it’s especially important when trying to hire for a position where you, as the employer, are at a disadvantage.

Good luck!

I hope this advice has been helpful. Have any more tips, advice, comments? Think I’m wrong? Please leave a comment!

1 High Performance MySQL is somewhat outdated at this point, but a lot of the advice in it is still valid. Take its advice on 5.0 with a grain of salt. I am eagerly awaiting the 2nd Edition. :)