Jeremy or Adrienne 2.0

Well, some of you already know—but many of you don’t. This is the official announcement, then… Adrienne and I are having a baby! It is due around mid-March. Exciting stuff! Here’s what the baby looked like as of a few days ago:

On 1U cases, PCI risers, and LSI MegaRAID

I’ve been working with one Proven Scaling customer that has had some interesting issues recently, involving InnoDB corruption, resulting in messages similar to these:

InnoDB: Page checksum 3156980109, prior-to-4.0.14-form checksum 577557610
InnoDB: stored checksum 741279449, prior-to-4.0.14-form stored checksum 577557610
InnoDB: Page lsn 0 2323869442, low 4 bytes of lsn at page end 2323869442
InnoDB: Page number (if stored to page already) 195716,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 0 2831
InnoDB: (index PRIMARY of table db/table)
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 195716.

The problem was encountered when testing hardware for a move from software RAID to hardware RAID using LSI MegaRAID SCSI 320-2 cards. The servers are 1U machines with Tyan motherboards, and a PCI riser card which the MegaRAID plugged into.

They were receiving the same messages on several different machines, ruling out a single bad piece of hardware. After spending weeks trying to figure out what the problem could be, testing different configurations and isolating variables, it was tracked down to the PCI riser cards. Searching for “lsi pci riser” shows quite a few people having similar issues.

It turns out that LSI “does not support” using their cards with PCI risers, at all. Maybe they should reword things a bit—if their cards don’t work with PCI risers.

The scariest part of the whole exercise, though, is that the corruption was occurring completely silently: data comes in, is written to disk, but gets corrupted in flight. Since the OS wrote certain data it is now caching the correct copy of the data, but the disks contain something different. The only way the corruption is discovered is when the page is read back quite a bit later, after having been flushed from cache.

You’d think that somewhere along the line, the OS or the RAID card would catch the corruption?

Linksys NSLU2, Unslugged

I bought a Linksys NSLU2 on Amazon, affectionately known as a “slug”. It’s a tiny little device, barely bigger than my hand, which is intended to be used as a network storage device, with the addition of a couple of USB 2.0 hard drives. Of course, I rarely use such gadgets for what they’re intended.

The reality is that it’s a tiny little box, which without modification1 has:

  • 266 Mhz X-Scale (ARM) IXP425 processor, underclocked to 133 Mhz
  • 32MB of RAM
  • 8MB of on-board flash
  • 2 USB 2.0 ports
  • 1 100Mbit Ethernet port

Most of Linksys’s consumer electronics run Linux, and the NSLU2 is no exception. In practice, that means that the Linux hacker community has adopted the NSLU2 and created many Linux distributions for it. You can easily add a USB keyfob, such as my 512MB SanDisk Cruzer Micro (link actually to a 1GB version), which gives you plenty of space for additional software. I’m currently using Unslung, as it is based on the original firmware from Linksys, and promises to be the least risky. Once I’m comfortable with the box, I will probably change to a pure Linux, minus the Linksys junk that I don’t need.

Last night, I got the box up and running on the Unslung firmware, got a whole bunch of software installed on it, and even got OWFS compiled on it (not including the FUSE filesystem stuff). Man, I forgot how slow things compile at 133Mhz with 32MB of RAM.

I plan to make a few hardware hacks on my slug to at least up the speed to 266Mhz and make it turn itself on when power is applied.2 I have some plans for using mine for running a weather station and webcam, which will (hopefully) make it to the top of a mountain around here. For the short term, though, it will go in our backyard for testing purposes (and well within range of our wireless network without special antennas).

1 With a few hardware hacks, you can add three additional USB ports, a serial port, and increase the clock speed to 266Mhz. Pretty awesome box. Did I mention it uses only 500mA with no disks attached?

2 I will never figure out why companies choose to make their products have stupid defaults like this. You might remember my post about “The Big Green Button“: my previous cable modem had the same stupid problem. Small consumer electronics that are largely designed to be “on” all the time should turn “on” by default, if they even have a power button at all. Sigh.

Leaving Yahoo!; storage, consulting

As many of you know already, I’ve decided to leave Yahoo!. Today was my last day. I’ve had a great time at Yahoo! and worked with a bunch of brilliant people. I will miss you all!

   

Where am I headed? That’s actually a complicated question. I’ll be working part time at Pivot3, a mature startup that’s doing great things in the storage industry. Contact me if you’re interesting in cutting storage costs, getting some amazing management tools, and some awesome performance. I would love to tell you more! MySQL will absolutely rock on this platform.

Yes, I said part time. I have a startup of my own, Proven Scaling, providing advanced MySQL consulting and training services. Looking to get more performance, scalability, or reliability out of MySQL? Need help with replication? Designing a database solution that really scales? Need help interviewing a candidate for a MySQL position? We can help.

Giving more via Kiva

If you are a regular reader, you might remember Helping fellow man… Kiva and my loan to Shawqat Al Kahwaji, a Palestinian electrician in Gaza.

Recently Shawqat has made two repayments on his loan, and he’s well on his way to having it repaid. Awesome!

While poking around on Kiva, I noticed that there are a bunch of new businesses listed and in need, so I chose a few more to lend to. Here’s what I chose:

Please join me in saying good luck to these businesses on the other side of the world!