I should’ve titled it differently

In reality, the problem posed by uneven allocation across nodes under NUMA is not entirely a swapping problem. I titled the previous post as it was and explained it in the way it was explained largely to address a specific problem seen in the MySQL community. However, the problem described actually has very little to do with swap itself. The problem is really related to Linux’s behavior under memory pressure, and specifically the pressure imposed by running a single NUMA node (and especially node 0) completely out of memory.

When swap is disabled completely, problems are still encountered, usually in the form of extremely slow performance and failed memory allocations.

A more thorough solution

The original post also only addressed only one part of the solution: using interleaved allocation. A complete and reliable solution actually requires three things, as we found when implementing this change for production systems at Twitter:

1. Forcing interleaved allocation with numactl --interleave=all. This is exactly as described previously, and works well.
2. Flushing Linux’s buffer caches just before mysqld startup with sysctl -q -w vm.drop_caches=3. This helps to ensure allocation fairness, even if the daemon is restarted while significant amounts of data are in the operating system buffer cache.
3. Forcing the OS to allocate InnoDB’s buffer pool immediately upon startup, using MAP_POPULATE where supported (Linux 2.6.23+), and falling back to memset otherwise. This forces the NUMA node allocation decisions to be made immediately, while the buffer cache is still clean from the above flush.

These changes are implemented in Twitter MySQL 5.5 as the mysqld_safe options numa-interleave and flush-caches, and mysqld option innodb_buffer_pool_populate, respectively.

The results

On a production machine with 144GB of RAM and a 120GB InnoDB buffer pool, all used memory has been allocated within 152 pages (0.00045%) of perfectly balanced across both NUMA nodes:

N0        :     16870335 ( 64.36 GB)
N1        :     16870183 ( 64.35 GB)
active    :           81 (  0.00 GB)
anon      :     33739094 (128.70 GB)
dirty     :     33739094 (128.70 GB)
mapmax    :          221 (  0.00 GB)
mapped    :         1467 (  0.01 GB)

The buffer pool itself was allocated within 4 pages of balanced (line-wrapped for clarity):

2aaaab2db000 interleave=0-1 anon=33358486 dirty=33358486
  N0=16679245 N1=16679241

Much more importantly, these systems have been extremely stable and have not experienced the “random” stalls under heavy load that we had seen before.

I’ve been a user of Kiva for a more than five years now, and have made 350 loans so far for a total of $9,400 loaned. In the first few years I only sporadically made some loans and let repaid money sit around for a long time. In the past couple of years I’ve been using Kiva more consistently, every month re-investing the full repayment amounts as soon as they come in, and usually adding 4-6 loans ($100-$150) of new money.

As I’ve been doing this I noticed an effect that makes perfect sense but I hadn’t considered before: Since the loans are anywhere from 9-24 months, but the repaid amounts are repaid typically monthly, if the repaid amounts are re-invested immediately, the original loan amounts stack on top of each other, allowing the same money to be invested several times over simultaneously.

Recently I’ve been thinking about actually quantifying that effect and figuring out what impact it could have. Since it’s not a very simple calculation, I put together a spreadsheet to calculate the full picture for me.

Assumptions

The following assumptions are made:

• Amount per loan: $25.00 — This is the standard loan amount on Kiva, so this just assumes you never double up on a single loan (which is not a good idea as it spreads the loss risk poorly).
• Investment per month: 4 loans, or $100 (and reinvest all repayments) — This is approximately what I’ve been doing, although frequently it’s a bit more than 4.
• Average loan duration: 15 months — This is the average loan duration for my loan portfolio, and seems about average for Kiva.
• Loss rate: 2.07% — This is the actual loss rate of my portfolio, which is a bit higher than the average Kiva user at 1.09% because I tend to loan to war-torn and riskier areas.

Results

After 5 years (60 months) of consistent and prompt investment, the results could be:

• Total investment: $6,000 — This is the actual amount you’ve paid out of pocket.
• Total loans made: 1,004 — The number of individuals or groups helped. This is the most amazing thing, watching all of these individuals succeed due to your help.
• Total amount loaned: $25,100 — The amount your $6,000 turns into after re-investment through immediate re-loaning.
• Total amount lost: $519.57 — Due to a combination of loan defaults and currency exchange loss, not all of your money will be returned.
• Total amount returned: $5480.43 — If you stopped making loans after the 60 months and started to withdraw your money from Kiva, at the end of it you’d get this much back (investment minus loss).

Check out the full calculator on Google Docs for all the details and per-month amounts.

Conclusion

The really amazing thing with following this plan is that the Kiva borrowers themselves end up—through prompt repayment of their loans—funding each other. For me, the amount being invested each month is quite modest, and through reinvestment of the repayments, the monthly impact is huge. This month, I received almost $400 in repayments, added an additional $100, and made $500 in new loans.

I have been considering a new way to think about, measure, graph, and monitor replication lag for a while. Previously I’ve always been primarily been using replication delay (lag), in seconds. Initially this came from SHOW SLAVE STATUS‘s Seconds_Behind_Master field, later replaced by mk-heartbeat‘s delay information. (These are equivalent, but mk-heartbeat is less buggy and works to represent true lag with relay slaves in the replication topology.) Using delay in seconds to monitor replication has a few problems though:

• The current number of seconds delayed provides no information about how long it might take to catch up.
• It’s very difficult to determine whether a slave is catching up at all, due to the scale. If a slave is a few thousand seconds behind it’s hard to tell whether it’s catching up, falling behind, or neither, at any particular moment.
• If multiple slaves are graphed together, they may have widely different absolute delay values, and thus scales, and it can be very difficult to compare them, or to see that a slave is having problems, until it’s too late. Two slaves may be falling behind at the same rate, but if one is 300 seconds behind, and one is 20,000 seconds behind, the graphs are difficult to interpret.

Given these problems, I determined that while we need the absolute delay information available, it’s not good enough by itself. I started to think about what it is that I’m really trying to determine when looking at the delay graphs.

Vector: Velocity and direction

The key bits of information missing from the delay information seem to be:

• Is the slave falling behind, catching up, or neither? We need a measure of the direction of replication’s delay.
• How fast is the slave falling behind or catching up? We need a measure of velocity of replication’s performance

Fortunately these two things can be combined together into a single number representing the vector of replication. This can then be presented to the user (likely a DBA) in an easy to consume format. The graphs can be read as follows:

• Y = 0 means the slave is neither catching up nor falling behind. It is replicating in real time. I chose zero for this state in order to make the other two states a bit more meaningful and to make the graph symmetric by default.
• Y > 0 means the slave is catching up at Y seconds per second. There is no maximum rate a slave can catch up, but in practice seeing velocities >1 for extended periods is relatively uncommon in already busy systems.
• Y < 0 means the slave is falling behind at Y seconds per second. As a special case, Y = -1 means that the slave is completely stopped and playing no events. Lagging is a function of the passage of time, so it is not possible to lag faster than one second per second.

I like the symmetry of having the zero line be the center point, and having healthy hosts idle with a flat line at zero. Lag appears in the form of a meander away from zero into the negative, matched by an always equal-area¹ (but not necessarily similarly shaped) correction into the positive. In practice the Y-scale of graphs is fixed at [-1, +1] and the graphs are very easy and quick to interpret.

Example 1

Most slaves are replicating real time; one slave fell behind for some time before catching up again.

Vector – A few small perturbations can be seen, and one slave replicated at less than real time time for many hours, before finally crossing over zero and catching up at an increasing rate until current time was reached.

Delay – The small perturbations are difficult to see due to the scale imposed by the one very delayed slave. Although it’s easy to see on a day view that the slave did catch up quickly, that is less obvious when monitoring in real time.

Example 2

Many slaves with different replication rates, and a lot of trouble.

Vector – Overall replication performance is quite poor, and shows evidence of being unlikely to catch up to current time or maintain real time replication in the future.

Delay – It’s difficult to know if things are getting better or worse. The replication performance of each host is almost impossible to compare.

Implementation

In basic terms, the number of seconds of replication stream applied per second of real time, should be measured frequently, and with reasonably good precision. I have mk-heartbeat writing heartbeat events into a heartbeat table once a second on the master (which has an NTP-synchronized clock), providing a ready source of the progression of “replicated heartbeat time”² (htime) to the slaves. The slaves of course have their own NTP-synchronized clocks providing a source of local “clock time” (ctime). Both of these are collected on each slave once a minute, as integers (Unix epoch timestamps). Both the current sample (subscript c) and the previous successful sample (subscript p) are available to the processing program. The vector is calculated, stored, and sent off to be graphed once per minute.

The implementation is actually quite simple, and tolerant of almost any sampling interval. In the future it could be extended to use millisecond resolution (although it can never be any higher resolution than the frequency the heartbeat is updated).

¹ This is kind of an interesting point. Since the graph is nicely centered on zero, negative numbers represent the exact same scale as positive numbers, on the same dimensions.

² SELECT UNIX_TIMESTAMP(ts) AS ts FROM heartbeat WHERE id = 1

As part of this migration I also changed the URL structure a bit, to match WordPress.com’s directory structure and make things simpler (and easier to move in the future if I want to). There is a mod_rewrite redirector in place to keep the old URLs working forever. (I hope.) The changes are:

• Moving from jcole.us/blog to blog.jcole.us.
• Removing archives/ from the permalink path structure.
• Simplifying the theme a bit. It’s pretty generic right now, but I’ll fix it up some more in the future.

Paul Mikesell (CEO and VP Engineering) and Aaron Passey (CTO) will be discussing the unique architecture behind Clustrix’s massively scalable, distributed, MySQL-compatible database solution. They will talk about how the company has addressed the common challenges associated with achieving massive scale for transactional (OLTP) relational workloads.

Prior to developing the Clustrix solution over the past four years, Paul was co-founder of Isilon Systems, just acquired by EMC in December 2010 for $2.25B and still has the largest clustering capability (>120 nodes) of any NAS solution.

Read more and RSVP on Meetup.com!

mysql> ALTER TABLE t
    ->   ADD KEY `index_a` (`a`),
    ->   ADD KEY `index_b` (`b`),
    ->   ADD KEY `index_c` (`c`);
ERROR 1114 (HY000): The table 't' is full

The error log did not bring particular enlightenment (as usual, InnoDB is extremely verbose with the logs, without saying anything useful):

110105 16:22:30  InnoDB: Error: Write to file (merge) failed at offset 4 1387266048.
InnoDB: 1048576 bytes should have been written, only 888832 were written.
InnoDB: Operating system error number 0.
InnoDB: Check that your OS and file system support files of this size.
InnoDB: Check also that the disk is not full or a disk quota exceeded.
InnoDB: Error number 0 means 'Success'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/operating-system-error-codes.html
110105 16:23:00 [ERROR] /usr/sbin/mysqld: The table 't' is full

I had to re-run the command while keeping a close eye on things, and I discovered that it was writing significant amounts of data to the root file system (which isn’t very big, as usual). I looked in all the usual places, and didn’t see any files of note. However, on a hunch, I checked out /proc/<pid>/fd (which can be a lifesaver). I found these:

# ls -l /proc/`pidof mysqld`/fd | grep deleted
lrwx------ 1 root root 64 Jan  5 17:33 14 -> /var/tmp/ibEnEaSj (deleted)
lrwx------ 1 root root 64 Jan  5 17:33 5 -> /var/tmp/ibEoZHQc (deleted)
lrwx------ 1 root root 64 Jan  5 17:33 6 -> /var/tmp/ibHlWZb3 (deleted)
lrwx------ 1 root root 64 Jan  5 17:33 7 -> /var/tmp/ibUtVhxT (deleted)
lrwx------ 1 root root 64 Jan  5 17:33 8 -> /var/tmp/ibt1daDR (deleted)

I can only assume it’s one of these files that’s growing. Changing the setting of tmpdir fixed things up, and it’s writing its large data files to a place with significant space now (and on a much bigger and faster RAID array, to boot). However, this brings with it a couple of questions:

1. Why does InnoDB need significant space in tmpdir? This is a new requirement with InnoDB plugin (due to online index addition only?), but I don’t see it documented anywhere.²
2. Why are the files deleted while in use? This makes it very painful for a DBA to manage it and see what’s using space. I know it’s a typical Unix paradigm, but cleanup-on-start and leaving the files linked is much easier to manage.
3. Why are the error messages useless? How else is a DBA supposed to track this down?

I could also note that using online index add makes useless the only previous way of getting some sort of a status update while adding indexes: watching the temporary file grow. Perhaps it’s time to bring back my patch to show progress (MySQL Bug #26182)?

¹ Running MySQL-server-percona-5.1.42-9.rhel5 with innodb_version = 1.0.6-9.

² Perhaps it should go on Fast Index Creation in the InnoDB Storage Engine: Limitations at very least?

A paper Did the Early Middle Ages Really Exist? by Dr. Hans-Ulrich Niemitz, lays out the theory (much of it originated from Heribert Illig in the early 1990s). An interesting FAQ written by Jan Beaufort rounds out some the questions. A summary and a few more bits of insight are provided by Gunnar Ries and Ruth Lelarge. The topic is also covered by a DamnInteresting post and thread of comments—some comical and some insightful. Of course any such theory is not without a lot of criticism and counterpoints, as it should have, and Phil John Kneis counters a lot of things nicely. I found the story of The Seven Sleepers of Ephesus as a potential link to possible truth of this story; not that the Seven Sleepers story was at all truth, but that in that time stories (fictions) were made up to relate truths or partial truths quite frequently.

Some of the claims made, often based at least tangentially on well-known reasons for having called them the “Dark Ages” in the first place (and in many cases refuted by others) are:

• Overall lack of reputable, accurately dated literature and historical documents from the time period.
• Much of the otherwise supporting evidence in support of the time period depending on Carbon-14 dating, which has had adjustments made in order to line up with perceived history, and which has been calibrated based on dendrochronology (tree-ring dating) which has had known flaws.
• Architectural development discontinuities; the seemingly complete stoppage of forward progress in architectural style for 300 years, with buildings known to be constructed after the Dark Ages (based on calibration backwards from modern times) compared to buildings known to be constructed before them (based on calibration forwards from ancient times) showing little or no difference.
• Farming, war, and scientific knowledge making practically no advancement during the period.
• A huge spate of forgery of official documents and religious texts otherwise before, during, and after the time period.

It’s not easy to believe one way or the other, but I personally find it not that hard to believe that three hundred years may have been accidentally or intentionally inserted into the calendar for whatever reason. Remember that calendars are man-invented tools, not scientific absolutes, and particularly the points of reference used in them are almost entirely arbitrary. It wouldn’t change our daily lives¹ in any way if the actual number of years having passed since the Roman empire was closer to 1700 than 2000. We’ve made a lot of adjustments to the calendar, and even switched points of reference and entire ways of counting several times during man’s history. Even today, not everyone uses the same calendar or agrees on the calendar.²

It’s easy to misinterpret the meaning of “missing years”, and it seems like many comment authors on the various articles have made the mistake of thinking that Illig and Niemitz are suggesting that the years themselves didn’t exist, which of course is nonsense. The only claim being made is that three centuries of already-sketchy history may have just been fabricated entirely, an offset to the calendar was introduced intentionally or not, and as a result of that, we’ve been mis-numbering years for the 1100 years following that.

If true, there would be no need to correct the current calendar date, just to note in history books, elapsed-time calculations involving the past and including those years, and other places that there is a gap. There are already other gaps in the Gregorian calendar, this would just be the largest.

What do you think? Do you find it plausible? Should I be signing this post January 2, 1714? If true, I guess that gives us an extra 298 years before we the Mayan-predicted end-times in 2012, so that’s a bonus!

¹ With the possible exception that some churches, and some acts of various churches, may be on slightly shakier footings, especially if it could be proven that they had been maliciously involved in the fabrication of the calendar.

² See Wikipedia’s List of calendars and Calendar pages for a few points of study. In addition to our Gregorian calendar, at least the Hebrew and Chinese calendars are still in widespread use.

Starting today, I am now “MySQL Database Architect” at Twitter, where I am joining some old friends on the small but hard-working DBA and operations teams there. I’ll be working to help debug, support, and scale the MySQL databases, of course, and who knows what else. I’m looking forward to the challenges and fast paced operations again. I’m also looking forward to writing a lot more on this blog about MySQL. I’ve had a Twitter account for a long time, but I suppose I’ll write a lot more on it now:

Since I will now be commuting from Sunnyvale to San Francisco nearly daily, I think my old “Commuting” category will get more of a work-out too.

• bitch.mysql.fi — English, of course.
• hundin.mysql.fi — German
• hynda.mysql.fi — Swedish
• narttu.mysql.fi — Finnish
• tik.mysql.fi — Swedish
• tramp.mysql.fi — English, probably. Similar in meaning to some slang uses of “bitch”.

There are a few honorable mentions, which I’m not sure are variants of “bitch”, but very well could be:

• donna.mysql.fi
• mashka.mysql.fi
• mishka.mysql.fi

It’s funny to see how those names live on in “infamy” on Google. Try searching for any of them like “+mysql +hundin“.