I’ve been a user of Kiva for a more than five years now, and have made 350 loans so far for a total of $9,400 loaned. In the first few years I only sporadically made some loans and let repaid money sit around for a long time. In the past couple of years I’ve been using Kiva more consistently, every month re-investing the full repayment amounts as soon as they come in, and usually adding 4-6 loans ($100-$150) of new money.

As I’ve been doing this I noticed an effect that makes perfect sense but I hadn’t considered before: Since the loans are anywhere from 9-24 months, but the repaid amounts are repaid typically monthly, if the repaid amounts are re-invested immediately, the original loan amounts stack on top of each other, allowing the same money to be invested several times over simultaneously.

Recently I’ve been thinking about actually quantifying that effect and figuring out what impact it could have. Since it’s not a very simple calculation, I put together a spreadsheet to calculate the full picture for me.

Assumptions

The following assumptions are made:

• Amount per loan: $25.00 — This is the standard loan amount on Kiva, so this just assumes you never double up on a single loan (which is not a good idea as it spreads the loss risk poorly).
• Investment per month: 4 loans, or $100 (and reinvest all repayments) — This is approximately what I’ve been doing, although frequently it’s a bit more than 4.
• Average loan duration: 15 months — This is the average loan duration for my loan portfolio, and seems about average for Kiva.
• Loss rate: 2.07% — This is the actual loss rate of my portfolio, which is a bit higher than the average Kiva user at 1.09% because I tend to loan to war-torn and riskier areas.

Results

After 5 years (60 months) of consistent and prompt investment, the results could be:

• Total investment: $6,000 — This is the actual amount you’ve paid out of pocket.
• Total loans made: 1,004 — The number of individuals or groups helped. This is the most amazing thing, watching all of these individuals succeed due to your help.
• Total amount loaned: $25,100 — The amount your $6,000 turns into after re-investment through immediate re-loaning.
• Total amount lost: $519.57 — Due to a combination of loan defaults and currency exchange loss, not all of your money will be returned.
• Total amount returned: $5480.43 — If you stopped making loans after the 60 months and started to withdraw your money from Kiva, at the end of it you’d get this much back (investment minus loss).

Check out the full calculator on Google Docs for all the details and per-month amounts.

Conclusion

The really amazing thing with following this plan is that the Kiva borrowers themselves end up—through prompt repayment of their loans—funding each other. For me, the amount being invested each month is quite modest, and through reinvestment of the repayments, the monthly impact is huge. This month, I received almost $400 in repayments, added an additional $100, and made $500 in new loans.

I have been considering a new way to think about, measure, graph, and monitor replication lag for a while. Previously I’ve always been primarily been using replication delay (lag), in seconds. Initially this came from SHOW SLAVE STATUS‘s Seconds_Behind_Master field, later replaced by mk-heartbeat‘s delay information. (These are equivalent, but mk-heartbeat is less buggy and works to represent true lag with relay slaves in the replication topology.) Using delay in seconds to monitor replication has a few problems though:

• The current number of seconds delayed provides no information about how long it might take to catch up.
• It’s very difficult to determine whether a slave is catching up at all, due to the scale. If a slave is a few thousand seconds behind it’s hard to tell whether it’s catching up, falling behind, or neither, at any particular moment.
• If multiple slaves are graphed together, they may have widely different absolute delay values, and thus scales, and it can be very difficult to compare them, or to see that a slave is having problems, until it’s too late. Two slaves may be falling behind at the same rate, but if one is 300 seconds behind, and one is 20,000 seconds behind, the graphs are difficult to interpret.

Given these problems, I determined that while we need the absolute delay information available, it’s not good enough by itself. I started to think about what it is that I’m really trying to determine when looking at the delay graphs.

Vector: Velocity and direction

The key bits of information missing from the delay information seem to be:

• Is the slave falling behind, catching up, or neither? We need a measure of the direction of replication’s delay.
• How fast is the slave falling behind or catching up? We need a measure of velocity of replication’s performance

Fortunately these two things can be combined together into a single number representing the vector of replication. This can then be presented to the user (likely a DBA) in an easy to consume format. The graphs can be read as follows:

• Y = 0 means the slave is neither catching up nor falling behind. It is replicating in real time. I chose zero for this state in order to make the other two states a bit more meaningful and to make the graph symmetric by default.
• Y > 0 means the slave is catching up at Y seconds per second. There is no maximum rate a slave can catch up, but in practice seeing velocities >1 for extended periods is relatively uncommon in already busy systems.
• Y < 0 means the slave is falling behind at Y seconds per second. As a special case, Y = -1 means that the slave is completely stopped and playing no events. Lagging is a function of the passage of time, so it is not possible to lag faster than one second per second.

I like the symmetry of having the zero line be the center point, and having healthy hosts idle with a flat line at zero. Lag appears in the form of a meander away from zero into the negative, matched by an always equal-area¹ (but not necessarily similarly shaped) correction into the positive. In practice the Y-scale of graphs is fixed at [-1, +1] and the graphs are very easy and quick to interpret.

Example 1

Most slaves are replicating real time; one slave fell behind for some time before catching up again.

Vector – A few small perturbations can be seen, and one slave replicated at less than real time time for many hours, before finally crossing over zero and catching up at an increasing rate until current time was reached.

Delay – The small perturbations are difficult to see due to the scale imposed by the one very delayed slave. Although it’s easy to see on a day view that the slave did catch up quickly, that is less obvious when monitoring in real time.

Example 2

Many slaves with different replication rates, and a lot of trouble.

Vector – Overall replication performance is quite poor, and shows evidence of being unlikely to catch up to current time or maintain real time replication in the future.

Delay – It’s difficult to know if things are getting better or worse. The replication performance of each host is almost impossible to compare.

Implementation

In basic terms, the number of seconds of replication stream applied per second of real time, should be measured frequently, and with reasonably good precision. I have mk-heartbeat writing heartbeat events into a heartbeat table once a second on the master (which has an NTP-synchronized clock), providing a ready source of the progression of “replicated heartbeat time”² (htime) to the slaves. The slaves of course have their own NTP-synchronized clocks providing a source of local “clock time” (ctime). Both of these are collected on each slave once a minute, as integers (Unix epoch timestamps). Both the current sample (subscript c) and the previous successful sample (subscript p) are available to the processing program. The vector is calculated, stored, and sent off to be graphed once per minute.

The implementation is actually quite simple, and tolerant of almost any sampling interval. In the future it could be extended to use millisecond resolution (although it can never be any higher resolution than the frequency the heartbeat is updated).

¹ This is kind of an interesting point. Since the graph is nicely centered on zero, negative numbers represent the exact same scale as positive numbers, on the same dimensions.

² SELECT UNIX_TIMESTAMP(ts) AS ts FROM heartbeat WHERE id = 1

As part of this migration I also changed the URL structure a bit, to match WordPress.com’s directory structure and make things simpler (and easier to move in the future if I want to). There is a mod_rewrite redirector in place to keep the old URLs working forever. (I hope.) The changes are:

• Moving from jcole.us/blog to blog.jcole.us.
• Removing archives/ from the permalink path structure.
• Simplifying the theme a bit. It’s pretty generic right now, but I’ll fix it up some more in the future.

Paul Mikesell (CEO and VP Engineering) and Aaron Passey (CTO) will be discussing the unique architecture behind Clustrix’s massively scalable, distributed, MySQL-compatible database solution. They will talk about how the company has addressed the common challenges associated with achieving massive scale for transactional (OLTP) relational workloads.

Prior to developing the Clustrix solution over the past four years, Paul was co-founder of Isilon Systems, just acquired by EMC in December 2010 for $2.25B and still has the largest clustering capability (>120 nodes) of any NAS solution.

Read more and RSVP on Meetup.com!

mysql> ALTER TABLE t
    ->   ADD KEY `index_a` (`a`),
    ->   ADD KEY `index_b` (`b`),
    ->   ADD KEY `index_c` (`c`);
ERROR 1114 (HY000): The table 't' is full

The error log did not bring particular enlightenment (as usual, InnoDB is extremely verbose with the logs, without saying anything useful):

110105 16:22:30  InnoDB: Error: Write to file (merge) failed at offset 4 1387266048.
InnoDB: 1048576 bytes should have been written, only 888832 were written.
InnoDB: Operating system error number 0.
InnoDB: Check that your OS and file system support files of this size.
InnoDB: Check also that the disk is not full or a disk quota exceeded.
InnoDB: Error number 0 means 'Success'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/operating-system-error-codes.html
110105 16:23:00 [ERROR] /usr/sbin/mysqld: The table 't' is full

I had to re-run the command while keeping a close eye on things, and I discovered that it was writing significant amounts of data to the root file system (which isn’t very big, as usual). I looked in all the usual places, and didn’t see any files of note. However, on a hunch, I checked out /proc/<pid>/fd (which can be a lifesaver). I found these:

# ls -l /proc/`pidof mysqld`/fd | grep deleted
lrwx------ 1 root root 64 Jan  5 17:33 14 -> /var/tmp/ibEnEaSj (deleted)
lrwx------ 1 root root 64 Jan  5 17:33 5 -> /var/tmp/ibEoZHQc (deleted)
lrwx------ 1 root root 64 Jan  5 17:33 6 -> /var/tmp/ibHlWZb3 (deleted)
lrwx------ 1 root root 64 Jan  5 17:33 7 -> /var/tmp/ibUtVhxT (deleted)
lrwx------ 1 root root 64 Jan  5 17:33 8 -> /var/tmp/ibt1daDR (deleted)

I can only assume it’s one of these files that’s growing. Changing the setting of tmpdir fixed things up, and it’s writing its large data files to a place with significant space now (and on a much bigger and faster RAID array, to boot). However, this brings with it a couple of questions:

1. Why does InnoDB need significant space in tmpdir? This is a new requirement with InnoDB plugin (due to online index addition only?), but I don’t see it documented anywhere.²
2. Why are the files deleted while in use? This makes it very painful for a DBA to manage it and see what’s using space. I know it’s a typical Unix paradigm, but cleanup-on-start and leaving the files linked is much easier to manage.
3. Why are the error messages useless? How else is a DBA supposed to track this down?

I could also note that using online index add makes useless the only previous way of getting some sort of a status update while adding indexes: watching the temporary file grow. Perhaps it’s time to bring back my patch to show progress (MySQL Bug #26182)?

¹ Running MySQL-server-percona-5.1.42-9.rhel5 with innodb_version = 1.0.6-9.

² Perhaps it should go on Fast Index Creation in the InnoDB Storage Engine: Limitations at very least?

A paper Did the Early Middle Ages Really Exist? by Dr. Hans-Ulrich Niemitz, lays out the theory (much of it originated from Heribert Illig in the early 1990s). An interesting FAQ written by Jan Beaufort rounds out some the questions. A summary and a few more bits of insight are provided by Gunnar Ries and Ruth Lelarge. The topic is also covered by a DamnInteresting post and thread of comments—some comical and some insightful. Of course any such theory is not without a lot of criticism and counterpoints, as it should have, and Phil John Kneis counters a lot of things nicely. I found the story of The Seven Sleepers of Ephesus as a potential link to possible truth of this story; not that the Seven Sleepers story was at all truth, but that in that time stories (fictions) were made up to relate truths or partial truths quite frequently.

Some of the claims made, often based at least tangentially on well-known reasons for having called them the “Dark Ages” in the first place (and in many cases refuted by others) are:

• Overall lack of reputable, accurately dated literature and historical documents from the time period.
• Much of the otherwise supporting evidence in support of the time period depending on Carbon-14 dating, which has had adjustments made in order to line up with perceived history, and which has been calibrated based on dendrochronology (tree-ring dating) which has had known flaws.
• Architectural development discontinuities; the seemingly complete stoppage of forward progress in architectural style for 300 years, with buildings known to be constructed after the Dark Ages (based on calibration backwards from modern times) compared to buildings known to be constructed before them (based on calibration forwards from ancient times) showing little or no difference.
• Farming, war, and scientific knowledge making practically no advancement during the period.
• A huge spate of forgery of official documents and religious texts otherwise before, during, and after the time period.

It’s not easy to believe one way or the other, but I personally find it not that hard to believe that three hundred years may have been accidentally or intentionally inserted into the calendar for whatever reason. Remember that calendars are man-invented tools, not scientific absolutes, and particularly the points of reference used in them are almost entirely arbitrary. It wouldn’t change our daily lives¹ in any way if the actual number of years having passed since the Roman empire was closer to 1700 than 2000. We’ve made a lot of adjustments to the calendar, and even switched points of reference and entire ways of counting several times during man’s history. Even today, not everyone uses the same calendar or agrees on the calendar.²

It’s easy to misinterpret the meaning of “missing years”, and it seems like many comment authors on the various articles have made the mistake of thinking that Illig and Niemitz are suggesting that the years themselves didn’t exist, which of course is nonsense. The only claim being made is that three centuries of already-sketchy history may have just been fabricated entirely, an offset to the calendar was introduced intentionally or not, and as a result of that, we’ve been mis-numbering years for the 1100 years following that.

If true, there would be no need to correct the current calendar date, just to note in history books, elapsed-time calculations involving the past and including those years, and other places that there is a gap. There are already other gaps in the Gregorian calendar, this would just be the largest.

What do you think? Do you find it plausible? Should I be signing this post January 2, 1714? If true, I guess that gives us an extra 298 years before we the Mayan-predicted end-times in 2012, so that’s a bonus!

¹ With the possible exception that some churches, and some acts of various churches, may be on slightly shakier footings, especially if it could be proven that they had been maliciously involved in the fabrication of the calendar.

² See Wikipedia’s List of calendars and Calendar pages for a few points of study. In addition to our Gregorian calendar, at least the Hebrew and Chinese calendars are still in widespread use.

Starting today, I am now “MySQL Database Architect” at Twitter, where I am joining some old friends on the small but hard-working DBA and operations teams there. I’ll be working to help debug, support, and scale the MySQL databases, of course, and who knows what else. I’m looking forward to the challenges and fast paced operations again. I’m also looking forward to writing a lot more on this blog about MySQL. I’ve had a Twitter account for a long time, but I suppose I’ll write a lot more on it now:

Since I will now be commuting from Sunnyvale to San Francisco nearly daily, I think my old “Commuting” category will get more of a work-out too.

• bitch.mysql.fi — English, of course.
• hundin.mysql.fi — German
• hynda.mysql.fi — Swedish
• narttu.mysql.fi — Finnish
• tik.mysql.fi — Swedish
• tramp.mysql.fi — English, probably. Similar in meaning to some slang uses of “bitch”.

There are a few honorable mentions, which I’m not sure are variants of “bitch”, but very well could be:

• donna.mysql.fi
• mashka.mysql.fi
• mishka.mysql.fi

It’s funny to see how those names live on in “infamy” on Google. Try searching for any of them like “+mysql +hundin“.

mysql> show processlist G
*************************** 1. row ***************************
     Id: 5979887
   User: root
   Host: localhost:55997
     db: NULL
Command: Sleep
   Time: 475
  State:
   Info: NULL

How do you find that client, especially if it’s on another host? MySQL is providing you all the information you need above: localhost:55997. Of course localhost is the host or IP address, and 55997 is the source port of the socket; the port number (usually randomly assigned) on the far end of the socket, from the MySQL server’s perspective. You can turn that number into something useful—the PID and user—, by running the following command on the host that made the connection:

# lsof -nPi :55997
COMMAND  PID  USER   FD   TYPE    DEVICE SIZE NODE NAME
mysqld  5026 mysql 1654u  IPv4 303329996       TCP 127.0.0.1:3306->127.0.0.1:55997 (ESTABLISHED)
mysql   9146 jcole    3u  IPv4 303329995       TCP 127.0.0.1:55997->127.0.0.1:3306 (ESTABLISHED)

(Note that here you can see both sides of the socket, since I’m running these commands on localhost. Disregard the first entry, as it’s the half of the connection owned by the MySQL server.)

You can then find out the full command-line of the process with ps:

# ps -fp 9146
UID        PID  PPID  C STIME TTY          TIME CMD
jcole     9146  8740  0 12:53 pts/3    00:00:00 mysql -h 127.0.0.1 -u root -p

And see what it’s doing with strace (waiting on a read of stdin in this dumb test):

# strace -p 9146
Process 9146 attached - interrupt to quit
read(0,  <unfinished ...>
Process 9146 detached

I hope that’s helpful! It’s a pretty common debugging trick for me.

When running MySQL on a large system (e.g., 64GB RAM and dual quad core CPUs) with a large InnoDB buffer pool (e.g., 48GB), over time, Linux decides to swap out potentially large amounts of memory, despite appearing¹ to be under no real memory pressure. Monitoring reveals that at no time is the system in actual need of more memory than it has available; and memory isn’t leaking, mysqld‘s RSS is normal and stable.

Normally a tiny bit of swap usage could be OK (we’re really concerned about activity—swaps in and out), but in many cases, “real” useful memory is being swapped: primarily parts of InnoDB’s buffer pool. When it’s needed once again, a big performance hit is taken to swap it back in, causing random delays in random queries. This can cause overall unpredictable performance on production systems, and often once swapping starts, the system may enter a performance death-spiral.

While not every system, and not every workload experiences this problem, it’s common enough that it’s well known, and for those that know it well it can be a major headache.

The history of “swap insanity”

Over the past two to four years, there has been an off-and-on discussion about Linux swapping and MySQL, often titled “swap insanity” (I think coined by Kevin Burton). I have followed it closely, but I haven’t contributed much because I didn’t have anything new to add. The major contributors to the discussion over the past years have been:

• Kevin Burton — Discussion of swappiness and MySQL on Linux.
• Kevin Burton — Proposed IO_DIRECT as a solution (doesn’t work) and discussed memlock (may help, but not a full solution).
• Peter Zaitsev — Discussed swappiness, memlock, and fielded a lot of discussion in the comments.
• Don MacAskill — Proposed an innovative (albeit hacky) solution using swap on ramdisk, and a lot more interesting discussion in the comments.
• Dathan Pattishall — Describes how Linux behavior can be even worse with swap disabled, and proposes using swapoff to clear it, but no real solution.
• Rik van Riel on the LKML — A few answers and proposal of the Split-LRU patch.
• Kevin Burton — Discussion of Linux Split-LRU patch with some success.
• Mark Callaghan — Discussion of vmstat and monitoring things, and a recap of a few possible solutions.
• Kevin Burton — More discussion that Linux Split-LRU is essential.
• Kevin Burton — Choosing the middle road by enabling swap, but with a small amount of space, and giving up the battle.
• Peter Zaitsev — More discussion about why swapping is bad, but no solution.

Despite all the discussion, not much has changed. There are some hacky solutions to get MySQL to stop swapping, but nothing definite. I’ve known these solutions and hacks now for a while, but the core question was never really settled: “Why does this happen?” and it’s never sat well with me. I was recently tasked with trying to sort this mess out once and for all, so I’ve now done quite a bit of research and testing related to the problem. I’ve learned a lot, and decided a big blog entry might be the best way to share it. Enjoy.

There was a lot of discussion and some work went into adding the relatively new swappiness tunable a few years ago, and I think that may have solved some of the original problems, but at around the same time the basic architecture of the machine changed to NUMA, which I think introduced some new problems, with the very same symptoms, masking the original fix.

Contrasting the SMP/UMA and NUMA architectures

The SMP/UMA architecture

The SMP, or UMA architecture, simplified

When the PC world first got multiple processors, they were all arranged with equal access to all of the memory in the system. This is called Symmetric Multi-processing (SMP), or sometimes Uniform Memory Architecture (UMA, especially in contrast to NUMA). In the past few years this architecture has been largely phased out between physical socketed processors, but is still alive and well today within a single processor with multiple cores: all cores have equal access to the memory bank.

The NUMA architecture

The NUMA architecture, simplified

The new architecture for multiple processors, starting with AMD’s Opteron and Intel’s Nehalem² processors (we’ll call these “modern PC CPUs”), is a Non-Uniform Memory Access (NUMA) architecture, or more correctly Cache-Coherent NUMA (ccNUMA). In this architecture, each processor has a “local” bank of memory, to which it has much closer (lower latency) access. The whole system may still operate as one unit, and all memory is basically accessible from everywhere, but at a potentially higher latency and lower performance.

Fundamentally, some memory locations (“local” ones) are faster, that is, cost less to access, than other locations (“remote” ones attached to other processors). For a more detailed discussion of NUMA implementation and its support in Linux, see Ulrich Drepper’s article on LWN.net.

How Linux handles a NUMA system

Linux automatically understands when it’s running on a NUMA architecture system and does a few things:

1. Enumerates the hardware to understand the physical layout.
2. Divides the processors (not cores) into “nodes”. With modern PC processors, this means one node per physical processor, regardless of the number of cores present.
3. Attaches each memory module in the system to the node for the processor it is local to.
4. Collects cost information about inter-node communication (“distance” between nodes).

You can see how Linux enumerated your system’s NUMA layout using the numactl --hardware command:

# numactl --hardware
available: 2 nodes (0-1)
node 0 size: 32276 MB
node 0 free: 26856 MB
node 1 size: 32320 MB
node 1 free: 26897 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10

This tells you a few important things:

• The number of nodes, and their node numbers — In this case there are two nodes numbered “0″ and “1″.
• The amount of memory available within each node — This machine has 64GB of memory total, and two physical (quad core) CPUs, so it has 32GB in each node. Note that the sizes aren’t exactly half of 64GB, and aren’t exactly equal, due to some memory being stolen from each node for whatever internal purposes the kernel has in mind.
• The “distance” between nodes — This is a representation of the cost of accessing memory located in (for example) Node 0 from Node 1. In this case, Linux claims a distance of “10″ for local memory and “21″ for non-local memory.

How NUMA changes things for Linux

Technically, as long as everything runs just fine, there’s no reason that being UMA or NUMA should change how things work at the OS level. However, if you’re to get the best possible performance (and indeed in some cases with extreme performance differences for non-local NUMA access, any performance at all) some additional work has to be done, directly dealing with the internals of NUMA. Linux does the following things which might be unexpected if you think of CPUs and memory as black boxes:

• Each process and thread inherits, from its parent, a NUMA policy. The inherited policy can be modified on a per-thread basis, and it defines the CPUs and even individual cores the process is allowed to be scheduled on, where it should be allocated memory from, and how strict to be about those two decisions.
• Each thread is initially allocated a “preferred” node to run on. The thread can be run elsewhere (if policy allows), but the scheduler attempts to ensure that it is always run on the preferred node.
• Memory allocated for the process is allocated on a particular node, by default “current”, which means the same node as the thread is preferred to run on. On UMA/SMP architectures all memory was treated equally, and had the same cost, but now the system has to think a bit about where it comes from, because accessing non-local memory has implications on performance and may cause cache coherency delays.
• Memory allocations made on one node will not be moved to another node, regardless of system needs. Once memory is allocated on a node, it will stay there.

The NUMA policy of any process can be changed, with broad-reaching effects, very simply using numactl as a wrapper for the program. With a bit of additional work, it can be fine-tuned in detail by linking in libnuma and writing some code yourself to manage the policy. Some interesting things that can be done simply with the numactl wrapper are:

• Allocate memory with a particular policy:
  • locally on the “current” node — using --localalloc, and also the default mode
  • preferably on a particular node, but elsewhere if necessary — using --preferred=node
  • always on a particular node or set of nodes — using --membind=nodes
  • interleaved, that is, spread evenly round-robin across all or a set of nodes — using --interleaved=all or --interleaved=nodes
• Run the program on a particular node or set of nodes, in this case that means physical CPUs (--cpunodebind=nodes) or on a particular core or set of cores (--physcpubind=cpus).

What NUMA means for MySQL and InnoDB

InnoDB, and really, nearly all database servers (such as Oracle), present an atypical workload (from the point of view of the majority of installations) to Linux: a single large multi-threaded process which consumes nearly all of the system’s memory and should be expected to consume as much of the rest of the system resources as possible.

In a NUMA-based system, where the memory is divided into multiple nodes, how the system should handle this is not necessarily straightforward. The default behavior of the system is to allocate memory in the same node as a thread is scheduled to run on, and this works well for small amounts of memory, but when you want to allocate more than half of the system memory it’s no longer physically possible to even do it in a single NUMA node: In a two-node system, only 50% of the memory is in each node. Additionally, since many different queries will be running at the same time, on both processors, neither individual processor necessarily has preferential access to any particular part of memory needed by a particular query.

It turns out that this seems to matter in one very important way. Using /proc/pid/numa_maps we can see all of the allocations made by mysqld, and some interesting information about them. If you look for a really big number in the anon=size, you can pretty easily find the buffer pool (which will consume more than 51GB of memory for the 48GB that it has been configured to use) [line-wrapped for clarity]:

2aaaaad3e000 default anon=13240527 dirty=13223315
  swapcache=3440324 active=13202235 N0=7865429 N1=5375098

The fields being shown here are:

• 2aaaaad3e000 — The virtual address of the memory region. Ignore this other than the fact that it’s a unique ID for this piece of memory.
• default — The NUMA policy in use for this region.
• anon=number — The number of anonymous pages mapped.
• dirty=number — The number of pages that are dirty because they have been modified. Generally memory allocated only within a single process is always going to be used, and thus dirty, but if a process forks it may have many copy-on-write pages mapped that are not dirty.
• swapcache=number — The number of pages swapped out but unmodified since they were swapped out, and thus they are ready to be freed if needed, but are still in memory at the moment.
• active=number — The number of pages on the “active list”; if this field is shown, some memory is inactive (anon minus active) which means it may be paged out by the swapper soon.
• N0=number and N1=number — The number of pages allocated on Node 0 and Node 1, respectively.

The entire numa_maps can be quickly summarized by the a simple script numa-maps-summary.pl, which I’ve written while analyzing this problem:

N0        :      7983584 ( 30.45 GB)
N1        :      5440464 ( 20.75 GB)
active    :     13406601 ( 51.14 GB)
anon      :     13422697 ( 51.20 GB)
dirty     :     13407242 ( 51.14 GB)
mapmax    :          977 (  0.00 GB)
mapped    :         1377 (  0.01 GB)
swapcache :      3619780 ( 13.81 GB)

An couple of interesting and somewhat unexpected things pop out to me:

1. The sheer imbalance in how much memory is allocated in Node 0 versus Node 1. This is actually absolutely normal per the default policy. Using the default NUMA policy, memory was preferentially allocated in Node 0, but Node 1 was used as a last resort.
2. The sheer amount of memory allocated in Node 0. This is absolutely critical — Node 0 is out of free memory! It only contains about 32GB of memory in total, and it has allocated a single large chunk of more than 30GB to InnoDB’s buffer pool. A few other smaller allocations to other processes finish it off, and suddenly it has no memory free isn’t even caching anything.

The memory allocated by MySQL looks something like this:

Allocating memory severely imbalanced, preferring Node 0

Due to Node 0 being completely exhausted of free memory, even though the system has plenty of free memory overall (over 10GB has been used for caches) it is entirely on Node 1. If any process scheduled on Node 0 needs local memory for anything, it will cause some of the already-allocated memory to be swapped out in order to free up some Node 0 pages. Even though there is free memory on Node 1, the Linux kernel in many circumstances (which admittedly I don’t totally understand³) prefers to page out Node 0 memory rather than free some of the cache on Node 1 and use that memory. Of course the paging is far more expensive than non-local memory access ever would be.

A small change, to big effect

An easy solution to this is to interleave the allocated memory. It is possible to do this using numactl as described above:

# numactl --interleave all command

We can use this with MySQL by making a one-line change to mysqld_safe, adding the following line (after cmd="$NOHUP_NICENESS"), which prefixes the command to start mysqld with a call to numactl:

cmd="/usr/bin/numactl --interleave all $cmd"

Now, when MySQL needs memory it will allocate it interleaved across all nodes, effectively balancing the amount of memory allocated in each node. This will leave some free memory in each node, allowing the Linux kernel to cache data on both nodes, thus allowing memory to be easily freed on either node just by freeing caches (as it’s supposed to work) rather than paging.

Performance regression testing has been done comparing the two scenarios (default local plus spillover allocation versus interleaved allocation) using the DBT2 benchmark, and found that performance in the nominal case is identical. This is expected. The breakthrough comes in that: In all cases where swap use could be triggered in a repeatable fashion, the system no longer swaps!

You can now see from the numa_maps that all allocated memory has been spread evenly across Node 0 and Node 1:

2aaaaad3e000 interleave=0-1 anon=13359067 dirty=13359067
  N0=6679535 N1=6679532

And the summary looks like this:

N0        :      6814756 ( 26.00 GB)
N1        :      6816444 ( 26.00 GB)
anon      :     13629853 ( 51.99 GB)
dirty     :     13629853 ( 51.99 GB)
mapmax    :          296 (  0.00 GB)
mapped    :         1384 (  0.01 GB)

In graphical terms, the allocation of all memory within mysqld has been made in a balanced way:

Allocating memory balanced (interleaved) across nodes

An aside on zone_reclaim_mode

The zone_reclaim_mode tunable in /proc/sys/vm can be used to fine-tune memory reclamation policies in a NUMA system. Subject to some clarifications from the linux-mm mailing list, it doesn’t seem to help in this case.

An even better solution?

It occurred to me (and was backed up by the linux-mm mailing list) that there is probably further room for optimization, although I haven’t done any testing so far. Interleaving all allocations is a pretty big hammer, and while it does solve this problem, I wonder if an even better solution would be to intelligently manage the fact that this is a NUMA architecture, using the libnuma library. Some thoughts that come to mind are:

• Spread the buffer pool across all nodes intelligently in large chunks, or by index, rather than round-robin per page.
• Keep the allocation policy for normal query threads to “local” so their memory isn’t interleaved across both nodes. I think interleaved allocation could cause slightly worse performance for some queries which would use a substantial amount of local memory (such as for large queries, temporary tables, or sorts), but I haven’t tested this.
• Managing I/O in and out to/from the buffer pool using threads that will only be scheduled on the same node that the memory they will use is allocated on (this is a rather complex optimization).
• Re-schedule simpler query threads (many PK lookups, etc.) on nodes with local access to the data they need. Move them actively when necessary, rather than keeping them on the same node. (I don’t know if the cost of the switch makes up for this, but it could be trivial if the buffer pool were organized by index onto separate nodes.)

I have no idea if any of the above would really show practical benefits in a real-world system, but I’d love to hear any comments or ideas.

Update 1: Changed the link for “Rik van Riel on the LKML — A few answers and proposal of the Split-LRU patch.” to be a bit closer to my intention. The old link points to the message that started the thread, the new link points to the index of the messages in the thread.

Update 2: Added a link above provided by Kevin Closson about Oracle on NUMA systems.

- – -

¹ Using free shows some memory free and lots of cache in use, and totalling up the resident set sizes from ps or top shows that the running processes don’t need more memory than is available.

² An article in Dr. Dobb’s Journal titled A Deeper Look Inside Intel QuickPath Interconnect gives pretty good high level coverage. Intel published a paper entitled Performance Analysis Guide for Intel® Core^TM i7 Processor and Intel® Xeon^TM 5500 processors which is quite good for understanding the internals of NUMA and QPI on Intel’s Nehalem series of processors.

³ I started a thread on the linux-mm mailing list related to MySQL on NUMA, and there are two other threads related on zone_reclaim_mode and on swapping.