The MySQL 5.5 (and 5.6) documentation says, in Identifying the File Format in Use:
“… Otherwise, the least significant bit should be set in the tablespace flags, and the file format identifier is written in the bits 5 through 11. …”
This is incorrect for any version due to a bug in how the tablespace flags were stored (which caused only 1 bit to be reserved, rather than 6). This was all re-worked in MySQL 5.6, so someone obviously noticed it, but the documentation has been left incorrect for all versions, and the incorrect and misleading code has been left in MySQL 5.5. I filed MySQL Bug #68868 about the documentation.
File formats and names
There are file format names in the documentation and code for values 0 through 25 (letters “A” through “Z”), although only 0 (“Antelope”) and 1 (“Barracuda”) are currently used. They are all defined in storage/innobase/trx/trx0sys.c:
97 /** List of animal names representing file format. */
98 static const char* file_format_name_map[] = {
99 "Antelope",
100 "Barracuda",
101 "Cheetah",
102 "Dragon",
103 "Elk",
104 "Fox",
105 "Gazelle",
106 "Hornet",
107 "Impala",
108 "Jaguar",
109 "Kangaroo",
110 "Leopard",
111 "Moose",
112 "Nautilus",
113 "Ocelot",
114 "Porpoise",
115 "Quail",
116 "Rabbit",
117 "Shark",
118 "Tiger",
119 "Urchin",
120 "Viper",
121 "Whale",
122 "Xenops",
123 "Yak",
124 "Zebra"
125 };
How only one bit was reserved
The code to store the file format identifier into an InnoDB tablespace file’s tablespace flags is in storage/innobase/include/dict0mem.h and follows, with my commentary.
The first bit is reserved for 1 = compact, 0 = redundant format:
70 /** Table flags. All unused bits must be 0. */
71 /* @{ */
72 #define DICT_TF_COMPACT 1 /* Compact page format.
73 This must be set for
74 new file formats
75 (later than
76 DICT_TF_FORMAT_51). */
The next 4 bits are reserved for the compressed page size:
78 /** Compressed page size (0=uncompressed, up to 15 compressed sizes) */
79 /* @{ */
80 #define DICT_TF_ZSSIZE_SHIFT 1
81 #define DICT_TF_ZSSIZE_MASK (15 << DICT_TF_ZSSIZE_SHIFT)
82 #define DICT_TF_ZSSIZE_MAX (UNIV_PAGE_SIZE_SHIFT - PAGE_ZIP_MIN_SIZE_SHIFT + 1)
83 /* @} */
Next we’re supposed to reserve 6 bits for the file format (up to 64 formats):
85 /** File format */
86 /* @{ */
87 #define DICT_TF_FORMAT_SHIFT 5 /* file format */
88 #define DICT_TF_FORMAT_MASK \
89 ((~(~0 << (DICT_TF_BITS - DICT_TF_FORMAT_SHIFT))) << DICT_TF_FORMAT_SHIFT)
Two values are currently defined, which correspond to Antelope and Barracuda (with rather strange names “51” and “ZIP” as defined):
90 #define DICT_TF_FORMAT_51 0 /*!< InnoDB/MySQL up to 5.1 */
91 #define DICT_TF_FORMAT_ZIP 1 /*!< InnoDB plugin for 5.1:
92 compressed tables,
93 new BLOB treatment */
94 /** Maximum supported file format */
95 #define DICT_TF_FORMAT_MAX DICT_TF_FORMAT_ZIP
96
97 /** Minimum supported file format */
98 #define DICT_TF_FORMAT_MIN DICT_TF_FORMAT_51
This is where things get interesting. It is not clear if DICT_TF_BITS (defined below) is supposed to represent the total number of flag bits (11 so far!), or the number of bits for the format above (6, but then shouldn’t it be called DICT_TF_FORMAT_BITS?). However since 6 is larger than the non-format related bits (5), and only 1 bit has actually been used for format in practice (0..1), nothing will blow up here, and the #error check passes cleanly.
100 /* @} */
101 #define DICT_TF_BITS 6 /*!< number of flag bits */
102 #if (1 << (DICT_TF_BITS - DICT_TF_FORMAT_SHIFT)) <= DICT_TF_FORMAT_MAX
103 # error "DICT_TF_BITS is insufficient for DICT_TF_FORMAT_MAX"
104 #endif
105 /* @} */
Also note that the #error there is easy enough to calculate. It works out to:
- (1 << (DICT_TF_BITS - DICT_TF_FORMAT_SHIFT)) <= DICT_TF_FORMAT_MAX
- (1 << (6 - 5)) <= 1
- (1 << 1) <= 1
- 2 <= 1
- FALSE
The “6 - 5” in the calculation above represents essentially the number of bits reserved for the table format flag, which turns out to be only 1.
The above defines go on to be used by DICT_TF2 (another set of flags) which currently only uses a single bit:
107 /** @brief Additional table flags.
108
109 These flags will be stored in SYS_TABLES.MIX_LEN. All unused flags
110 will be written as 0. The column may contain garbage for tables
111 created with old versions of InnoDB that only implemented
112 ROW_FORMAT=REDUNDANT. */
113 /* @{ */
114 #define DICT_TF2_SHIFT DICT_TF_BITS
115 /*!flags. */
117 #define DICT_TF2_TEMPORARY 1 /*!< TRUE for tables from
118 CREATE TEMPORARY TABLE. */
119 #define DICT_TF2_BITS (DICT_TF2_SHIFT + 1)
120 /*!flags. */
122 /* @} */
It’s very easy to see here that if DICT_TF2_SHIFT is DICT_TF_BITS, which is 6, the DICT_TF2_TEMPORARY flag is being stored at 1 << 6, which is only leaving the file format a single bit, when it should be reserving 6 bits.
The end result of this is that the DICT_TF2_TEMPORARY bit is being stored into a bit reserved for the table format, rather than after the table format. The DICT_TF2 stuff seems to only be stored in the data dictionary, and never in the IBD file, so this would I guess manifest when Cheetah would be implemented and a temporary table is created.
Why this could happen
This code is unnecessarily complex and confusing, and to make matters worse it is inconsistent. There is no concise description of the fields being stored; only the code documents the structure, and since it is badly written, its value as documentation is low.
The bug is two-fold:
- There should be a DICT_TF_FORMAT_BITS define to capture the expected number of bits required to store the DICT_TF_FORMAT_* structure (dictionary, table flags, format) which is defined to 6, and that should be used in the masks associated with DICT_TF_FORMAT_*.
- The DICT_TF_BITS define should mean the total size of the DICT_TF structures (which precede the DICT_TF2 structures obviously), and should be 1 + 4 + 6 = 11 bits, but this should be defined only by summing the previous structures sizes.
Because of the way this is written, it’s actually quite difficult to discern that there is a bug present visually, so I am not surprised that this was not caught — however I am dismayed about the code quality and clarity, and that this passes any sort of code review.
Jeremy Cole 