How to load a IBD file in MySQL?

Ending 2020 by losing all your MySQL data seems about right doesn’t it? I spent a full day recovering my data after updating to the latest Docker version on a NAS. It nuked my MySQL container in the process. Luckily, I had some database files but they were in .frm and .ibd format which are not easy to import, especially once the server is gone. In this article I’ll show you how to restore your MySQL data on MacOS if you only have .frm and .ibd files available. Hopefully, this will save you some time if you are in the same position I found myself in.

A brief overview of the files we have and what we’ll be doing to salvage the data:

• .frm: Contains a single table definition, essentially create table statements
• .ibd: This file holds the data and indexes for a single table on the file system.

Restore Process High-Level:

• Generate create statements from .frm files to get our table structures back
• Reset the tablespace: when an InnoDB table is truncated, or dropped and recreated, it gets a new table ID. Any ID mismatch between the table in the database and the backed-up table can prevent it from being restored. Since we’re restoring this on a different MySQL Server instance, we need to drop the tables ids.
• Put our .ibd files into the data directory
• Sync the tablespace: Import new tables ids
• Backup our data so we have full backup which can easily be restored

Let’s Get Started

Install MySQL shell client:

brew install mysql-client

Add it to your path:

export PATH="/usr/local/opt/mysql-client/bin:$PATH"

Reload your shell

source ~/.zshrc

Let’s begin by starting up a fresh docker instance that we’ll use to restore our database.

Create config and data volumes. The config volume will hold additional MySQL configuration and the data volume will be where our data goes on disk.

mkdir ~/docker/mysql/conf.d
mkdir ~/docker/mysql/data

Add the following to a ~/docker/mysql/conf.d/config.cnf

[mysqld]
innodb_force_recovery = 0

Run a MySQL docker instance

docker run \
--detach \
--name=mysql2 \
--env="MYSQL_ROOT_PASSWORD=mypassword" \
--publish 32780:3306 \
--volume=/pathToMysqlConfig/conf.d:/etc/mysql/conf.d \
--volume=/pathToDataDirectory/mysql2:/var/lib/mysql:rw mysql:5.7

Step 1: Create Files to Reconstruct Table Structure

There are two ways to reconstruct the tables. The first option is to use MySQL command line utilities and the mysqlfrm tool. The other option is to create our tables with a dummy structure using some command line magic and then replace the dummy .frm’s in our data directory with the actual .frms. I’ve used both but prefer the former, I’ll show you how to do both.

Step 1.1: Use MySQL’s command line utilities

The Python MySQL connector is required. Download it from here. Download MySQL Utilities source code by selecting “Source Code” from “Select Platform”. I downloaded the Windows distribution and unzipped it (running on a mac) From a terminal:

$ cd path/to/unzipped/mysql-utilities-1.5.6
$ python ./setup.py build
$ sudo python ./setup.py install

Use mysqlfrm to generate table creates

mysqlfrm  --diagnostic ~/folder/*.frm --quiet > ~/folder/recoverme-table-structures.sql

Set the ROW_FORMAT to compact

sed -i '' -e 's/ENGINE=InnoDB/ ENGINE=InnoDB ROW_FORMAT=COMPACT DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci/' recoverme-table-structures.sql

Step 2 - Generate dummy tables creates using .frm files

Even though we’re going to use the output from Step 1, we will use the files generated in this step to reset the tablespace and import so go ahead and follow along. This is useful when you have many tables to restore.

First we’ll get the table names from our .frm files

ls -1 *.frm > tables.sql

Prepend create table in front of each table name

export PATH="/usr/local/opt/mysql-client/bin:$PATH"

Replace .frm and add a dummy int column with engine innodb

export PATH="/usr/local/opt/mysql-client/bin:$PATH"

At this point, you should have three files: recoverme-table-structures.sql, tables.sql, rtables.sql.

Step 3 - Create Our Tables

First let’s create our database

export PATH="/usr/local/opt/mysql-client/bin:$PATH"

Next, run sql

export PATH="/usr/local/opt/mysql-client/bin:$PATH"

If you were to run rtables.sql instead of recoverme-table-structures.sql, you would have your tables added with a dummy id column. At that point, you would stop mysql and then copy over your .frm files into the mysql database directory, replacing the existing .frm files. After that you would run

source ~/.zshrc

Step 4 - Replace Data

Now the spot we’re in is that our internal id of our tables is different than the one in our .ibd files. We’ll manually sync our tablespaces.

Create remove-tablespace.sql to remove the existing internal ids.

export PATH="/usr/local/opt/mysql-client/bin:$PATH"

Create sync-tablespace.sql to import tablespace from existing data.

export PATH="/usr/local/opt/mysql-client/bin:$PATH"

Run remove-tablespace.sql

export PATH="/usr/local/opt/mysql-client/bin:$PATH"

Remove any existing .ibd files in our MySQL directory

export PATH="/usr/local/opt/mysql-client/bin:$PATH"

Copy .ibd files over to MySQL directory

export PATH="/usr/local/opt/mysql-client/bin:$PATH"

Sync tablespace

export PATH="/usr/local/opt/mysql-client/bin:$PATH"

If you get error - Schema mismatch (Table has ROW_TYPE_COMPACT row format, .ibd file has ROW_TYPE_DYNAMIC row format.). Restart the process and sed ROW_FORMAT=DYNAMIC instead of ROW_FORMAT=COMPACT

Dump our database to create a backup

source ~/.zshrc

Now we have a legitimate MySQL dump of our data. We can import that in with the correct database name

source ~/.zshrc

That’s it! There are certainly more issues that can come up if the ibd files are corrupted. I encourage you to check out the Twindb github for some excellent tools on repairing ibd files and general data recovery practices.

How do I load a file into MySQL?

How we can open IBD file?

How to import tablespace in MySQL?

What is .IBD file in MySQL?