Comments on: TokuDB configuration variables of interest

By: Igor Shevtsov

Igor Shevtsov — Tue, 12 Jan 2016 09:18:09 +0000

Hi Shlomi, what a nice article! I keep coming back to it.
Percona included Hot TokuDB backup into 5.6 Percona Server.
It works great, but I found 1 issue I want to share here. If tokudb_commit_sync = 0 and flushing to a storage controlled by tokudb_fsync_log_period, on a busy server you might have issues connecting to the Master. I take tokudb_backup snapshot on the slave with replication position recorded, so when the snapshot is restored and TokuDB recovered, replication fails with ERROR for update/delete row missing. I don’t have issues when tokudb_commit_sync = 1.

By: Operationalizing TokuDB | MySQL and Stuff

Operationalizing TokuDB | MySQL and Stuff — Mon, 09 Feb 2015 19:44:49 +0000

[…] seeing “low available memory” alerts. A bit of digging led me to Shlomi Noach’s blog post where he tried TokuDB, As he mentions, by default, TokuDB relies on the OS file cache for […]

By: Shantanu Oak

Shantanu Oak — Sun, 27 Oct 2013 03:11:32 +0000

TokuDB default parameters worked surprisingly well for me. I got an error “Too many open files”, so I increased the file limit.

open-files-limit=40000

By: Igor Shevtsov

Igor Shevtsov — Thu, 24 Oct 2013 18:54:19 +0000

@shlomi I did it both ways: LOAD INTO and INSERT INTO SELECT *… both failed. I didn’t want to ALTER in place as that table was still in use by our customers. I thought you had the same problem altering a big table and you had your binary log enabled, so why didn’t you have this issue as I did? what is your secret? 🙂 I will ask this on Tokudb google-list,sure.

By: shlomi

shlomi — Thu, 24 Oct 2013 18:23:50 +0000

But, I’m not too focused. If this is TokuDB you’re importing to, best to ask on TokuDB-user google group to get answer from Tokutek.

By: shlomi

shlomi — Thu, 24 Oct 2013 18:19:45 +0000

You tried to LOAD DATA INFILE a 31GB cvs file? I guess the problem is now solved. See, it loads as a single transaction.
Try splitting the file (eg using unix split command) and load in smaller chunks.

Here’s an old yet relevant read: http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/

By: Igor Shevtsov

Igor Shevtsov — Thu, 24 Oct 2013 11:38:37 +0000

@shiomi the max_binlog_cache_size was a default value of 4294963200 . I changed it to 62914560000 .
SET GLOBAL max_binlog_cache_size = 60000 * 1024 * 1024;
The Transaction size.. How to measure it? I did a load into csv file 31Gig. – failed
Than I did another import of 5Gig csv file. failed as well with max_binlog_cache_size set to 60gig
binlog_format set to ‘MIXED’.

By: shlomi

shlomi — Thu, 24 Oct 2013 10:43:22 +0000

@Igor, well what was your max_binlog_cache_size? What was the size of your transaction?

By: Igor Shevtsov

Igor Shevtsov — Thu, 24 Oct 2013 08:03:11 +0000

@Shiomi once again thanks for a great article. I’m going to send it to my manager, so he appropriate the work we did with the migration process.
One more issue I had trying to load a big file was an error: “ERROR 1197 (HY000): Multi-statement transaction required more than ‘max_binlog_cache_size’ bytes of storage; increase this mysqld variable and try again” I fixed it by disabling writing to binlog for my session. SET sql_log_bin = 0; Did you have this issue and how did you deal with it? many thanks.

By: shlomi

shlomi — Thu, 24 Oct 2013 05:29:34 +0000

@Gerry,
Thanks. Apologies on the typo, fixed.