Skip to content

# On compiling TokuDB from source

MySQL TokuDB May 2, 2013May 2, 2013

Sharing my experience of compiling TokuDB + MariaDB 5.5. Why? Because I must have this patch to Sphinx 2.0.4.

Note: I was using what seems to be the “old” method of compiling; quoting Leif Walsh:

> … We are looking at deprecating that method of building (MariaDB source plus binary fractal tree handlerton). It only really needed to be that complex when we were closed source.

I also tried the “new” method of compiling, which I couldn’t work out.

Here’s how it goes: TokuDB is newly released as open source. As such, it got a lot of attention, many downloads and I hope it will succeed.

However as stable as the product may be, it’s new to open source, which means anyone compiling it from source is an early adopter (at least for the compilation process).

#### Installation process

This is an unorthodox, and actually weird process. See section 6 on the Tokutek docs. In order to compile the project you must download:

- The source code tar.gz
- And the binary (?!) tar.gz
- And the binary checksum
- And the Tokutek patches
- And the patches checksum

You extract the source tarball. But instead of doing the standard “./configure && make && sudo make install” you need to copy a shell script called tokudb.build.bash one directory level up, and run it from there.

tokudb.build.bash lists gcc47 and g++47 on rows 3, 4. Modify “gcc47” to “gcc”, modify “g++47” to “g++”. I’m assuming you don’t have a binary called gcc47. Why would you?

#### Dependencies

You will need CMake >= 2.8

This means Ubuntu LTS 10.04 users are unable to compile out of the box; will need to manually install later version of CMake.

Also needed is zlib1g-dev, rpmbuild.

#### While compiling

I ran out of disk space. What? I was using a 10G partition I use for my compilations. Looking at “df -h” I get that:

- The source tarball is extracted (I did it)
- The binary tarball is also extracted (someone has to explain this for me)
- And inside the source directory we have:

>     bash$ df -h
>     ...
>     1484    build.RelWithDebInfo.rpms
>     5540    build.RelWithDebInfo

At about 7GB (and counting) of build… stuff?.

UPDATE: just ran out on disk space again. Is this an incremental thing? Like every time my compilation fails and I recompile some files are not cleaned up? If so, put them on /tmp! OK, moving everything to a 300GB partition and starting all over.

#### More while compiling

I got errors on missing libraries. Like I was missing libssl, rpmbuild. This is what the “configure” script is for — to test for dependencies. It’s really a bummer to have to recompile 4-5 times (and it’s a long compilation), only to find out there’s another missing package.

#### After compiling

What is the result of the compilation? Not a “make install” prepared binary. The result is a MySQL-binary package. Se need to extract and put on /usr/local/somewhere etc.

#### Conclusions

The compilation process is unexpected and non-standard. The output is unexpected.

The correct way of doing this is a “./configure && make && sudo make install”. I don’t understand the need for a binary package while compiling from source. Isn’t this the chicken and the egg?

A source distribution is no different from a binary distribution. You must have a testing environment to verify the source distribution actually works. This test environment is typically a bare-new-RedHat or a bare-new-Ubuntu etc. The machines at Tokutek are already installed with needed packages. Not so on my compilation machine. I suggest that apt-gets and yum installs for dependencies are added to the source distribution testing. This is the only reliable way for you guys at Tokutek to know that clients will actually be able to install via source.

- Twitter
- LinkedIn
- WhatsApp
- Email
- Facebook
- Print

###
14
Leave a Reply

11 Comment threads
3 Thread replies
0 Followers
Most reacted comment
Hottest comment thread
6 Comment authors
Recent comment authors

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe
Notify of
Guest
Rich Prohaska

Hello Shlomi,
Thanks for the feedback on the build process. Sorry for the unneeded complexity. We are working to make this much simpler in the next point release.

Guest
Bradley C. Kuszmaul

Schlomi, I hate to be critical, but this whole blog posting smells like pilot error. I don’t think it makes much sense to start out knowing that we support only build from git, and then to complain that the old closed-source way of compiling doesn’t work. Yes the old closed-source way needed the fractal tree library binary, because you couldn’t build it from source. No you shouldn’t use that any more. As Leif told you. Siince mysql uses cmake, why would we add autotools? We clearly state our build dependencies in the README and Leif apparently pointed them out to… Read more »

Author
shlomi

Hi Bradley, Very good that you’re being critical. Here are a few thoughts: – I’m perfectly OK to define this as a pilot error: but this is my pilot error, and might be the next person’s pilot error. I’ve used the online documentation to work out the build process; the new compilation method is not described in the online documentation, so this is what I had to work with. – Sure, I don’t need to build RPMs; is there a “–skip-rpm” flag to pass so that the compilation process does not terminate with error? Since the compilation did exit with… Read more »

Guest
Leif Walsh

Hi Shlomi, Thanks for the feedback so far. I think the compilation process will become clearer soon. For now, the best documentation is in the readmes in the github repo: http://github.com/Tokutek/ft-engine/blob/master/README.md . This describes building using the make.mysql.bash script, which is what the rest of my advice applies to. The script has a “–build_rpm=0” parameter, which is now the default. This will skip building rpms. Running on Ubuntu 10.04 is certainly fine, and it should work. Building without a C++11 compiler or a recent cmake is not going to work though, sorry. We describe the build requirements in the readme,… Read more »

Author
shlomi

Hi Leif, With regard “make install” — great, but I never got to see a Makefile in the installation process. There is none. There are plenty Makefile instances in inner directories, but no one Makefile for the entire project. Of course if you generate a Makefile than a “make install” is wonderful. That’s what I was suggesting myself: to have “./configure && make && make install” (the last can be skipped). Thanks for the link! Will look more closely into it. Question: since it builds directly from Git, does that mean I get a source code which is modified constantly?… Read more »

Guest
Leif Walsh

“./configure” is the way Autotools works. MySQL and MariaDB don’t use Autotools, they use CMake instead. You should read up on what CMake does, but it’s basically the same as “./configure”, and it generates Makefiles (but it generates them in a separate build directory, which is often very useful, you can read up on “cmake out-of-source builds”). If you go into this build directory you can type “make install” there.

You should pass “–git_tag=tokudb-7.0.1” as described in the README. This will fetch a specific version for you.

Author
shlomi

Thank you,
I will follow up on this.

Guest
Alexei

Hi, how did you fix this?

> CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
> Please set them or make sure they are set and tested correctly in the CMake files:
> TOKUFRACTALTREE_LIB
> linked by target “tokudb” in directory /home/ft-engine/mariadb/storage/tokudb
> TOKUPORTABILITY_LIB
> linked by target “tokudb” in directory /home/ft-engine/mariadb/storage/tokudb

> — Configuring incomplete, errors occurred!

Author
shlomi

Alexei,
Looks to me as if you just tried compiling on your own. You need to copy bash compilation script and execute it (described in post)

Guest
Alexei

I did not compiled by myself, just executed script:

scripts/make.mysql.bash –git_tag=tokudb-7.0.1 –mysql=mariadb-5.5.30 –cc=gcc –cxx=g++

I saw your comment with the same error on mysqlperformanceblog. Thanks anywasy for your answer!

Author
shlomi

Hrm… So my comment on mysqlperformanceblog related to myself trying to do the normal “./configure && make” from the Percona build tarball.
I haven’t tackled the error you’re describing with the bash script. However, I see you are using the “new method” of building the source, whereas I was using the “old method”; haven’t tried the new one as yet.

Author
shlomi

@Alexei,
Now trying this for myself, I get same error as yours — did you solve this?

Guest
Alexei

@shlomi I have read article “Benchmarking Percona Server TokuDB vs InnoDB” on Mysqlperformanceblog and decided not to waste my time playing with TokuDB installation and decided to remain on PerconaDB 🙂

Guest
Suresh

This is two years after the actual article date and TokuDB still has the same compilation issues. I thought of running benchmarks on different index and picked Tokudb too. Unfortunately, the compilation and installation needs too much effort and I left after spending a few hours on it.

wpDiscuz
Send to Email Address Your Name Your Email Address Cancel
Post was not sent - check your email addresses!
Email check failed, please try again
Sorry, your blog cannot share posts by email.