Installation – code.openark.org http://shlomi-noach.github.io/blog/ Blog by Shlomi Noach Tue, 11 Sep 2012 06:49:01 +0000 en-US hourly 1 https://wordpress.org/?v=5.3.3 32412571 How common_schema installs itself https://shlomi-noach.github.io/blog/mysql/how-common_schema-installs-itself https://shlomi-noach.github.io/blog/mysql/how-common_schema-installs-itself#comments Tue, 11 Sep 2012 06:48:13 +0000 https://shlomi-noach.github.io/blog/?p=5033 Up till common_schema version 1.1, the user would need to choose from distinct distribution files: an install compatible with MySQL 5.1, one compatible with InnoDB Plugin enabled servers, and one compatible with Percona Server. The difference between the three is the availability of certain INFORMATION_SCHEMA tables.

With 1.1, this is no longer the case: common_schema auto-detects the server and available feature set, and installs accordingly.

Wait, isn’t common_schema just a SQL file?

Yes. It’s not like there’s an installer like InstallShield or anything. Nevertheless, common_schema offers a smart way of conditional handling, which is uses in itself. It’s called QueryScript.

common_schema is installed by importing the SQL file (via SOURCE command; the mysql client; your favorite GUI). This creates your usual tables, views and routines. But some of these routines make for an interpreter for QueryScript. Somewhere along the installation process (remember – it’s just a SQL import), common_schema switches over to executing scripts to manage the installation. In particular, there are a few views which depend on optional tables, such as InnoDB Plugin’s tables for INFORMATION_SCHEMA.

Consider the following CREATE VIEW statement:

CREATE OR REPLACE
ALGORITHM = UNDEFINED
SQL SECURITY INVOKER
VIEW innodb_transactions_summary AS
  SELECT 
    COUNT(*) AS count_transactions,
    IFNULL(SUM(trx_state = 'RUNNING'), 0) AS running_transactions,
    IFNULL(SUM(trx_requested_lock_id IS NOT NULL), 0) AS locked_transactions,
    COUNT(DISTINCT trx_requested_lock_id) AS distinct_locks
  FROM 
    INFORMATION_SCHEMA.INNODB_TRX
  WHERE 
    trx_mysql_thread_id != CONNECTION_ID()
;

Such a statement would fail the import if the underlying tables (INFORMATION_SCHEMA.INNODB_TRX in our example) do not actually exist. Which is why this CREATE statement is not invoked just like that. It is wrapped within a script:

set @script := "
  set @common_schema_innodb_plugin_expected := 0;
  try {
    set @common_schema_innodb_plugin_expected := @common_schema_innodb_plugin_expected + 1; 

    CREATE VIEW ... (as above)

    set @common_schema_innodb_plugin_installed := @common_schema_innodb_plugin_installed + 1;
  }
  catch {
  }
";

call run(@script);

I really want to stress this: this is a try-catch statement running from within your MySQL server. QueryScript accepts this statement and behaves very similarly to the way you are used to with your favorite programming language.

If the CREATE VIEW cannot be fulfilled, an error is generated. But instead of failing the entire script, it is caught by the catchclause; the “set @common_schema_innodb_plugin_installed := …” line is never executed upon such error.

Upon CREATE VIEW failure, @common_schema_innodb_plugin_installed falls short of @common_schema_innodb_plugin_expected and . By the end of installation process, the two are compared, so as to determine the install success:

+-----------------------------------------------------------+
| complete                                                  |
+-----------------------------------------------------------+
| - Base components: installed                              |
| - InnoDB Plugin components: installed                     |
| - Percona Server components: not installed                |
|                                                           |
| Installation complete. Thank you for using common_schema! |
+-----------------------------------------------------------+

Can’t you just use “mysql –force”?

So, one can import a SQL file using mysql –force, which does nto break down on first sight of error, but continues to next statements.

True, but:

  • I don’t want to ask that of the user.
  • It won’t give me a status on what went wrong.

By using QueryScript‘s try-catch:

  • I can take alternate action on failure.
  • common_schema can (and does) inform the user at the end of installation process which components are installed and which are not.
  • It can (and does) write this info down to a metadata table such that I can ask the user for that information when submitting a bug report.

Eat your own dog food!

]]>
https://shlomi-noach.github.io/blog/mysql/how-common_schema-installs-itself/feed 3 5033
Installing MySQL from source/binary tarball as a Linux service https://shlomi-noach.github.io/blog/mysql/installing-mysql-from-sourcebinary-tarball-as-a-linux-service https://shlomi-noach.github.io/blog/mysql/installing-mysql-from-sourcebinary-tarball-as-a-linux-service#comments Tue, 01 May 2012 08:10:10 +0000 https://shlomi-noach.github.io/blog/?p=4858 I’ve written before I prefer to do a manual install of MySQL over a repository one. I still do: I typically install from binary tarball or by compiling from source.

I’d like to share my setup procedure for Linux installation and service setup. I’ve done this dozens of times, on different Linux flavors, and it works well for me.

Installing from source

To get this straight: you sometimes have to compile the source files. I, for example, happen to use the Sphinx MySQLSE extension. You can only use it if compiled with MySQL. You had to compile a “vanilla” 5.1 version without query cache in order to completely remove the cache’s mutex contention.

Anyway, I find the easiest way is to install onto a path associated with the server version. For example, I would install a 5.5 server onto /usr/local/mysql55

This way, a new version gets its own path, and no ambiguity.

To do that, use the prefix option on configuration step:

cd /path/to/extracted/source/tarball
sh BUILD/autorun.sh
./configure --prefix=/usr/local/mysql55
make
sudo make install

Once this is complete, you have everything under /usr/local/mysql55. This means binaries, libraries, scripts, etc.

To install the MySQL server as a service, copy the mysql.server script to /etc/init.d:

sudo cp /usr/local/mysql55/support-files/mysql.server /etc/init.d/mysql55

Again, I’m naming the script after the MySQL version. This avoids conflict with possible past or future installations of the MySQL server, which typically create a service named mysql or mysqld.

A thing to note about the mysql.server script is that it allows you (at around line #45) to set two variables:

  • basedir: path to your installation directory. When compiling from source this is already setup with the path provided to the configure script. Thus, in our example, you can expect this variable to read /usr/local/mysql55. So basically nothing to do here.
  • datadir: path to your data directory. If you’re putting your my.cnf file in /etc or /etc/mysql, then setting datadir in my.cnf suffices. However, if you’re going to put my.cnf itself on the data directory (e.g. so as to avoid collisions) then make sure to set the variable in the mysql.server init script.

Depending on your $PATH configuration, it is also a good idea to specify basedir variable on your my.cnf‘s [mysqld] section.

Which leads us to $PATH: your linux system is still unaware of the many binaries you’ve got in there. I typically add the following line at the end of /etc/bash.bashrc:

export PATH=/usr/local/mysql55:${PATH}

This is the most global PATH settings one can do. Alternatively, use /etc/profile, ~/.bashrc etc. (you may have noticed by now I’m working with bash).

Finally, need to setup the init script to run at startup and stop at shutdown.

  • On Debian/Ubuntu/related I use rcconf (I’m too lazy to remember the command line setup).
  • On RedHat/CentOS/related I use chkconfig –add mysql55, or  linuxconf (since I’m lazy).

Installing from binary tarball

The only difference is that the mysql.server script is unaware of our deployment path. So the basedir variable must be set in that file. Other than that, follow same steps as for source installation (oh, of course no need to configure & make…).

]]>
https://shlomi-noach.github.io/blog/mysql/installing-mysql-from-sourcebinary-tarball-as-a-linux-service/feed 1 4858
Guide to MySQL installation files https://shlomi-noach.github.io/blog/mysql/guide-to-mysql-installation-files https://shlomi-noach.github.io/blog/mysql/guide-to-mysql-installation-files#respond Wed, 01 Dec 2010 10:27:14 +0000 https://shlomi-noach.github.io/blog/?p=3089 Even for DBAs already familiar with MySQL, the choice of installation methods and the variety of install/package files is overwhelming and confusing.

I’ll make a (very!) brief introduction to the various installation options, concentrating on the Linux operating system, and provide with a simple shopping list.

For illustration, I’ll refer to the MySQL 5.1.52 community edition, the latest at the time of this writing. Downloads are available at http://dev.mysql.com/downloads/mysql/5.1.html.

I will then refer to alternate distributions.

A note for Windows users

You are lucky: your choice is very clear. Download the Windows MSI package. Choose 32 or 64 bit architecture according to your OS. If you know about the other ways to install and setup MySQL under Windows, you don’t need this post.

Linux: repositories

Easiest way to install MySQL on linux would be to use your distro’s repository. Just:

sudo apt-get install mysql-server-5.1

or

sudo yum install MySQL-Server-5.1

Your distro should resolve any package dependencies.

I’m in the opinion that if MySQL is the main application to be used on a server, distribution’s default repositories are not the way to go. Reasons include MySQL outdated version, incapability of installing multiple instance, danger of automatic upgrades or downgrades. Read this post and discussion that follows for more.

Linux: packages

If you’re a RedHat/CentOS/SuSE user, you’re in luck: MySQL provides pre-built RPM packages for your system. I vaguely remember MySQL announcing that Ubuntu is to be supported. That was a couple years ago; there is still no package for Debian/Ubuntu.

So, assuming you’re a RedHat/CentOs user, which packages should you download?

There’s over 60 different RPMs available for download. I always need to stress my eyes to get it right. Here’s the simple answer (again, if you know better, you don’t need this post). Choose “RedHat & Oracle Enterprise Linux”; download:

  • MySQL-client-community-5.1.52-1.rhel5.x86_64.rpm
  • MySQL-server-community-5.1.52-1.rhel5.x86_64.rpm
  • MySQL-shared-community-5.1.52-1.rhel5.x86_64.rpm
  • MySQL-shared-compat-5.1.52-1.rhel5.x86_64.rpm

The above assumes a RedHat/CentOS 5.x and a 64 bit Intel/AMD processor.

The aforementioned post relays my opinion of using RPMs; these are still susceptible to yum‘s whims. Be careful.

Linux: binary

Not afraid to install by hand? Want to avoid limitations introduced by pre-built packages? Download a binary distribution:

  • Choose “Linux – Generic”. Downlaod mysql-5.1.52-linux-x86_64-glibc23.tar.gz

This tar.gz distribution includes server & client. It includes glibc so it does not depend on your OS installed glibc version (a pain to upgrade/downgrade as it is used by so many packages).

You may still want to download and install the shared-compat RPM package (see previous section) to have all possible libmysqlclientX.X packages installed.

Linux: source

If you got here, then you either know your way around (why do you keep reading?) or you have a good reason to use a source distribution.

What good reason could that be?

A more and more common reason is that you want to add something to MySQL. Sphinx search storage engine is such a common addition.

  • Choose “Source Code”: download mysql-5.1.52.tar.gz (Architecture Independent).

Non MySQL downloads

You don’t have to download the official MySQL distribution. Two good alternatives are:

]]>
https://shlomi-noach.github.io/blog/mysql/guide-to-mysql-installation-files/feed 0 3089
To not yum or to not apt-get? https://shlomi-noach.github.io/blog/mysql/to-not-yum-or-to-not-apt-get https://shlomi-noach.github.io/blog/mysql/to-not-yum-or-to-not-apt-get#comments Tue, 16 Feb 2010 11:44:25 +0000 https://shlomi-noach.github.io/blog/?p=1776 I’ve written shortly on this before. I like yum; I love apt-get; I prefer not to use them for MySQL installations. I consider a binary tarball to be the best MySQL installation format (source installations being a different case altogether).

Why?

I use yum and apt-get whenever I can and for almost all needs (sometimes preferring CPAN for Perl installations). But on a MySQL machine, I avoid doing so. The reason is either dependency hell or dependency mismatch.

Package managers are supposed to solve the dependency hell issue. But package managers will rarely have an up to date MySQL version.

I’ve had several experiences where a simple yum installation re-installed the MySQL version. I’ve had customers calling me up when, having installed something with yum, MySQL would not work anymore.

yum install package-which-depends-on-mysql-server will install MySQL server on your system if it hasn’t been installed with yum. Are you on CentOS 5.0? You’ll get MySQL 5.0.22. Oh, did you already have a RPM installation for MySQL 5.0.81? Sorry – it’s just been downgraded, plus it won’t work anymore since the error messages file has been changed since then.

Don’t press ‘Y‘ too soon!

Things are slightly better with apt-get. I’ve encountered less situations where mysql-server was on the dependency list. Many times it’s just the libmysqlclient package or the mysql-common one.

But wait! Did you install mysql-common? Bonus! You get the elusive /etc/mysql/my.cnf file created, and there goes your server configuration. Future spawns of the MySQL server / clients will read from the wrong configuration file, and will probably fail to load.

Not to mention neither will help you out with multiple instances installation.

My argument

A sys admin recently argued with me that it was wrong of me to have the entire machine set up with yum, but have MySQL installed with binary tarball. He argued that it broke the entire setup. I expressed my opinion: on a MySQL dedicated server, MySQL gets to be prioritized. It’s special. It is the reason for the existence of the machine. I would imagine that same would hold for Apache on an Apache dedicated machine, for Sendmail on a Sendmail dedicated machine, etc. As a DBA, I want to have best control of the MySQL installation; I want to be able to upgrade minor versions quickly: I often find newer versions to solve bugs I was concerned with; I want to be able to install multiple instances; I want to be able to downgrade without having to remove and uninstall the previous version.

I want to have control. World domination aside, that is.

]]>
https://shlomi-noach.github.io/blog/mysql/to-not-yum-or-to-not-apt-get/feed 26 1776
SphinxSE 0.9.9-RC2 bug workaround https://shlomi-noach.github.io/blog/mysql/sphinxse-0-9-9-rc2-bug-workaround https://shlomi-noach.github.io/blog/mysql/sphinxse-0-9-9-rc2-bug-workaround#respond Mon, 07 Sep 2009 08:23:21 +0000 https://shlomi-noach.github.io/blog/?p=1245 There is a serious bug with the sphinx storage engine, introduced in 0.9.9-RC2 (and which has not been fixed in latest revisions, as yet – last checked with rev 2006).

I would usually just revert to an older version (0.9.9-RC1 does not contain this bug), but for the reason that RC2 introduces an important feature: the sphinx_snippets() function, which allows for creation of snippets from within MySQL, and which makes the sphinx integration with MySQL complete, as far as the application is concerned.

The bug

The bug is described here and here (and see further discussions). Though it’s claimed to have been fixed, it’s been re-reported, and I’ve tried quite a few revisions and verified it has not been fixed (tested on Debian/Ubuntu x64). Essentially, the bug does not allow you to set filters on a query issued from within the SphinxSE. For example, the following queries fail:

SELECT ... FROM ... WHERE query='python;mode=any;sort=relevance;limit=200;range=myUnixTimestamp,1249506000,1252184400;'
SELECT ... FROM ... WHERE query='python;mode=any;sort=relevance;limit=200;filter=my_field,1;'

While the following query succeeds:

SELECT ... FROM ... WHERE query='python;mode=any;sort=relevance;limit=200;'

The error message is this:

ERROR 1430 (HY000): There was a problem processing the query on the foreign data source. Data source error: searchd error: invalid or truncated request

I see this as a serious bug in the SphinxSE: it renders it useless; searching without the ability to filter is not something I can live with.

The motivation

Sphinx does not store the actual text content. To get search results with snippets, you need to:

  • Ask sphinx for the documents ids
  • Get the content for those documents
  • Ask sphinx for snippets based on the provided content and search phrase.

With the introduction of the sphinx_snippets() function, this can all be done with a single query, like this:

SELECT my_docs.my_docs_id,  my_docs.publish_time,  CONVERT(sphinx_snippets(my_docs.id.content, 'my_docs_index', 'python') USING utf8) AS snippet  FROM tets.my_docs INNER JOIN test.my_docs_sphinx USING(my_docs_id) WHERE query='python;mode=any;sort=relevance;limit=200;range=publish_time_unix,1249506000,1252184400;';

This is really a life saver; without this function, you need to get the resutls back to your application, then send the data again to MySQL, in which case you might altogether discard the SphinxSE and talk to sphinx directly. But with a single query you get to ask the results just as if you were asking for any result set from your database (with extra syntax).

The workaround

My setup is Percona’s mysql-5.1.34-xtradb5 source, on Ubuntu server 8.04 amd64. The trick is to first compile MySQL with sphinx 0.9.9-RC2, in order to produce the sphinx.so file (where the sphinx_snippets() function is found), backup the sphinx.so file, then recompile everything with sphinx 0.9.9-RC1. The steps being:

Compile MySQL with sphinx 0.9.9-Rc2 (I choose to install MySQL on /usr/local/mysql51):

tar xzfv mysql-5.1.34-xtradb5.tar.gz
cd mysql-5.1.34-xtradb5
cp -R /tmp/resources/sphinx-0.9.9-rc2/mysqlse storage/sphinx
sh BUILD/autorun.sh
./configure --with-plugins=innobase,sphinx --prefix=/usr/local/mysql51
make

This produces the sphinx.so, sphinx.so.0, sphinx.so.0.0.0 files. Back them up!

Next, recompile with sphinx 0.9.9-RC1. I’ve found that simple copying and recompiling doesn’t work well. So just cleanup everything and start afresh:

cd ..
rm -rf mysql-5.1.34-xtradb5
tar xzfv mysql-5.1.34-xtradb5.tar.gz
cd mysql-5.1.34-xtradb5
cp -R /tmp/resources/sphinx-0.9.9-rc1/mysqlse storage/sphinx
sh BUILD/autorun.sh
./configure --with-plugins=innobase,sphinx --prefix=/usr/local/mysql51
make
sudo make install

Copy the sphinx.so files into the MySQL plugin directory (/usr/local/mysql51/lib/mysql/plugin in our case).

Then build sphinx (you must have MySQL includes for sphinx to compile, so this must be the second step):

cd /tmp/resources/sphinx-0.9.9-rc1/
./configure --prefix=/usr/local/sphinx --with-mysql=/usr/local/mysql51
make
sudo make install

Essentially, we’re working now with 0.9.9-RC1, but the sphinx_snippets() function is from the 0.9.9-RC2 version, and happily no one bothers about this mix.

I hope this helps.

]]>
https://shlomi-noach.github.io/blog/mysql/sphinxse-0-9-9-rc2-bug-workaround/feed 0 1245
Manually installing multiple MySQL instances on Linux: HOWTO https://shlomi-noach.github.io/blog/mysql/manually-installing-multiple-mysql-instances-on-linux-howto https://shlomi-noach.github.io/blog/mysql/manually-installing-multiple-mysql-instances-on-linux-howto#comments Thu, 19 Feb 2009 06:02:00 +0000 https://shlomi-noach.github.io/blog/?p=531 Installing a single MySQL instance on a linux machine is a very simple operation. It may be as simple as:

apt-get install mysql-server

But you cannot use this method to install another MySQL instance. Moreover, if you try to manually install another instance, you may find that some collisions occur.

For example, when trying to install two 5.0 servers, apt-get or yum will just tell me “package is already installed”.

When trying to install a new 5.1 server along with 5.0, an implicit upgrade is performed.

But even if we try forcing the installation, there are collisions:

  • A dpkg or rpm will install my.cnf under /etc. Two installations will override one another. With RPM you may get a .rpmsave backup file, but that doesn’t help too much.
  • The daemon file: /etc/init.d/mysql(d) is overwritten.
  • The default data directory is used for both installations: /var/lib/mysql
  • The binaries are overwritten
  • Both installations will use port 3306.
  • In both installations, the same socket file (e.g. /var/run/mysql/mysql.sock) is used.

Interestingly, on Windows, multiple MySQL installations are by far easier:

  • Binaries are under Program Files\\MySQL\\MySQLX.X. With two installations, you specify different directories.
  • Data files are by default directly under the installations paths (MySQL 5.0) or under “Documents And Settings…” (MySQL 5.1) with no collisions.
  • The my.ini files are located directly under the installation paths.
  • The installer asks you for a service name, and notifies you if that name is already in use.
  • The installer let’s you know if port 3306 is already taken, and allows you to specify another one.
  • Of course, there’s no unix socket file.

I usually install MySQL on Linux using the binary tarball. When there’s only one instance expected, I go with the standards: my.cnf is in /etc, mysqld is under /etc/init.d, etc. (no pun intended)

Steps for multiple installation on Linux

When more than one installation is expected, here’s a safe way to ensure no collisions occur. We will assume a 5.0 and 5.1 installation (say we want to upgrade):

Install the MySQL binaries under /usr/local

Following the INSTALL document file, we make symbolic links to the full path in the names

ln -s /usr/local/your-mysql-5.0-full-installation-path /usr/local/mysql50
ln -s /usr/local/your-mysql-5.1-full-installation-path /usr/local/mysql51

Do not put my.cnf under /etc

Instead, put them directly in the installation path:

touch /usr/local/mysql50/my.cnf
touch /usr/local/mysql51/my.cnf

Setup different port numbers in the my.cnf files

For example, in /usr/local/mysql50/my.cnf, use port 3350:

[mysql]
port=3350

[mysqld]
port=3350

Choose another port (e.g. 3351) for the 5.1 installation, then have it written as above in the 5.1 my.cnf file.

Choose distinct socket files

For example, in /usr/local/mysql50/my.cnf, add:

[mysql]
port=3350
socket=/tmp/mysql50.sock

[mysqld]
port=3350
socket=/tmp/mysql50.sock

Choose another socket and set it up in the second my.cnf file. You may also choose to put the socket files under the data paths or installation paths.

Choose distinct data paths

Either do not specify them at all, in which case they will reside under the installation path, or, if you want to enjoy another partition, use:

[mysql]
port=3350
socket=/tmp/mysql50.sock

[mysqld]
port=3350
socket=/tmp/mysql50.sock
datadir=/my_raid_path/mysql50/

Create distinct daemons

Manually copy support_files/mysql.server to /etc/init.d under distinct names. For example:

cp /usr/local/mysql50/support_files/mysql.server /etc/init.d/mysqld50
cp /usr/local/mysql51/support_files/mysql.server /etc/init.d/mysqld51

Other settings

You may wish to set up a soft link for the client binaries, for example:

ln -s /usr/local/mysql50/bin/mysql /usr/bin/mysql50

chkconfig (RedHat and derived) can be used to start/stop daemon as service:

chkconfig --add mysqld50

Conclusion

I would prefer MySQL to come bundled in self-contained directory. The tarball is almost that, except it expects socket file to be on /tmp, and by default uses the 3306 port. I would further like to have a dpkg-reconfigure script to setup the above issues.

Till then, it’s manual configuration.

]]>
https://shlomi-noach.github.io/blog/mysql/manually-installing-multiple-mysql-instances-on-linux-howto/feed 28 531
Do we need sql_mode? https://shlomi-noach.github.io/blog/mysql/do-we-need-sql_mode https://shlomi-noach.github.io/blog/mysql/do-we-need-sql_mode#comments Thu, 08 Jan 2009 06:14:38 +0000 https://shlomi-noach.github.io/blog/?p=462 sql_mode must be one of the most elusive issues in setting up a MySQL database. It is also responsible for difficulties in migration, upgrading and securing of databases.

MySQL usually has little issues with migrating database from one machine to another, one version to another, one OS to another, one architecture to another. If we’re not sure, we can always migrate using mysqldump’s logical dump, right?

Not entirely right. The elusive sql_mode (empty by default) may affect out backup+restore, may affect our application behavior, may affect our data integrity.

  • Did we remember to set NO_AUTO_CREATE_USER? Oh dear, we have some users without passwords.
  • Did we remember to set NO_AUTO_VALUE_ON_ZERO? Oh dear, we’ve dumped our database for backup, restored, but AUTO_INCREMENT values have changed!
  • Did we set STRICT_ALL_TABLES? How do we know if the 255 value in our TINYINT really stands for 255 or if it was a truncated 299?
  • Do we allow selecting non aggregated columns in GROUP BY? Did we set ONLY_FULL_GROUP_BY? Will our application crash now?
  • Our old database has zero for empty date values (columns are NOT NULL). But what settings do we have for NO_ZERO_IN_DATE on our new installation? Will import fail?
  • And how did the NULL get in? Was it because we divided by zero, and forgot to set ERROR_FOR_DIVISION_BY_ZERO? How can we tell?

The fact is: two mysql instances, same version, same OS and architecture, with different sql modes – can be incompatible!

As said, the sql_mode is empty by default. This is very non-strict. But more than that: it can be changed even while the database is running; even on a per connection basis.

Setting sql_mode should be one of the first things to do after installation. The usual manuals talk about setting the innodb_buffer_pool_size and the query_cache_size, when sql_mode will dictate the nature of your database and application on an entirely grander scale.

I think it would be best if MySQL chooses a desired set of sql modes (like ‘TRADITIONAL’), then make it the default. I further believe it would be best if MySQL would not allow changes to sql_mode. Not globally and certainly not per session. Choosing the stricter mode is better, I believe: errors such as overflow values should be reported to the application, not just swiped under the carpet.

Backward compatibility? Indeed a problem (inherent to the very existence of sql_mode). Perhaps allow one setting per installation. Perhaps just go for it.

]]>
https://shlomi-noach.github.io/blog/mysql/do-we-need-sql_mode/feed 17 462
MySQL parameters & configuration sample file https://shlomi-noach.github.io/blog/mysql/mysql-parameters-configuration-sample-file https://shlomi-noach.github.io/blog/mysql/mysql-parameters-configuration-sample-file#comments Fri, 17 Oct 2008 13:03:33 +0000 https://shlomi-noach.github.io/blog/?p=6 The following is a sample my.cnf configuration file. It includes some common and uncommon parameters, along with occasional comments.

You can use this file, and tune as appropriate for your machine, operating system, and application needs.

In particular, note the high values set for the InnoDB related parameters, below, and change them to match your system and application’s requirements. You have been warned.

[client]
#password = [your_password]
port = 3306

# The following location is not the default location for many
# pre-packaged utilities. For example, mytop or mysqlreport
# (or any Perl or Python based client, for that matter), will
# NOT look in the /tmp path for the mysql.sock file.
# The value is set to common location by your RPM or DEB
# package on GNU/Linux, yet defaults to the following in the
# .tgz mysql-server downlaod files.
socket = /tmp/mysql.sock

default-character-set=utf8

[mysqld]

# For a new installation, best to use a strict sql_mode:

sql_mode=TRADITIONAL
datadir= [path to mysql data]
# I prefer working case-sensitive. However, some 3rd party tools
# demand case insensitivity. If you’re working with such a tool,
# uncomment the following line.
#lower_case_table_names=1

# 5-10 minutes timeout is usually more than required for idle
# connections. If a connection can be idle for so long, then
# creating a new connection is usually very cheap. This applies
# for many common web based applications. However, depending
# on your application, you may wish to set this parameter to a
# higher value.
wait_timeout=300

# The following is unfortunately still used on many instances. If
# you need backwards compatability, you may need to uncomment
# the following line.
#old_passwords=1

# generic configuration options
port = 3306
# Make sure this is the same location as in the [client] section
socket = /tmp/mysql.sock

back_log = 50
max_connect_errors = 10

# Resources:
# The following values are liberal. You may reduce them
# according to your needs.
# NOTE: verify that your OS’s open files limit is at least as
# specified here.
# I usually set ‘ulimit -n’ to allow for 8192 files to mysql.
open_files_limit = 8000
# The following is usually cheap, so go ahead and set a high value.
table_cache = 2048
# Connections take memory, so don;t be too liberal here. A common,
# well tuned application, using proper connection pools, shouldn’t
# require more than 1000 concurrent connections.
max_connections = 500

# Memory:
# When required, the following parameters will be used to allocate
# more memory. The value specified is always the amount of memory
# allocated, regardless of the real need.
binlog_cache_size = 1M
max_heap_table_size = 64M
sort_buffer_size = 8M
join_buffer_size = 8M
# The following value sets an upper limit only: MySQL will only
# allocate as much as required. Setting this parameters to a high
# value may be required if you’re handling very large statement,
# such as queries reading/writing BLOBs.
max_allowed_packet = 16M

# Even on a very busy server, a thread cache of 32 threads is
# usually enough to make for good thread recycling. I rarely go
# above this number.
thread_cache_size = 32
thread_concurrency = 8

# Query cache:
# Make sure you really need the query cache. I’ve seen many
# applications where the query cache almost utterly unused
# (due to frequent invalidation). Check for your hit:insert ratio.
# A query cache type of value “2” (DEMAND) is a more programmatic
# oriented setting, where most queries are not cached, but a few,
# recognized to benefit from the query cache, are executed with
# the SQL_CACHE flag.
query_cache_type = 1
query_cache_size = 32M
query_cache_limit = 2M

ft_min_word_len = 4
# I usually go with InnoDB, which is ACID compliant and crash-safe.
default_table_type = InnoDB

thread_stack = 192K

transaction_isolation = REPEATABLE-READ
tmp_table_size = 64M

# Logs
# Having log-bin enabled allows for replication and for point in time
# recovery. It’s a good idea to always have binary logs at hand.
# binary logs can also be reviewed to see what kind of modifications
# are made on your database (the general query log also logs
# SELECTs and other queries, which do not appear in the binary log)
# I usually name the logs after my host name. When dealing with
# replication, having master and slave logs named after the machines
# on which they are running makes for clearer distinction between
# them.
log-bin=mymachine-bin
relay-log = mymachine-relay-bin
relay-log-index = mymachine-relay-bin.index
# Since a slave can suddenly be promoted to Master’s role, I always
# take care that it also has the binary logs prepared.
log_slave_updates
log-error=mymachine-mysql-error.log
log_slow_queries
long_query_time = 2
log-queries-not-using-indexes
# Automatically remove binary logs. Make sure this number is not too
# high and not too low for you, depending on your backups frequency
# and maximum slave lag.
expire_logs_days=7

# Replication related settings:
# Unique server identification number between 1 and 2^32-1.
server-id = [a unique number]
# Setting the report_host allows for SHOW SLAVE HOSTS on the master
# to see which slaves are connected.
report_host=[machine’s ip]
# Limit the total size of the relay logs: set this value to a high enough
# value. When the slave can’t keep up with the master, the relay logs
# fill up. If no limit is set, all disk space can be consumed.
# If a value too low is set, the master’s binary logs cannot (shouldn’t)
# be purged, and so the master gets to have a disk space problem. So
# provide a high enough value, and monitor your disk space!
relay_log_space_limit=10G

# MyISAM:
key_buffer_size = 32M
read_buffer_size = 2M
read_rnd_buffer_size = 16M
bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 10G
myisam_max_extra_sort_file_size = 10G
myisam_repair_threads = 1
myisam_recover = force,backup

concurrent_insert=2

# INNODB:

innodb_file_per_table
# Your machine’s memory capacity dictates many of the following
# values. ***Do NOT take these values as they are. Change them***
innodb_buffer_pool_size = 10G
innodb_additional_mem_pool_size = 16M
innodb_data_file_path = ibdata1:50M:autoextend
innodb_file_io_threads = 4
# Keep the following one commented, unless you need to recover
# from disaster.
#innodb_force_recovery=4
innodb_thread_concurrency = 8
# Set “1” for full ACID compliance. Set “2” for IO performance gain
# (will only flush transaction log to disk once per second, instead
# of at each commit). If you have battery backed disk cache ,set
# to “1” and enjoy both worlds.
innodb_flush_log_at_trx_commit = 1
# Keep the following one commented, unless you really have an
# emergency. Fast shutdown makes for slower start up time.
#innodb_fast_shutdown
innodb_log_buffer_size = 8M
# The default here is 5M, which is really too low for our modern
# machines.
innodb_log_file_size = 128M
innodb_log_files_in_group = 2
innodb_max_dirty_pages_pct = 90
# Consult The manual. The desired values for the following
# parameter change according to your OS type, OS version and
# hardware types. Benchmark after changing it!
#innodb_flush_method=O_DIRECT
innodb_lock_wait_timeout = 120

[mysqldump]
quick
max_allowed_packet = 64M

default-character-set=utf8

]]>
https://shlomi-noach.github.io/blog/mysql/mysql-parameters-configuration-sample-file/feed 1 6