{"id":4002,"date":"2011-12-02T06:59:03","date_gmt":"2011-12-02T04:59:03","guid":{"rendered":"http:\/\/code.openark.org\/blog\/?p=4002"},"modified":"2011-12-02T13:44:06","modified_gmt":"2011-12-02T11:44:06","slug":"mysql-foreach","status":"publish","type":"post","link":"https:\/\/code.openark.org\/blog\/mysql\/mysql-foreach","title":{"rendered":"MySQL foreach()"},"content":{"rendered":"<p>A new routine is now available in <a href=\"http:\/\/code.google.com\/p\/common-schema\/\" rel=\"nofollow\">common_schema<\/a>, which makes for an easier execution syntax for some operations:<\/p>\n<blockquote>\n<pre>foreach(<span style=\"color: #808000;\"><em>collection_to_iterate_over<\/em><\/span>, <span style=\"color: #003366;\"><em>queries_to_execute_per_iteration_step<\/em><\/span>);<\/pre>\n<\/blockquote>\n<p>To illustrate what it can do, consider:<\/p>\n<blockquote>\n<pre>call foreach(<span style=\"color: #808000;\">'table in sakila'<\/span>, <span style=\"color: #000080;\">'ALTER TABLE <strong>${schema}<\/strong>.<strong>${table}<\/strong> ENGINE=InnoDB ROW_FORMAT=COMPACT'<\/span>);\r\n\r\ncall $(<span style=\"color: #808000;\">'schema like shard_%<\/span>', <span style=\"color: #000080;\">'CREATE TABLE <strong>${schema}<\/strong>.messages (id INT)'<\/span>);\r\n\r\ncall $(<span style=\"color: #808000;\">'2000:2009'<\/span>, <span style=\"color: #000080;\">'INSERT IGNORE INTO report (report_year) VALUES (<strong>${1}<\/strong>)'<\/span>);<\/pre>\n<\/blockquote>\n<p><strong>$()<\/strong> stands as a synonym to <em>foreach()<\/em>. I suspect it should look familiar to web programmers.<\/p>\n<p>The idea for <em>foreach()<\/em> was introduced by Giuseppe Maxia during a correspondence. At first I was skeptic: this isn&#8217;t <a href=\"http:\/\/api.jquery.com\/jQuery.each\/\">jQuery<\/a>; this is SQL. Why would I want to use <em>foreach()<\/em>?<\/p>\n<p>Then Giuseppe provided some use cases, and as I started thinking about it, I found more and more cases where such a tool might considerably reduce scripting overhead and avoid requiring SQL-fu skills. In fact, I have been using it myself for the past few weeks<\/p>\n<p>I provide examples which I hope will convince the reader as for the simplicity of using such syntax. Showing off the types of input <em>foreach()<\/em> accepts (query, table search, schema search, set of constants, single or double numbers range), and the types of queries it can execute (single, multiple, using placeholders).<\/p>\n<p>I stress that this is not a replacement for common queries (i.e. does <em>not<\/em> come to replace your common <strong>SELECT<\/strong>, <strong>UPDATE<\/strong>, <strong>DELETE<\/strong>), but more for working out administrative tasks. Nevertheless, the last example in this post does provide with an interesting insight on possible &#8220;normal&#8221; use.<!--more--><\/p>\n<h4>Use case: using values from query<\/h4>\n<p>Let&#8217;s kill all queries running for over <strong>20<\/strong> seconds:<\/p>\n<blockquote>\n<pre>call foreach(<span style=\"color: #808000;\">'SELECT id FROM INFORMATION_SCHEMA.PROCESSLIST WHERE time &gt; 20'<\/span>, <span style=\"color: #000080;\">'KILL QUERY <strong>${1}<\/strong>'<\/span>);<\/pre>\n<\/blockquote>\n<p>The thing I like most about <em>foreach()<\/em> is that it&#8217;s self explanatory. Nevertheless, I note:<\/p>\n<ul>\n<li>The <strong>KILL<\/strong> command is executed for each process running for more than <strong>20<\/strong> seconds (I did round up corners, since I didn&#8217;t check for sleeping processes, for simplicity).<\/li>\n<li>I also use the <strong>${1}<\/strong> placeholder: much like in <em>awk<\/em>, this will get the first column in the result set. In our case, it is the single column, <strong>id<\/strong>.<\/li>\n<li>I chose to invoke a single query\/command per iteration step.<\/li>\n<\/ul>\n<p>Compare the above with another solution to the same problem, using <a href=\"http:\/\/code.openark.org\/blog\/mysql\/mysql-eval\">eval()<\/a>:<\/p>\n<blockquote>\n<pre>call eval('SELECT CONCAT(\\'KILL QUERY \\',id) FROM INFORMATION_SCHEMA.PROCESSLIST WHERE TIME &gt; 20');<\/pre>\n<\/blockquote>\n<p>They both get the same thing done. But <em>foreach()<\/em> is just a bit more friendly to write (and read).<\/p>\n<p>Let&#8217;s move to a more complicated example.<\/p>\n<h4>Use case: using multiple values from a query, invoking multiple commands<\/h4>\n<p>Let&#8217;s kill some queries, as above, but also write down a log entry so that we know what happened:<\/p>\n<blockquote>\n<pre>call foreach(\r\n  <span style=\"color: #808000;\">'SELECT id, user FROM INFORMATION_SCHEMA.PROCESSLIST WHERE time &gt; 20'<\/span>,\r\n  <span style=\"color: #000080;\">'KILL QUERY <strong>${1}<\/strong>; INSERT INTO my_log VALUES (\\'have just killed query <strong>${1}<\/strong>, executed by <strong>${2}<\/strong>\\');'<\/span>)\r\n;<\/pre>\n<\/blockquote>\n<p>In the above, for each long running process, we:<\/p>\n<ul>\n<li>Kill the process&#8217; query. <strong>id<\/strong> being the first column, is referenced by <strong>${1}<\/strong>.<\/li>\n<li>INSERT to my_log that said process has been killed. We note both <strong>id<\/strong> and <strong>user<\/strong> using placeholders <strong>${1}<\/strong> and <strong>${2}<\/strong>, respectively.<\/li>\n<\/ul>\n<p>It&#8217;s possible to invoke as many queries\/commands per iteration step. It is possible to use placeholders <strong>${1}<\/strong> through <strong>${9}<\/strong>, as well as <strong>${NR}<\/strong>, which works as in <em>awk<\/em>: it is a row-counter, <strong>1<\/strong>-based.<\/p>\n<p>This example can still be written with <em>eval()<\/em>, but in much uglier form. I can&#8217;t just first <strong>KILL<\/strong> the processes, then log about them, since by the time I want to log, the queries will not be running; the commands <em>must be coupled<\/em>. This is naturally done with <em>foreach()<\/em>.<\/p>\n<h4>Use case: iterating constant values, invoking DDL<\/h4>\n<p>The commands invoked by <em>foreach()<\/em> can take the form of DML (<strong>INSERT<\/strong>\/<strong>UPDATE<\/strong>\/&#8230;), DDL (<strong>CREATE<\/strong>\/<strong>ALTER<\/strong>\/&#8230;) or other (<strong>KILL<\/strong>\/<strong>SET<\/strong>\/&#8230;). The placeholders can be used anywhere within the text.<\/p>\n<p>Take an installation where different schemata have the same exact table structure. We want to refactor a table on all schemata:<\/p>\n<blockquote>\n<pre>call $(<span style=\"color: #808000;\">'<strong>{USA, UK, Japan, NZ}<\/strong>'<\/span>, <span style=\"color: #000080;\">'ALTER TABLE db_region_<strong>${1}<\/strong>.customer ADD COLUMN personal_notes VARCHAR(4096) CHARSET utf8'<\/span>);<\/pre>\n<\/blockquote>\n<p>The above translates to the following commands:<\/p>\n<blockquote>\n<pre>ALTER TABLE <strong>db_region_USA<\/strong>.customer ADD COLUMN personal_notes VARCHAR(4096) CHARSET utf8;\r\nALTER TABLE <strong>db_region_UK<\/strong>.customer ADD COLUMN personal_notes VARCHAR(4096) CHARSET utf8;\r\nALTER TABLE <strong>db_region_Japan<\/strong>.customer ADD COLUMN personal_notes VARCHAR(4096) CHARSET utf8;\r\nALTER TABLE <strong>db_region_NZ<\/strong>.customer ADD COLUMN personal_notes VARCHAR(4096) CHARSET utf8;<\/pre>\n<\/blockquote>\n<p>In the above, we:<\/p>\n<ul>\n<li>Provide a list of constant values. These can be strings, numbers, whatever. They are space delimited.<\/li>\n<li>Use the <strong>${1}<\/strong> place holder. We can also use <strong>${NR}<\/strong>.<\/li>\n<\/ul>\n<h4>Use case: loop through number sequence<\/h4>\n<p>We wish to populate a table with values:<\/p>\n<blockquote>\n<pre>call foreach(<span style=\"color: #808000;\">'<strong>1970:2038<\/strong>'<\/span>, <span style=\"color: #003366;\">'INSERT INTO test.test_dates (dt) VALUES (DATE(\\'<strong>${1}<\/strong>-01-01\\'))'<\/span>);<\/pre>\n<\/blockquote>\n<p>The above results with:<\/p>\n<blockquote>\n<pre>mysql&gt; SELECT dt FROM test_dates;\r\n+------------+\r\n| dt\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 |\r\n+------------+\r\n| 1970-01-01 |\r\n| 1971-01-01 |\r\n| 1972-01-01 |\r\n| 1973-01-01 |\r\n| 1974-01-01 |\r\n...\r\n| 2036-01-01 |\r\n| 2037-01-01 |\r\n| 2038-01-01 |\r\n+------------+<\/pre>\n<\/blockquote>\n<p>With numbers range:<\/p>\n<ul>\n<li>Integers are assumed<\/li>\n<li>Range is indicated by low and high values, both inclusive<\/li>\n<li>Negatives allowed (e.g. <strong>&#8216;-5:5&#8217;<\/strong>, resulting with <strong>11<\/strong> steps)<\/li>\n<li>Placeholders <strong>${1}<\/strong> and <strong>${NR}<\/strong> are allowed.<\/li>\n<\/ul>\n<h4>Use case: iterating through two dimensional numbers range:<\/h4>\n<p>We use <strong>3<\/strong> template tables; we create <strong>15<\/strong> schemata; in each we create <strong>3<\/strong> tables based on the template tables:<\/p>\n<blockquote>\n<pre>call foreach(<span style=\"color: #808000;\"><strong> '1:15,1:3'<\/strong><\/span>,\r\n  <span style=\"color: #003366;\">'CREATE DATABASE IF NOT EXISTS db_test_${1}; CREATE TABLE db_test_${1}.tbl_${2} LIKE db_template.my_table_${2};'<\/span>\r\n);<\/pre>\n<\/blockquote>\n<p>Notes:<\/p>\n<ul>\n<li>Each of the number ranges has the same restrictions and properties as listed above (integers, inclusive, ascending)<\/li>\n<li>We can now use <strong>${1}<\/strong> and <strong>${2}<\/strong> placeholders, noting the first and second numbers range, respectively.<\/li>\n<li>We may also use <strong>${NR}<\/strong>, which, in this case, will run <strong>1<\/strong> through <strong>45<\/strong> (<strong>15<\/strong> times <strong>3<\/strong>).<\/li>\n<li>We use multiple queries per iteration step.<\/li>\n<\/ul>\n<h4>Use case: overcoming MySQL limitations<\/h4>\n<p>MySQL does not support <strong>ORDER BY<\/strong> &amp; <strong>LIMIT<\/strong> in multi-table <strong>UPDATE<\/strong> and <strong>DELETE<\/strong> statements (as noted <a href=\"http:\/\/code.openark.org\/blog\/mysql\/three-wishes-for-a-new-year\">last year<\/a>). So we <em>cannot<\/em>:<\/p>\n<blockquote>\n<pre>DELETE FROM t1 USING t1 JOIN t2 ON (...) JOIN t3 ON (..) WHERE x = 7 ORDER BY ts LIMIT 100;<\/pre>\n<\/blockquote>\n<p>However, we <em>can<\/em>:<\/p>\n<blockquote>\n<pre>call foreach(\r\n  <span style=\"color: #808000;\">'SELECT t1.id FROM t1 JOIN t2 ON (...) JOIN t3 ON (..) WHERE x = 7 ORDER BY ts LIMIT 100'<\/span>,\r\n  <span style=\"color: #003366;\">'DELETE FROM t1 WHERE id = ${1}'<\/span>\r\n);<\/pre>\n<\/blockquote>\n<p>Of course, it will do a lot of single row <strong>DELETE<\/strong>s. There are further MySQL limitations which complicate things if I want to overcome this. Perhaps at a later blog post.<\/p>\n<h4>Acknowledgements<\/h4>\n<p>I hit a weird <a href=\"http:\/\/bugs.mysql.com\/bug.php?id=62406\">bug<\/a> which prevented me from releasing this earlier on. Actually it&#8217;s a duplicate of <a href=\"http:\/\/bugs.mysql.com\/bug.php?id=12257\">this bug<\/a>, which makes it <strong>6<\/strong> years old. Hurray.<\/p>\n<p>To the rescue came <a href=\"http:\/\/rpbouman.blogspot.com\/\">Roland Bouman<\/a>, who suggested an idea so crazy even I was skeptic: to parse and modify the original query so as to rename column names according to my scheme. And of course he made it happen, along with some additional very useful stuff. It&#8217;s really a <em>super-ultra-meta-meta-sql-fu<\/em> magic he does there.<\/p>\n<p>So, thanks, Roland, for joining the ride, and thanks, Giuseppe, for testing and helping out to shape this functionality. It&#8217;s great fun working with other people on open-source &#8212; a new experience for me.<\/p>\n<h4>Continued<\/h4>\n<p>In this post I&#8217;ve covered the general-purpose iterations. There are also more specific types of iterations with <em>foreach()<\/em>. <a href=\"http:\/\/code.openark.org\/blog\/mysql\/more-mysql-foreach\">Continued next<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A new routine is now available in common_schema, which makes for an easier execution syntax for some operations: foreach(collection_to_iterate_over, queries_to_execute_per_iteration_step); To illustrate what it can do, consider: call foreach(&#8216;table in sakila&#8217;, &#8216;ALTER TABLE ${schema}.${table} ENGINE=InnoDB ROW_FORMAT=COMPACT&#8217;); call $(&#8216;schema like shard_%&#8217;, &#8216;CREATE TABLE ${schema}.messages (id INT)&#8217;); call $(&#8216;2000:2009&#8217;, &#8216;INSERT IGNORE INTO report (report_year) VALUES (${1})&#8217;); $() [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"enabled":false},"version":2}},"categories":[5],"tags":[67,74,50,21],"class_list":["post-4002","post","type-post","status-publish","format-standard","hentry","category-mysql","tag-common_schema","tag-hack","tag-scripts","tag-sql"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p2bZZp-12y","_links":{"self":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts\/4002","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/comments?post=4002"}],"version-history":[{"count":77,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts\/4002\/revisions"}],"predecessor-version":[{"id":4516,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/posts\/4002\/revisions\/4516"}],"wp:attachment":[{"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/media?parent=4002"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/categories?post=4002"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/code.openark.org\/blog\/wp-json\/wp\/v2\/tags?post=4002"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}