<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Coffee on the Keyboard &#187; cli</title>
	<atom:link href="http://coffeeonthekeyboard.com/tag/cli/feed/" rel="self" type="application/rss+xml" />
	<link>http://coffeeonthekeyboard.com</link>
	<description>by James Socol</description>
	<lastBuildDate>Mon, 06 Feb 2012 23:33:42 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.3.1</generator>
<atom:link rel="hub" href="http://pubsubhubbub.appspot.com"/>		<item>
		<title>Mistakes: Importing Data with MySQL</title>
		<link>http://coffeeonthekeyboard.com/mistakes-importing-data-with-mysql-309/</link>
		<comments>http://coffeeonthekeyboard.com/mistakes-importing-data-with-mysql-309/#comments</comments>
		<pubDate>Tue, 03 Nov 2009 01:53:00 +0000</pubDate>
		<dc:creator>James</dc:creator>
				<category><![CDATA[Articles]]></category>
		<category><![CDATA[cli]]></category>
		<category><![CDATA[command line]]></category>
		<category><![CDATA[learn from me]]></category>
		<category><![CDATA[mistake]]></category>
		<category><![CDATA[MySQL]]></category>
		<category><![CDATA[pipe viewer]]></category>
		<category><![CDATA[pv]]></category>

		<guid isPermaLink="false">http://coffeeonthekeyboard.com/?p=309</guid>
		<description><![CDATA[I blew most of the day trying to load data into MySQL because I didn't read. Hopefully my trials with 'max_allowed_packet' will help you avoid the same fate.]]></description>
			<content:encoded><![CDATA[<p>I spent the better part of today trying to import—in various attempts—between 800 MB and 3.1 GB of data into a local MySQL server. The whole time I was Doing It Wrong™. Now I feel like <a href="http://www.flickr.com/photos/zachklein/54389823/">this</a>:</p>
<p style="text-align: center;"><img class="aligncenter size-full wp-image-310" title="Image by Zach Klein" src="http://coffeeonthekeyboard.com/wp-content/uploads/2009/11/frustrated.jpg" alt="frustrated" width="240" height="160" /></p>
<p>I was using <a href="http://spindrop.us/2009/09/16/getting-started-with-pipe-viewer/">pipe viewer</a>, which obscured the error for most of the day. (If there&#8217;s a way to stop <code>pv</code> from eating messages to stderr, please let me know!) But eventually I figured out it was hitting the <code>max_allowed_packet</code> limit and my import was dying. I was using the MySQL docs as my reference point, and <a href="http://dev.mysql.com/doc/refman/5.0/en/using-system-variables.html">they imply</a> that the following should work:</p>
<pre>$ mysql --max_allowed_packet=32M db &lt; data.sql</pre>
<p>I know now that this does not work. And I feel like a moron. If you read carefully, which I did not, you&#8217;ll see that they&#8217;re talking about server startup, not client startup, even though the example uses &#8220;mysql&#8221; instead of &#8220;mysqld&#8221;.</p>
<p>When I stopped using pipe viewer, I got this series of errors:</p>
<pre>$ mysql --max_allowed_packet=16M db &lt; data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes
$ mysql --max_allowed_packet=128M db &lt; data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes
$ mysql --max_allowed_packet=256M db &lt; data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes
$ mysql --max_allowed_packet=1G db &lt; data.sql
ERROR 1153 (08501) at line 175458: Got a packet \
    bigger than 'max_allowed_packet' bytes</pre>
<p>Now here&#8217;s the thing, data.sql was just shy of 80 MB. So obviously something was wrong.</p>
<p>A quick edit to <code>/etc/my.cnf</code>:</p>
<pre>[mysqld]
...
max_allowed_packet=32M
...</pre>
<p>Then all I had to do was restart mysqld, and the next time I tried:</p>
<pre>mysql db &lt; data.sql</pre>
<p>It finished without a problem.</p>
<p>7 hours and a massive facepalm later&#8230; I hope this keeps someone from wasting the same amount of time I did on such a dumb mistake.</p>
]]></content:encoded>
			<wfw:commentRss>http://coffeeonthekeyboard.com/mistakes-importing-data-with-mysql-309/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
	</channel>
</rss>

