High MySQL CPU Load Today? Quick Fix
If you started seeing a load spike in MySQLd (or apparently Java) processes this morning, it may be the fault of yesterday’s leap second.
Apparently due to tides slowing the rotation of the earth, there was an extra second added to 30 June 2012, so at midnight GMT (8pm Eastern, 5pm Pacific, in the US), something funny happened:
- 23:59:59 30/06/2012
- 23:59:60 30/06/2012
- 01:01:00 01/07/2012
What? It can do that? Oh yeah, and it can do so much worse. This was nothing. Don’t get me started on timezones.
And yet MySQLd and Java apparently didn’t like it, at least some version and OS combinations. Fortunately, there’s an easy fix. Sheeri from Mozilla’s IT group pointed me to their blog post.
But if you aren’t dealing with puppet or anything, this is all you need to do:
# Log in as root
$ /etc/init.d/ntpd stop
$ date -s "`date`"
$ /etc/init.d/ntpd start
I didn’t even restart MySQLd and its CPU usage plummeted back to normal.
Now if that’s not the solution, you’ll have to start looking at things like SHOW PROCESSLIST and the slow query log and EXPLAIN to figure out what’s wrong. Or maybe you just have too much traffic. (Yay!)