A couple days after xmas while I wanted to kill myself due to having the Norovirus I got a call that our database servers at servercentral were all down. Sweet timing. Working next to a puke bowl I got in and saw that connections were hanging during login. It looks like this:

+-----+----------------------+------------------+------+---------+------+-------+-----------------------+
| Id  | User                 | Host             | db   | Command | Time | State | Info                  |
+-----+----------------------+------------------+------+---------+------+-------+-----------------------+
| 200 | unauthenticated user | 192.168.1.3:1312 | NULL | Connect | NULL | login | NULL                  |
+-----+----------------------+------------------+------+---------+------+-------+-----------------------+

Pasting "unauthenticated user" into google it became obvious the problem is with DNS lookups. Due to some asshatery at mysql thinking it'd be a good idea to authenticate against hostname, it does DNS lookups for every connection by DEFAULT. Simple fix:

add this to /etc/my.cnf:

skip-name-resolve

I checked two days later and servercentral's DNS servers were still messing up frequently. I wonder how many other Christmas breaks were ruined...