A couple days after xmas while I wanted to kill myself due to having the Norovirus I got a call that our database servers at servercentral were all down. Sweet timing. Working next to a puke bowl I got in and saw that connections were hanging during login. It looks like this:
+-----+----------------------+------------------+------+---------+------+-------+-----------------------+ | Id | User | Host | db | Command | Time | State | Info | +-----+----------------------+------------------+------+---------+------+-------+-----------------------+ | 200 | unauthenticated user | 192.168.1.3:1312 | NULL | Connect | NULL | login | NULL | +-----+----------------------+------------------+------+---------+------+-------+-----------------------+
Pasting "unauthenticated user" into google it became obvious the problem is with DNS lookups. Due to some asshatery at mysql thinking it'd be a good idea to authenticate against hostname, it does DNS lookups for every connection by DEFAULT. Simple fix:
add this to /etc/my.cnf:
skip-name-resolve
I checked two days later and servercentral's DNS servers were still messing up frequently. I wonder how many other Christmas breaks were ruined...