Turns out, to disable password-based access, you need to set:

UsePAM no
PasswordAuthentication no

And, while you're at it, you might as well disable reverse DNS verification, since it's already pretty obvious that all of those hits are attempted security attacks.

UseDNS no