Have you disabled caching? 5.5 caches a lot more than older versions of mysql.
Keep in mind that even if you do get mysql to use less memory, you'll be fighting a losing battle.
To explain, since you don't have swap thats why mysql is "using" more memory. Its got nowhere to page to, so its got to use real memory. 5.1 used less total memory (it paged too), 5.5 uses more with the additional caching capabilities. In reality if you had swap you'd only be using 25MB with 5.5, but since you dont you get to keep everything resident using up real memory you dont need to use. So part of the problem is the server configuration itself. Your virtual server is robbing you of memory by forcing you to keep everything resident, so I'd fix that ASAP. If your virtualization technology doesnt support swap move to a different technology that does (kvm, vmware, xen, etc.). Its killing you to not have any paging capability, you're wasting memory on things you could page out.
With all that said, 512MB is a really small footprint for a modern server, as a basis of comparison, an iPad has 512MB of memory, and a server is going to be asked to do a LOT more than an ipad.
So eventually even with swap and 51MB of real memory you'll hit a limit again. swapping is SLOW compared to real memory, so conversely you dont want to swap if you can help it. Too little RAM and you start swapping things that should be resident (as opposed to the stuff that doesnt need to be). So, I'd recommend at least 2GB on a 64bit platform.
I'm passing this on because you're going to find more and more problems with such a small memory footprint (512MB) with other modern software you want to use, not just mysql. For example apache has some neat caching modules, there are some great accelerators that use RAM and speed up things, and so on. So on top of not being able to use those kool things to make things faster, these days developers assume the system is going to have more memory (and swap), so they are going to design other software around that assumption to try to use as much memory as possible to speed things up for their code too (memory is really really fast compared to reading from disk).
So, bottom line - RAM is cheap, get as much as you can!