I think wobbling up and down would indicate its not centered. Does the serpentine pulley move around some when you place it on the dampner and try to slide it around, or does the lip snap in the hole?
If its moving around, you might be able to make a shim of sorts to center it by laying a 2 foot length of masking tape on something like a glass table that's clean so that when you remove the tape it doesn't tear or pick up dirt. Then use an xacto or box knife to cut a long thin strip(sort of like pin stripping) about an 1/8" wide. After cleaning the dampner hole, and the pulley lip with alcohol, wrap the strip around the pulley lip a few times until its just larger in diameter than the hole in the dampner. Then try to lay the pulley on the dampner with light pressure as you unwrap/cut some of the tape a little at a time until it just fits if you know what I mean. It won't center it perfectly as you'll have more layers of tape on one side vrs the other, but it might get it close enough. I think masking tape is about .004" thick. It will probably take some trial and error.
Just a thought. FWIW,