Could be a clearance problem or a support problem. I ALWAYS set up a starter in the following manner. With NEW bolts I bolt up the unit with no shims. I use a .040" torch tip cleaner for a gauge, pull the pinion gear out to mesh with the ring gear and measure the clearance between the teeth. If it's much too large I will half shims to the OUTBOARD bolt only to bring the teeth in closer. Shoot for an 0.040" clearance. If it's too tight use FULL shims to braing the starter away from the ring gear. I've found that .040" clearance in the teeth works PERFECTLY.
Could be a support issue. Some blocks utilize a starter support bracket on the end of the starter that bolts to the block to keep it from tweaking.