I believe Moroso (or Mr Gasket) makes a shim kit for this, it is a group of Nylon/Teflon shims/washers that look just like the distributor base gasket. Stack up what you need and then add a gasket.
You could also alter the distributor to use an adjustable collar as has been mentioned, MSD sells one just for this purpose. Requires the current collar be machined off the housing and the adjustable collar added.
Are you sure the oil is coming from the distributor base?? Not the back of the intake or the oil pressure sender port??