How much metal are you talking about? How about running a magnet over it and finding out how much is steel, and how much is bearing material? You know for certain that 400 chevy motors usually run low on oil pressure anyway. Given that I'm sure you used a one piece seal, take the pan off the motor, remove a main and a rod cap and look at the bearing half to see if your concerns are justified. If you take the motor out, you're gonna have to remove the pan anyway to replace your bearings.
I would avoid just yanking everything out in a fit of panic until you find out what you're dealing with. If your bearings are decent and you find a problem with your turbo, by all means, put some clean oil in it, run the pump with a dril for 20 min to get the crap out of it, rebuild, replace or do whatever you have to do to the turbo and leave it together. How about a pic of the stuff found in your motor?