Duece is right about the spooling. On the street, it will be fine. It also really depends on which powerstroke unit you have. The old T04 Garrets are a 1.10 range turbine and the compressor is only good for about 450 HP. The next generation is a GP38 which I think is in the .90 turbine range and will spool a little faster. The later GTP38 is the most desirable because it comes with a larger compressor.
I believe that all those units also all come with an integral wastegate which can cause issues. The problem comes in when, like in your case, the turbo is a bit small. The wastegate looses its ability to control boost and turbine speed and you can end up overboosting or overrunning the turbo. Either has its set of problems. I have heard of guys porting the wastegates to help, or you could just get another external gate to help out.