Yes, there will be a loss of low-end torque with a larger system (this engine doesn't appear to make much low-end, though). We tend to recommend, for maximum performance, and not necessarily the "sound", stay with 2 1/2" pipe until CID gets up around 500.
Particularly with smaller headers like you have, you need that "step" in the pipe to increase exit velocity. For "driving" (street use), this is important. If it's a dedicated race car that requires an exhaust system, then yes, 3".
BTW, those headers are awefully small for a 7,000 RPM 400. Just my opinion. I'd like to see 1 7/8" units on there. Upper end power would significantly increase. Regardless, this car will need some "stiff" gear to be a decent driver. What heads are they?