Kind of a tough one. I sold my 81 Firebird that had a strong running earlier 400 a few months back and my current driver is an 83 El Camino with a stock rebuilt 500 Cadillac in it.
I really enjoyed the 400 in the Pontiac, it had a lot of the same traits as the Cadillac; lots of low end torque, reliable and relatively speaking decent economy. Neither engine in stock form really winds up to tight.
Personally I like the idea of the 400 and 4 speed…..it’s just a fun combination. You can also add Tri-Power to it which I’ve always had a soft spot for. Even as a non-original unrestored car it would probably be worth more with a Pontiac motor in it.
The torque on the 400 is impressive (for the displacement), but nothing compared to the big inch Cadillacs. Like the Cadillac guy you talked to indicated, I think if you go the 472 route I think you will be disappointed if you go back to the 400.
If you go the Caddy route at a minimum you will want to replace the stock nylon timing chain set and put a set of valve seals in it, while you’re there put in a set of decent springs and the engine will be able to go to 5000 RPM. With the right gear set (3.00-3.50 range) 5000 will be plenty.