Depends on the panel for me. If it's a panel that hangs upside down, like a headliner or trunk lid, I sand everything (except of course the back of the material).
If it's kick panel or other small panel, I wouldn't bother.
Your best bet, consider you are probably using a different brand of foam and a different brand of adhesive, would be to do a test for yourself. Take 2 scraps of the plastic and sand one, but don't sand the other. Glue a piece of material to both of them. Let it sit a couple days, and peel the material off of both pieces of plastic. If they both come off with equal force, you know it's a waste of time to sand the plastic!