I have done body work for more years than I care to remember. I know what works for me and the rpoducts that I work with. Working at home, I will regularly shoot SPI HB primer over 80 grit with no problems. I do nothing in a hurry at home, so there is plenty of flash and cure time, and I have no shrinkage problems. Before final priming/sealing, I sand with either 180 or 320, depending on what ever is handy at that time. At work, where everything is rush rush, I have to work differently. We use SW products there, so the primer is shot over 180 or 320. I always apply a minimum of 3 coats before I sent the vehicle to the paint shop. At this point, I have yet to have a problem with sand scratches showing up on my work. On the other hand, there have been many jobs of other techs that have had problems, mainly because of trying to cover 80 grit with the wrong primer, in a hurry.
If you have limited time for the primer to cure, and are short on your flash times, definately sand with the finer grits. That will reduce the need for heavier build, and reduce the chance of shrinkage showing on the final product.
Aaron