HalloweenKing, a rim swap is not easy at all, I don't do it unless it is necessary. One thing that might help, I don't know if you know this or not:
When using any selection tool (i.e. magic want, lasso, polygonal lasso ect.) if you hold shift it will add to what you have already selected, if you hold alt it will subtract from what you have already selected. I think those are the right buttons. You will know if they are right cause a + or - will appear right by the cursor.
Hope that helps a little