If you run the bit back and forth it will dig a hole and cut too deep on the outsides. Ends with a crown in the middle.
Sanding rolls will help smooth out the pinch points around the guiides. Might need to tear some off to get into the deepest part. Take your time and use smooth cuts agasit the direction of the bit. Dont use die grinder air power does not have good tq and will stall the bit and dig a hole when it does. Electric is much easier to control and produces a smoother finish. But dremel is not going to cut it you will need something with a hp rating and works at low and fast rpm.
Focus on just removing the ruff casted surface and leaving a smooth cut finish. Hanging the shape is not always the best idea. Usally the shape is made with a purpose in mind and is not easy to redesign with just cutting the port and not adding metal back in. This basic clean up will net you a real 3-5 percent increase.
Also changing the shape of the port can really mess up swirl porting and other flow devices that will actually hurt flow when the air is moving at hundreds of mph thru the port.