I'm no expert but it sounds to me like you may have a little "death wobble" going on. Check to make sure you have between 5 and 9 degrees of positive caster and proper toe for the tires you're using.
Borrowed this from an article in Street Rodder:
One of the most common mistakes when setting caster is making the adjustment while the chassis is not at ride height front and rear. Caster is the relationship of the axle to the ground, not the chassis. That means if it's set at 5 degrees positive with the frame sitting level on a jig, then the car is put on the ground with a 3 degree nose-down rake, the caster will be 2 degrees positive-and you'll be wondering why the car wanders all over the road.
Another thing that might also be part of your problem is tow in. Again borrowed from a Street Rodder article:
Most straight axle street rods are sensitive to toe adjustment, and this is another one that will be influenced by the type of tires being used. Radials require less toe in, 1/16 to a 1/8-inch is typical, bias ply tires require a little more, 1/8 to 3/16-inch is usually required.