There are several different secondary diaphragms that have different length stems. The wrong one can tip the secondaries open some, allowing in too much air. This will be like a big vacuum leak and can cause the idle to be too high.
Be sure the diaphragm is holding vacuum. If it isn't sealing the two halves of the vacuum chamber, you'll have a vacuum leak.
Be sure the small cork gasket is in place between the vacuum chamber and main body. Should use a new one, they get compressed during use and sometimes won't seal so good if reused. I have taken to using a small rubber O-ring for this, but it takes an uncommon size. So if you were to try this be sure the O-ring is only a little thicker than a new cork seal.
Check for carb-to-intake gasket leak, vacuum leaks in general.
If you have a choke, be sure the fast idle cam isn't hanging up. Be sure the gasket under the carb isn't holding the throttle blades open.