Haven't done much stick welding but have done plenty of MIG. When I find that happening, it usually indicates I'm not getting good enough penetration. Poor penetration can cause the welds to end up barely attached. I attached two pics. The first one is poor, and the second one is good. I would start by examining your welds and maybe practice on some pieces till you get a weld that can't be broken with a hammer.