Figuring gauge when changing yarn weight

I am planning to make a sweater that calls for #4worsted yarn. The yarn I want to use is #5heavy. I made a gauge swatch. There is one or two fewer sts in 4" and one more row in 4", so how do I change my needle size to get the gauge right? I tried on one size lower needles and it was off both ways( too many rows and sts) It seems to me that the gauge isn't reflecting the yarn size like I thought it would (although the yarn says its #5 heavy, it looks like a worsted weight to me) Any suggestions or am I just fussing too much and the stitch count is close enough to just go with the original size needles and yarn.