I have burned wood in many different stoves and I NEVER have had cold air come back in.
I have seen sluggish drafts but it always picked up as soon as some heat was generated. It sounds like something is pulling hard on the interior space. bathroom vents, range hoods, etc. have to get air from somewhere to push air out. As to thinking metalbestos sucks, I have had both kinds and I will take the metal any day. Cheaper to install, cheaper to replace, cleans easy and drafts just fine. Masonry will last longer if you only burn coal, but wood is a different matter. Either you have a downdraft situation or something is creating a suction that pulls air down the chimney. And you are not far from a few of us.