The barometric damper does one thing. It doesn't allow a change in draft to pull more air through your coal bed, thereby attempting, to the best of it's abilities, to limit your heat loss. If a more draft is created than set point on the baro, it opens to draw room air. When that draft settles down the baro closes.
Either way you're going to have a draw of fresh air into the house. The difference is whether it takes your heat out with it from the coal at 300°F or just room air at 75°F.
It's a hard concept to understand since we're so conditioned to believe that an open hole sucking heated room air should be avoided at all costs. I know it took me a very long time to understand the benefits personally.
You mentioned the baro was always part way open high or low fire. It has more to do with the draft out of your chimney than the fire you have going. A hot fire will produce a better draft...but...so will a good breeze, whether the fire is going or not. In the middle of summer my baro can be banging open and closed.
From watching my father in laws Chubby work I can tell you this. I've seen his stove sit at 350°F give or take 20° and be able to hold my hand on the flue pipe above the baro. It sits at around 100-120°F after the baro. That's what you're looking for.