Hi E.alleg, your chimney is doing what chimneys do. With more heat differential between the outside temps, and the hot gasses in the chimney while the stoker is making heat, the baro will have to open more to keep the draft from being too high.
But when the stoker is at an idle, not making much heat, and the outside temps are warmer, you will have less draft in the chimney, The baro if set for say .05 in the chimney, and the chimney is only pulling .04, then the baro flap will be closed.
If you have draft when the stoker is idling, and it increases when the stoker fires up making heat, and the baro keeps the the Maximum draft at your target setting, then that is all that can be done. Personally if it has when idling some draft over the fire, and at least .02 in the chimney, and increases to .05 [your set point] when the stoker fires up, that sounds perfect. Having a lot more draft when the stoker is idling is just pulling heat out of the boiler.