Since I was doing this I thought I would post a thread with photo's of doing exactly this...
If your damper is simply missing the flapper or only the flapper is damaged you can replace these fairly easy (If its missing I suggest you look for it up on the rear shelf inside the chimney first as that is the most common place to store it).
In my case I wanted the flapper and the entire damper housing out and gone in order that I could achieve a full 6" stove pipe and strait up into my Terra Cotta chimney liner
I chose NOT to listen to these so called experts telling me a stainless liner is better (its NOT and I like being able to convert back to a fireplace easy as pie anytime). I have seen 100+ year old Terra Cotta liners still in use today and maybe if I ever see a stainless liner last more then 15 years with a coal stove I'd be surprised.
I chose not to listen to these so called experts telling me "you cannot remove a mortared in Damper... I decided to just do it all myself and pay nothing (the only thing im going to be buying is the tile to reface the fireplace, everything else is either used or stuff i have laying around).
The key with removing these buggers is to try to NOT NOT NOT disturb the Lintel (usually angle iron or metal piece that spans underneath the bricks above the fireplace opening), If your lucky enough to have a stone fireplace or such custom work that they used the "Keystone" method then you want to be careful not to upset that archway because its load bearing.
Other then that its pretty simple as these photo's show... The trick is to attack the mortar by wedging then prying.