Flue Size

I just finished installing a 1987 Harman Magnum stoker. The guy I bought it from suggested I use wood pellets to start it.
The smoke came out the door and I had to open the window to increase the draft. The masonry chimney has a 8x12 flue liner,
and an 8" pipe crock thimble. I used a 6"x8" reducer to connect the 6" Stove pipe. There is also a barometric damper.
It doesn't seem to have good draft and I can't keep it lit. Is the flue liner too large for the stove.

How tall is the chimney? is the baro damper closed and adjusted properly? it should be closed to increase the draft on the stove, the weight should be set correctly. You need a draft gauge/manometer to accurately set the draft on the stove.

Maybe post a pic of your setup, the baro damper, etc...

Wood pellets burn up pretty quick, many of us use starter bags, or some type of charcoal, you need to get coal pretty hot to start it and keep it going.

Is the blower under the coal (combustion blower) working correctly when you start it. ?

It might just be that you have to get the chimney warmed up first, it is a little large. Does it have an outside cleanout door? They cause alot of problems.

If that 8x12 chimney has been lying idle in this cold weather you may need to warm that cold column of air stacked inside to get it to pull for you. Chimney clear? Can you feel any draft [negative pressure] at the chimney cleanout door or at the barometric? A Dwyer draft gage is not a large amount of money and can be very helpful in figuring out exactly what is going on. Mike

As others have stated you may just need to get the rather large flue warmed up so it drafts properly. Put foil over the baro until you have good draft going when starting up. When I fired up this year I put a hair dryer on high in the flue for a few minutes to get the draft headed the right direction.

It's a good idea to have a manometer for diagnosing this type of issue.

Just to make sure you have allowed for adequate combustion air since it is a used unit and you mention not being able to keep the coal lit....

Can you post a pic of the combustion blower, the side that has the baffle plate so we can see how open it is? Did you clean off the fins on the squirrel cage combustion blower? (And the distribution blower too?)
Did you clean out the fines from under the grates and use a drill bit to ream out the grate holes of boogers? Doing this will make sure you are getting adequate combustion air to the coal on the grates.

And keep the doors closed...smoke can only get in house with you having them open.

I have great results using wood pellets to start flat and incline coal stokers. A 40lb. bag is a lifetime supply.

The very first thing to do when going to start w/ pellets is to start soaking them in a jar w / a lid in some starter fluid or kero/diesel...even rubbing alcohol will work. The longer they soak the better...use sparingly the pellets take very little liqiuid up, You don't want to pour liquid into your stove.

Make a dam for the pellets w/ a wood paint stirrer across the front edge,, fill coal from bottom just up high to peak on grates, leave room for a good 2 handful of soaked pellets. Wood stirrer will burn and no need worry about it.
Add the pellets and light them up and SHUT THE DOOR
Now turn on the combustion blower and regulate the air to keep the roaring pellet fire under control.
Wait...... now as it will take some time for the coal to catch fire where coal and pellets are touching.(10 min.)

Your chimney will be drafting nicely at this point, you may open the door to quickly add another handfull or two of coal on top of burning pellets. Wait........let this catch fire and then turn on your stoker feed motor and you should be up and running.