If you can get you air temperature up to 300*, you won't have to worry about the house catching on fire.
Wood that is exposed to very high temps for long periods can ignite at much lower temps, perhaps 220*. I would place a thermometer at the opening and see what the actual air temp is flowing into the duct. If it is less than 200* there should be no problem. If it exceeds that, insulate the duct.