It's not clear what happened. If you are trying to replace the pot bushing, its probably easier to remove the gooseneck if you already have the pipe out. Just drive the old one through; there are no stops. Install the new bushing until the edge is flush with the casting on the driven side.
They can be very difficult to remove. If I were in the business, I would have a special tool made at a machine shop for that purpose. Use Never Seize when you install the new one.