The fence might have some effect on the gene pool IIRC. The fence would have to be pretty high for a deer to not jump it if it really wanted to. I think the farmers around here put up an 8' high deer fence, quite an investment. They still try to clear them and some do while other get hung up.
I do recall that herds don't really move too far from their usual stomping grounds in the course of their life time. It's not a matter of miles but of thousands of feet. When they're pushed, well, that's another thing. The doe don't move far w/o pushing them. During the rut is when the big guys get around and help dilute the gene pool. We see the same local heard members around our place all year until the rut and hunting season. Then I think they hold up in the thickets around the streams but we'll be seeing them act like yard ornaments soon enough.
Although not a white deer, this fawn's momma was always hanging in the 7 acer of woods behind our place.