I would trim the inner fans too, only the ones that point towards the main stem so it's uniform all the way round. All the buds growing on the branches that will lean in the direction of the fans you leave on. It's going to be a problem if the part of the plant that's always near your fence doesn't get light you could get mold from the leaves touching each other (from personal experience in the same situation)
On the plus side stripping the leaves on the inside should make the branches lean outwards as there will be more weigh on that side... mid-late flower there will be nice seperation.
I have grow a few big plants outside it always helped me doing this