If the plants are not having air flow/humidity issues or light penetration issues then there's no reason to remove them, they're powering your plant so leave them alone if you can.
I won't do any major defoliation after Week 3 of flower, the first few weeks are transition period.
After this period, it's to stressful to do any major defoliation IMO, a few here and there would be ok.
If you want to remove some for light penetration, move the leaf by hand first and see how much more light is gained, if not much then don't cut it.
Too much defoliation, especially at the wrong time, can cause unwanted stress and the potential for hermies.