If you're in Week 4 of flower then most of the stretch is done, I don't see any airflow restrictions or anything that would make defoliation beneficial.
I like to defoliate 1-2 weeks before flip and the last time to defoliate (if needed) is during Week 3 of flower. The first 3 weeks of flower are a transition period where they are doing through a lot of vegetative growth still, defoliating during this time is less stressful on the plants.
After Week 3, taking leaves here and there is OK but nothing heavy or it could be to much stress and cause issues.