I like to wait until thellant is done stretching, for sure, to defoliate. So, it's usually week 4-5, for me.
When I defoliate, I only remove leaves that cover buds. AND! This part is very important: lollipop BEFORE you defoliate leaves. Like someone else mentioned, taking off the tiny branches and buds that don't get much light will help the plant focus on buds that DO get plenty of light. This makes for bigger buds and less scurfy buds. If you lollipop right, you won't get any scurf at all!
Once you lollipop, then you defoliate.
If you want to talk more on this topic, feel free to message me on here by clicking the little airplane button in the bottom-right corner of the screen.