I don't see a need to defoliate your plant. It got plenty of room for light to reach lower parts.
If you really want to do it tho, do it now, before it starts flowering. Once it starts to flower, you're just gonna slow it down and get less of a yield.