I think light defoliation during early flowering should not be a problem. Just make sure that it's not too much.
I usually grow bonsai-style plants, due to height constraints, and there i regularly have to defoliate, even up to mid-bloom.
So far it never seems to have hurt my plants, but then again i wouldn't have a real comparison that wasn't defoliated, but with the other factors being the same to see if the result would have been any better.