If it is not doing any harm, why remove it? It is making energy, energy your plant needs to grow its biggest and fastest.
If your plant did not need it, it would not have grown it in the first place!
Hope this helps, Organoman. (and strident anti defoliation advocate!)