You can remove it or leave it. It's not going to make much difference either way. The way i see it, removing lower growth I personally don't believe it is of any benefit to the upper bud development. I like to leave the plants bud sites in tact, because even though they don't fill out to be pretty smokable bag appeal nugs, what they do lend themselves to is making bubble hash, alcohol/oil soluble edible material. Or just trim it, dry it, cure it, and grind it up and drop it in your vape. You'll get high off it i promise.
Sometimes these modern day grower practices I think are just silly.