Yeah, get rid of all the small crappy branches, do a bit of "lollipopping" but try and leave as many leaves as possible.
Just remember, -
Maximum amount of healthy green leaves = maximum energy production = maximum growth.
Or to put it another way, -
Less leaves = less energy = less growth.
I would suggest only ever removing yellow leaves, for by the time they are yellow, they have served their purpose!