Removing the dead or fully yellow leaves will force the plant to suck into greener leaves, which also means more sugar and a better smoke. It's just a few ones, it won't hurt them.
The dead leaves can also bring mold if the humidity is too high but cutting them off, even is the stress is small, does injure the plant - so mold too.
I think it's just a matter of feeling, preferences and grow conditions, some will tell you yes and other no.