Let the plant recycle what is there already, it does not go to waste then.
As 0011 said, if you give them a slight tug and they fall off, then that is the time.
Cutting them off just because they look less than perfect is pointless, they won't harm anything, but your plant will get benefits from the leaves being left on until the very end.