Leave them, the plant is drawing pre-made nutrients from them to feed itself during flowering. Only remove them when they are nearly dead/brownish. You would be surprised as to how much photosynthesis they are still able to do, besides, they are doing no harm. In fact it is more beneficial to leave them rather than remove them because they "look ugly"!