If they are very yellow or brown, yes, remove them.
If they are still a bit greenish, wait until they are a yellowy brown colour before removing them.
While ever they are still a bit green they are still making energy for growth and while they are going yellow, your plant is recyling certain elements from within those leaves to be used elsewhere in the plant.
This is why it is so important to leave the older leaves until they are a yellowy brown colour, those wonderful pre made elements for recycling, as it takes a lot less energy to recycle than it does to have to make new elements (carbs, sugars, amino acids etc) from new at the same time as growing flowers, meaning there is more energy that can go towards flower growth and the plant will not have to divide energy between flower growth and element re-manufacturing, had the leaves been removed while still green. So...
Yellowy brown - remove.
Greeny yellow - leave until yellowy brown before removal.
Hope this helps, Organoman.