depends on severity and your capacity for attention to detail
nanners are less of a problem. go ahead and pluck them. a little spritz before can help reduce any potential airborne pollen. if oyu see pollen sacs with a whole set of nanners, that's a fully functioning sex organ... much different context with all the anatomy present. if sacs start dropping, i'd harvest sooner than later.
nanners can be impotent and/or produce little pollen per day... not always, but they typically don't cause 1000s of seeds. you may find a couple per ounce from unchecked nanners which is no big deal. but, you can't tell with your eye which are impotent and which will cause trouble. so, pluck them all... rotate plants... scour buds for them... some will be buried and nothing you can do about a couple seeds there.