plucking is best, yest.
Typically lone nanners don't produce a lot of pollen and potentially invialbe pollen produced too. Plucking will still reduce total number of seeds that'll occur. Better safe than sorry.
when they grow entire pollen sacks, you strongly consider removing the plant unless so little time it can't do much.
When the hairs darken and curl up, they are also not receptive to pollen anymore... So if it happens late, pollination isn't even possible. On top of the facat it takes weeks to grow large enough to notice with the human eye. Even so, you dont' want tiny immature seeds as much as you don't want mature seeds, lol. Still tastes like shit even if smaller. Probably cant' taste anything that develops with only a few days of time or some minimal amount.