pollen sacs are worrisome. Even if you put a lot of effort searching for them, chances are you will miss one. If you've only found 3 or 4, there still might be one lurking.
If they are few and far between i might risk it. If I keep finding more and of different development/ripeness, that tells me it's not a 1-off event and its popping them over time continuously. In that case i'm more likely to remove it.
If you caused it with stress, then removed that stress, plucking and praying has a chance, lol.
nanners.. recently had a plant pop a ton of them all on lower growth. I removed that plant, but in hindsight i should have left it. Think i had some light pollution because another plant had a similar issue, but i just pruned off the useless lower growth and didn't have a problem after that. Nanners don't often have much pollen, and if it does, there's a good chance it is inviable.
Even so, you won't get many seed from nanners. They won't ruin anything.