this is rather common no necessarely need to worry about it.
Banana inside the bud happen sometime near the end of flowering, this is a process know as rhodelization an ultimate effort by the plant to pollinate herself and reproduce. Most of the time such banana don't get mature enough to drop some pollen before you harvest, even though they drop some pollen plant won't have enough time to produce mature seed before harvest and in anycase trichrome maturation is advanced enough not to impact significantly your harvest potency.
other hermie case is the plant growing a few pollen sac and banana the first few weeks of flowering then stopping afterward and normally producing actively calix and pistilafterward.
In this case just touching the pollen sac before they open up and they fall easily
if you spot them early and fear about pollination you can always remove them safely with some tweezer before they get mature and open up.
finnaly if your plant don't stop producing more actively male flowers than calyx and pistil after the 4-5 first flowering weeks it could be a rare severe hermie case you'll have to rip the plant