Thats an ovary or ovule.. whatever... Female sex gamete compliment to pollen. No worries.
pollen sacs extend from a twig and has no white hair-like projection. When mature they will open up and what looks like a ring of bananas will plop out... these dispense pollen. often refered to as "nanners"... will be yellow when mature. May also see "nanners" in buds by themselves or in a similar cluster.. The isolated ones are less risk than any cluster. Tweeze nanners out anytime you find 1...
if only a single here or there, you'll be fine.. maybe find a couple seeds here or there but not a big deal..... a full pollen sac or cluster is more productive and is a big deal... one closter can potentiall pollenate entire area.
Calyxes have no twig projection and have a white hair sticking out. This will never have pollen production. You may find an ovary at base of hairlike projection if you disect it.