they could have been popped under layers of calyxes and unseen. Easy to miss a couple near surface, too.
As long as it's a rare seed, no big deal.. with some strains you can expect it to happen, while others stay cleaner. If you pulled a nanner than touched another bud, that's a good chance too. Just the act of pulling a nanner will potentially put some pollen in the air.
if you pay attention while smoking, i bet you find a nanner or sac.