i have this happening rather often with nonr or a few phenotypes per batch os seeds, the first weeks of flowering moslty bottom of the canopy.
just keep a watchfull eye for 2-4 weeks and they should stop to show.
simply touching with your finger will make them fall easily from the plant preferably before they are mature enough to show banana and drop some pollen. don't worry you have usually some time before it happen.