If the rest of your plants are those little seedlings, it can go back, but if you have girls in flower, it will be a big risk. If you see one pollen sac, its extremely likely that you will see more and have more nanners too.
Keep in mind that pollen travels very efficiently. Theres a good chance you might get some pollination even if they're separated. If it's not hurting your babies, I would keep it segregated.