There's always that worry - but if there aren't many and you do it really carefully, should be ok. Some of my friends tell me they pitch the whole plant at the first sign of a male part... others will remove them VERY carefully.... I guess it just depends on how many you're having to deal with...
Good luck!