Yes, unfortunately it is :-/
If you discover a true hermaphrodite plant the simplest solution is to remove the plant immediately. You can either destroy it or, if space permits, isolate it in a controlled environment to harvest the seeded flowers.
Another option, which carries higher risk but potentially greater reward, involves trimming off pollen sacs as they develop. With careful observation, you can identify and remove them long before they open. Successfully managing this allows you to harvest unfertilized flowers at harvest time.