depends on your tolerance for mistakes on the crop, and your intentions... also would behoove you to differentiate between herms and intersex flowers. The nature of the plant kind of implies one, and not the other so much. Cultivators take both for the same.
if this is somethin you expected because of a genetic flaw that you knew of (like growing zkittles whatever TF) then proceed
if this is putting your medicine at risk, then perhaps remove
a rule of thumb i use for problematic plants or newer cuts is that if the pollen, presumed to be viable, would be dropping on a timeframe to pollinate itself....then kill it now. If it is throwing too late to self-pollinate, i take that as a vigorous and life-loving plant