as long as your plant is producing more actively calyx and pistil this shouldn't be a problem just remove the male flower and you should be oki. just pass your finger on them (preferably before they open up and are mature enough to drop some pollen and unlike calyx they should fall easily
few males flowers the first few weeks of flowering is something pretty common and you don't necessarely need to worry about it check my "gorilla cbd" week 5 6 for exemple more than 40 males preflower, not a single seed in the end.