It's called Botrytis bro.... it's a fungus that spreads quickly when you have very high humidity during flowering..
It starts turning gray and then brown, it dries out and crumbles into powder.. That spreads, when you clean it you have to be careful not to touch the parts that are not yet affected by the fungus.. that way you can clean the affected areas and keep the rest clean.. you don't always have to harvest early, you should clean if there's time, and keep an eye on them, if you control it you can take it all the way to the end of the grow!
I hope I helped you!
Happy growing, hope everything goes well!
(You can use a preventative from the vegetative stage, if you use bio neem + propolis you'll definitely keep everything safe)