The brown stuff is mold. You must check all your buds by pulling the sugar leaves , if it plugs off easily it means the bud is rotten inside at the stem.
2 solutions, the safest one is to harvest now. Or you can leave it longer but you will to check all your buds every other 2-3 days.