it it does rub off, take some hydrogen peroxide 3%, dilute to anywhere near 1/8th concentration and gently wipe it off. i'd hit both top and bottom of leaf...
once you have mildew, it it more about controlling it than getting rid of it...
Check out "Cornell Formula" for a diy fungicide. Copper soaps, sodium bicarbonate, potassium bicarbonate etc.. all can help slow it down or kill it, but you will likely be applying as long as it keeps popping back up.