Ants usually only go to a plant when they find aphids there that they can milk. Check again if you have aphids. Otherwise, I think the ants are just selling themselves.
worry about what ?
no neem oil in flower, thats golden rule.
i would not recomend to use any neem oil, there is plenty better alternatives to fight pests or fungus.