Hey man, first, you are already giving nuts? That’s crazy, you are killing the plant!
Then, buy good specific and sterilized soil to avoid this kind of problems, if this is high contaminated you can take out the plant and put it in a good and clean soil ( gnats put the eggs in the soil) , a good helper is neem flour on the ground or mixed in it , and neem oil sprayed on the plant also but i’ve get rid of them without the oil easily