It takes a pretty bad infestation to actually affect your plants. The flies are harmless but the larvae can damage roots in large numbers. The best defense against them is not promoting an environment they like. Don't keep the top of your soil wet all the time. Maybe try bottom feeding until you get them under control. You can top dress your soil with diatomaceous earth .
It's food grade and kills both gnats and larvae on contract.