Hi,
the mushrooms are basically not a problem. They like it moist and therefore feel particularly comfortable at the line between clay pebbles and soil. They are definitely not pathogenic, so they don't harm the plant. On the contrary, they are more of a sign of healthy soil life.
As long as they don't take over, you can just leave them be. If they bother you visually, you can of course remove them; what is visible are just the fruiting bodies. The actual fungus is in the pot or in the substrate.
If you want to contain it later, you can let the substrate dry out completely once, and then the mushrooms usually disappear as well. However, this only makes limited sense, as it changes the microclimate in the pot and if you overdo it, it can even damage the roots of your plant.