some solids deposited by your fertilizer water? The color is a bit odd, but if any of your fertilizer is brown like that, could still make sense. could be something growing on the deposits, but looks more like something dried up and left some solids behind repeatedly over time. This is fairly normal for fabric pots.. have to scrape/scrub/soak them between use to remove it.
as long as you have some sembalance of a wet-dry cycle, i wouldn't think it's microbial or fungal colony(ies). if it's growing as opposed to slowly deposited by ebb/flow of water, you'll know very quickly, lol.