A good 4" fan is probably enough, but may need to run over 50% which increases noise. Might be worth the 6" cost, especially if you have any intention of scaling up in future. (Get an "EC" motor. They will come with 0-100% or 10-100% control.)
the key is to match cfm of duct fan to your cfm-rated fitler. I don't know what maximizes effectiveness, but i'd wager less than the maximum... one more reason to buy a size that you can run at 50% or less relative to tent size. They are fairly standard, if i recal.. the 4" filters usually are rated for ~170 CFM, which should be roughly what any decent 4" duct fan is.
A little negative pressure is all you need. If using a carbon filter, use a small length of duct for the carbon filter to be in the tent, but can have the fan outside of the tent to save room. Again, much easier to do when the fan is a bit on the strong side, rather than 'just enough.' I don't have any carbon filters, but none of my fans are inside my tent.. don't want that shit in my way :P