4" is plenty for anythign 3x3 and smaller... adding a carbon filter may make bumping up to a 6" a worthwhile idea -- especially if you plan to upgrade the tent in future.
Can always buy 2 4" fans and that should allow you to run at less than 50% power. This will be the quietest option (or 1 6" at under 50% power, but a 6" carbon filter jump up quite a bit in price i believe). This is a good goal to have. If it's in an isolated room and a little noise is okay, one 4" fan can definitely do it, too. You might need to run it at 70-80% power.
For odor control, you need negative pressure. Walls should bow in slightly -- doesn't have to be extreme in nature. This prevents any air exiting without going through the carbon filter.
Not all carbon filters are built the same. Some have more active carbon. The 4" ones should be rated fro ~170cfm, which will usually match up well with the 4" duct fans etc. Make sure you don't over power them and they work well.. Make sure the manufacturer doesn't skimp on the active carbon. A couple inquisitive emails can help, here.
Put at least 180-degrees of bends in your duct to help prevent light leaks. After you have everything attached, slowly adjust power of fan and when you initially get a little bowing-in of walls, you know you got the negative pressure you need to control the stank.
ducts can let in a ton of light... you need a baffle or bends to prevent this. I use bends as well as a cardboard box that the fan rests partially inside - blocks any light from directly entering. Some situations require the fan to be inside, so just rest the duct partially inside a box -- plenty of room aroudn duct/fan so as not to impedge airflow too much. Baffles and bends do reduce cfm.