I’m agree with @Foreigner and the others about using a bigger fan than you need and running it slower with a controller or variac.
Like @Gpaw i noticed most of the noise comes from the motor end of the fan, fixing the fan directly to the filter helps cut noise quite considerably, I’ve never used a silencer but they’re not too different to a carbon filter, can fan or rvk are better suited to fixing directly.
For my climate i get on much better now with slow and continous ventilation, i get the best temp/rh% figures and close to 0 noise.