If you want to select for certain traits it is better to have hundreds or thousands of plants to choose from. If you are only able to grow four or even a dozen plants you won't have the diversity.
I'm going to start a project that challenges this idea. I think a more correct statement is that using the traditional geno/pheno isolation methods, you won't have the diversity.
I'm no Einstein, but I'll steal his ideas. There's another dimension other than space, and that's time! Example: You start with 50 plants, and do selection back to a dozen or whatever you have space to flower. Then open pollenate them and use seeds from a few females that you like the best every run. By using multiple plants each run, you are effectively extending/slowing the genetic convergence into the time dimension. If you cross 4 plants every run, instead of just 2, then maybe it'll take twice as long to reach a true breeding strain -- but it'll allow you to not need to have all the desired traits you're looking for in a single plant right off the bat.
Eventually after enough generations, you'll increase the probability of finding all the traits you want so that you can find "the one" even in small populations. The second thing is that you don't have to grow the males at the same time as the females. Grow out males and kill all females for a few runs. Collect pollen from the best and then do the females.
I'm calling my project "micro landracing", because that's essentially what it is. And I think this kind of breeding style can be applied to heirlooms or polyhybrids alike. You also don't have to start with just 2 strains. Hell, pick 5 or 30 if you want. It'll take me a lot of work for sure. But if you're not trying to make a living at it, then you can afford to spend a couple of years working on a single population of genetics.