The Withers was specifically chosen due to the fact that it was an allowance race type field of 4 horses.
Luxembourg had only won a maiden and non winners one. Doc Cheney not even that. I don't recall the straggler but he was the same.

The Preakness contained a small field as well, and other then Sweet, Bro D, and Barb, there was noone of any real consequence.
Those moves were not as aggressive as you paint them. They then backed off the Belmont where he would have been 3-5, that was very conservative and the key to his development.