After a mere 40 or 50 years, I am still trying to figure this out.
I don't think you can look at just one thing. I try to look at speed, pace, condition, class, connections, etc., but if there is one thing, I think it would be the old racetrack advice that "the pace is the race."