Make sure you are assessing judgment and not technical competence or raw intelligence. Some very bright people have lousy judgment, and some people of average competence have extraordinary judgment. It is essential to be clear about the mix of knowledge and judgment you need from key people.
One way to assess judgment is to work with a person for an extended time and observe whether he or she is able to (1) make sound predictions and (2) develop good strategies for avoiding problems. Both abilities draw on an individual’s mental models, or ways of identifying the essential features and dynamics of emerging situations and translating those insights into effective action. This is what expert judgment is all about. The problem, of course, is that you don’t have much time, and it can take a while to find out whether someone did or did not make good predictions. Fortunately, there are ways you can accelerate this process.
One way is to test people’s judgment in a domain in which feedback on their prediction abilities will emerge relatively quickly. Experiment with the following approach. Ask individuals whose judgment you want to evaluate about a topic that they are passionate about outside work. It could be politics or cooking or baseball; it doesn’t matter. Challenge them to make predictions: “Who do you think is going to do better in the debate?” “What does it take to bake a perfect soufflé?” “Which team will win the game tonight?” Press them to commit themselves—unwillingness to go out on a limb is a warning sign in itself. Then probe why they think their predictions are correct. Does the rationale make sense? If possible, follow up to see what happens.
What you are testing is a person’s capacity to exercise expert judgment in a particular domain. Someone who has become an expert in a private domain is likely to have done so in his or her chosen field of business too, given enough passion about it. However you do it, the key is to find ways, beyond just waiting to see how people perform on the job, to probe for the hallmarks of expertise.