Ensuring a Fair and Thorough Review of Candidates
This document is based on Searching for Excellence & Diversity: A Guide for Search Committee Chairs, a guide developed by the Women in Science & Engineering Leadership Institute (WISELI) at the University of Wisconsin Madison.
(Pages 43‐71 in WISELI Book)
A. DISCUSS AND DEVELOP CRITERIA FOR EVALUATION (pp. 44‐54)
Meet with your search committee to discuss and agree in advance on the criteria to be used in evaluating candidates. Note that:
- Relatively broad criteria not tied to specific experience or narrow specialty will generally lead to a more interesting “long short list” and leave room for “targets of opportunity.”
- A candidate might bring interesting strengths or attributes to the department other than those originally sought. If such cases appear, it is advisable to reevaluate and possibly modify the review criteria.
- It is also advisable to periodically evaluate your criteria and their implementation. Are you consistently relying on the criteria developed for the position? Are you inadvertently relying on unwritten or unrecognized criteria? Are you inadvertently, but systematically, screening out women or underrepresented minorities?
B. CONDUCT THE REVIEW IN STAGES (pp. 54‐59)
Conduct the review in stages, with the first stage confined to the construction of a “long short list.” This should retain all potentially interesting candidates, and not just those regarded initially as the top candidates.
STAGE 1: Creating the “long short list”
If you have a large pool of candidates, it may be difficult for all members of the search committee to thoroughly review all the candidates. In such a situation, consider assigning specific review responsibilities to members of the committee, consistent with the sizes of the committee and the pool of candidates. To generate the “long short list”:
- Ask all the members of the committee to review, even if briefly, all applications as they come in to get a sense of the possibilities. (Some search committee chairs recommend including a sign‐in sheet in each candidate’s file on which search committee members can indicate that they have briefly reviewed the file. Others also keep a checklist in each file to track receipt of required application materials. Some prefer to keep a master checklist for all candidates. See sample forms on pp. 21‐23.)
- Divide the task of thoroughly evaluating the qualifications of each candidate amongst the search committee. Try to make sure that each candidate receives a thorough and in‐depth review from at least two, and preferably more, members of the committee, and that each committee member is responsible for thoroughly evaluating the qualifications of a manageable group of candidates.
- Warn your committee about how much time reading and evaluating the candidates’ files will take. Inexperienced or busy committee members may otherwise put off reading the files until it is too late to do a thorough evaluation. Most search committee chairs recommend devoting at least 15–20 minutes per applicant. (Some search committee chairs find it helpful to provide a form that committee members can use to keep track of their evaluations; others prefer to let committee members devise their own methods for evaluating and comparing candidates. See sample forms on pp. 21‐23.)
- Advise your reviewers to concentrate on selecting all potentially strong candidates in their review group regardless of their personal preferences. In cases of doubt, advise the reviewer to retain a candidate for review by the entire committee.
- At subsequent meetings decide how long the “long short list” should be and construct the “long short list” by having the reviewers present their conclusions.
- Evaluate your “long short list” before finalizing it. Are qualified women and underrepresented minorities included? If not, consider whether evaluation biases or assumptions have influenced your ratings.
- The selection of the “short list” of candidates for interviews should be conducted at a later meeting, scheduled to allow committee members sufficient time to thoroughly review the strengths of the candidates on the longer list.
STAGE 2: Selecting a “short list” of candidates to interview
This is likely to be the most difficult part of the review process, since committee members will inevitably have different perspectives or preferences with respect to the open position.
Search committee chairs should think of ways to handle the delicate issues that can arise.
Many successful search committee chairs recommend the following:
a. To get the review off to a good start, with the entire committee willing to consider all candidates objectively:
- Review your objectives, criteria, and procedures.
- Emphasize that the committee represents the interests of the department as a whole and, in a broader context, the interests of the entire university.
- Remind the committee that the deans will expect the search committee chair to make a convincing case that the review was thorough and handled fairly. Some committee members may otherwise want to start with only their favorite candidates, and to argue against others without considering them objectively.
- Remind the committee that increasing the diversity of the faculty is an important criterion to consider in choosing among otherwise comparable candidates.
b. To make sure that diversity is considered seriously:
Remind the committee of possible inadvertent biases or assumptions before starting. If necessary, review the brochure Reviewing Applicants: Research on Bias and Assumptions. Require uniform application of standards in retaining or dropping candidates on the original list.
c. To handle the mechanics of selecting the short list efficiently and systematically:
- Have all members of the search committee thoroughly review and evaluate the applications of those selected for the “long short list.”
- Remind your committee members to devote at least 15–20 minutes to the evaluation of each applicant.
- Consider evaluating applicants on several different rating scales—one for teaching ability, one for research potential, one for mentoring potential, etc. Discuss the relative importance of different criteria. There is a sample form that can be used in the resources section.
- Schedule subsequent meetings to allow search committee members sufficient time to conduct thorough evaluations.
- After search committee members present initial evaluations, review the ratings a second time.
- Opinions expressed early in the process can change after many candidates are considered and comparisons become clear.
- Consider including the top candidates from various separate rating scales in your “short list.”
- Decide on the “short list” and possible alternates only after the entire committee has had a chance to review the longer list in depth.
- Do not allow individuals to dominate the process or to push for dropping or retaining candidates without defending their reasons.
- Ask quieter members of the committee for their opinions.
- Be sure that standards are being applied uniformly. Be able to defend every decision for rejecting or retaining a candidate.
- Do not allow personal preferences or narrow views of the review to dominate the process.
- Evaluate each candidate’s entire application; don’t depend too heavily on only one element such as the letters of recommendation, or the prestige of the degree‐granting institution or postdoctoral program.
- Evaluate your short list before finalizing it. Are qualified women and underrepresented minorities included? If not, consider whether evaluation biases or assumptions may be influencing your ratings.
- Keep sufficiently detailed notes so that the reasons for decisions will still be clear later.
C. RESOURCES (pp. 61‐71)
1. Sample Forms to Help Keep Track Of and Communicate With Applicants
Please note that the forms below are intended only as samples. You may choose to use, modify, or ignore these forms according to your needs or preferences.
These forms were adapted from the following sources:
- Pribbenow, Dean. 2002. Improving the interview and selection process. Madison, WI: Office of Quality Improvement, UW–Madison.
- Bensimon, Estela Mara, Kelly Ward, and Karla Sanders. 2000. The Department Chair’s Role in Developing New Faculty into Teachers and Scholars. Bolton, MA: Ankar Publishing Co.
Sample Form 1 Checklist for application materials for individual applicants.
Some search committee chairs recommend including a form such as this one in a folder created for each applicant. Most search chairs recommend that a single form to track the application materials for all candidates supplement or replace this form for individual applicants.
Sample Form 2 Checklist for ALL applicants.
Most search committee chairs recommend using a single form to track the application materials for all candidates.
A thank‐you letter along with an information packet should be sent promptly to all applicants upon receipt of their materials. This letter should state that unless confidentiality is requested in writing, information regarding the applicants and nominees must be released upon request; finalists cannot be guaranteed confidentiality. It may also contain information about the search committee’s time frame, since candidates will undoubtedly be anxious to know when they will be hearing about possible interviews. Additional information, e.g., papers or publications or a statement on his or her philosophy of education, may also be requested at this time. In addition, applicants should be asked to complete the anonymous Affirmative Action Data Questionnaire (administered by the Office of the Provost).
Sample Form 3 Checklist for communicating with applicants.
Sample Form 4 Sign‐in sheet for evaluation of applicants
Search committee chairs who use this type of form recommend keeping one in each applicant’s folder.
At least two search committee members should perform a thorough and complete evaluation of each candidate.
Sample Form 5 Sign‐in sheet for evaluation of candidates on the long short list
Search committee chairs who use such a form recommend keeping it in each applicant’s folder.
All search committee members should perform a thorough evaluation of every candidate on the “long short list.”
Sample Form 6 Checklist for developing the “long short list”
Sample Form 7 Evaluation of candidates for short list
Although evaluation procedures vary, the search committee may want to either develop a rating form based on job‐related criteria or keep the notes that the search committee generates. A rating form may consist of a series of job‐related questions or issues that the committee believes are crucial to the position. Written comments reflecting the judgment of each member of the committee should be made for each candidate. Not only will this allow the search committee to determine which candidates are to be interviewed, it will also save time if it becomes necessary to return to the applicant pool at a later date. The sample form here can be used for the entire faculty evaluation of a candidate after an interview.