by Charlotte Wager, NALP Vice President

NALP Bulletin, September 2010

Surveys can provide useful information and data to people making decisions. They are a reference point for anyone doing research. They provide a way to assess, compare, and measure. They serve a purpose. Presumably, that’s why so many of us spend so much time responding to surveys and then hours reviewing the results. In my humble opinion, surveys, like many other things in life, are imperfect. Sometimes it’s the poorly drafted definitions (or absence of definitions). Sometimes it’s the meaningless questions. Given the inherent imperfections in survey tools, we need to be especially cautious of relying too heavily on survey results. They have a place in our research, sure, but they are simply one data point that should not be given disproportionate weight. Survey results need to be considered along side all of the other relevant and available material. That way we can hope to make educated and informed decisions.

In the NALP world of legal recruiting and professional development, we see lots of surveys — and it seems as if a new one is added every few months. NALP itself uses several survey-like tools every year including the member salary survey and perspectives on fall recruiting survey. The NALP Foundation is also known for several high-profile surveys including the Associate Attrition Survey, After The JD, and the most recent Benchmarks, Competencies and Compensation Approaches survey. These surveys are designed to help provide members (and students) with information that can be weighed and assessed, along with all of the other relevant material available, to help us make reasoned decisions.

But we also see an increasing number of surveys in our profession that purport to rank and rate our members — specifically schools and employers. Rankings and ratings may be useful in some contexts, but for many of the decisions that law students face, rankings and ratings should not be given too much weight. How to choose a law school, for example, likely is not a decision that should be made based exclusively — or even significantly — on the law school rankings. It just depends. What students to hire, what summer program to select, where to go to work — all of these decisions should be made after reviewing all available and relevant information, not just a student’s class rank, or firm’s score or rank on a national survey.

What does NALP think about ratings and rankings? Policy Memorandum 102 answers this question directly. The policy begins: “NALP does not rank law schools or legal employers and discourages the use of rating systems or rank-ordered lists in evaluating law schools, legal employers, or individual candidates for employment.” Why? The Policy goes on to explain: “Law schools and legal employers are defined by many different … characteristics whose significance and relationship … may vary from year to year. For example, no rigid ranking of law schools can quantify the subjective value of education available at each school; the quality of teaching cannot be quantified on a national basis. Similarly, rankings that purport to rate legal employer summer programs or the quality of life and work available to associates cannot fully capture the broader culture and environment of an employer.”

Employers evaluating candidates also need to look beyond class rank: “Legal employers are encouraged to define their unique organizational characteristics and recruiting needs, to establish interview and selection criteria based on their realistic professional requirements, and to evaluate each candidate individually” (Policy Memorandum 102). Recruiting students based solely on class rank does not guarantee a great fit.

Bottom line: my list of top five movies and top ten songs likely looks quite different from yours! In fact, none of my favorite songs may be on your list — you may positively dislike my favorite songs. Your five favorite books may feature titles I have never heard of and my favorite cities might be places you don’t like or haven’t visited. We are talking about very subjective decisions and to help our students, colleagues, and each other make smart choices we need to encourage consideration of a broad range of information. Rankings and ratings can’t possibly tell the whole story. Worse yet, rankings and ratings can mislead.

In conclusion, consider this advice —

• Law Schools: Encourage your students to learn as much as they can about a school, employer, or summer program before choosing. Students should gather information from a variety of sources, weigh it all, and then make a subjective decision based on all of the circumstances. Specifically, whether during the law school selection phase or in connection with on-campus interviews, encourage students to use the NALP directories (NALP Directory of Legal Employers and NALP Directory of Law Schools), visit schools and employers, talk to alums and associates, consult websites, and read whatever other material the school or employer has prepared and made available. Tell students to use the information to create their own rankings and ratings that apply exclusively to them and their own individual circumstances.

• Employers and schools: Make a broad range of information available to students and be forthright and responsive when providing information. The broader and more effective the exchange of information, the better decisions everyone will make and the more likely it is that everyone’s expectations will be met.

When you look back on the important decisions you have made in your life I expect that relatively few of them have been made based on someone else’s ranking or rating of your options. Some decisions are just too big to delegate to someone else’s opinion. Like your top ten movie list!