How Justice Toolbox Computes Attorney Win Rates

Free Information

Justice Toolbox is the first and only service to compute attorney win rates and open them to the public for free. We believe that attorney selection should be open and transparent and that consumers should have the opportunity to choose the best attorney possible for their case. Fairness requires that everyone have access to the protection of the law.

We calculate attorney win rates using an artificial intelligence program. There are three basic steps:

  1. Collect court records
  2. Use AI program to read court records and determine cases won and lost by each attorney
  3. Compute win rates and other statistics, both overall and per case type

An AI program must be used because there are too many court records for an attorney to read by hand. We have already collected close to five million records, and it is time-consuming for an attorney to analyze each record. We developed a proprietary AI that analyzes court records much more quickly (and can work 24 hours a day!). The AI was developed by our founder and CEO who is a Harvard-educated lawyer and holds degrees in computer science from Carnegie Mellon University and the University of Maryland. He has published multiple scientific papers and won both a National Science Foundation graduate research fellowship and National Defense Science and Engineering graduate research fellowship. He is the inventor of ten patents issued by the U.S. Patent and Trademark Office related to computer technology.

Court records generally do not use the terms "win" or "lose" but instead use more complex legal jargon (e.g., "judgment," etc.). Court records are sort of like doctor's notes in that they require special training to read. The AI must interpret the legal jargon in the context of the case to determine which party won or lost.

The AI determines wins and losses according to objective criteria. Our criteria for a win and loss is intended to track how a neutral, third-party observer would evaluate a case. Basically, a win is awarded when the court has ruled in favor of the attorney's client. A loss occurs when the court has ruled against the attorney's client. Some attorneys will argue that a "win" and "loss" can mean different things to different people, but using such subjective criteria would make win rates harder to interpret. For example, one attorney might argue that a ruling against his client was a "win" because the outcome "could have been much worse," but other attorneys could easily disagree that the client's outcome was really the best that could be expected. With our AI, the determination is clear-cut: an adverse court ruling is counted as a loss. Using standardized, objective criteria makes it easier for users of our website to understand win rates and make comparisons across attorneys.

Identifying Wins and Losses

The following tables explain in more detail how different outcomes correspond to a "win" or "loss" using our criteria. The first table discusses civil cases. A civil case is just about any case where two individuals or businesses are in a lawsuit for money or other relief. The person who filed the civil lawsuit is called the "plaintiff" and is the one who is asking the court to award him money or other relief. The person defending against the lawsuit is called the "defendant." The second table discusses criminal cases. A criminal case is when the government prosecutes an individual for a crime, and a fine or jail time is at stake. The person accused of the crime is called the "defendant." The attorney who argues on behalf of the government that the accused be should fined or jailed is called the "prosecutor."

Outcomes in Civil Cases

Court Outcome Explanation
Judgment for Plaintiff

A judgment is a ruling by the court in favor of one party or the other and is therefore a fairly clear-cut instance of a win. A judgment can occur after a trial but can also be awarded without a trial if the court decides a case based on documents filed by the attorneys called "motions." A judgment can also be awarded if the defendant does not respond, in which case there is said to be a "default judgment."

A judgment for plaintiff is a win for the plaintiff's attorney. Typically, the court will award the plaintiff some money or other relief. Even if the court awards the plaintiff less than he initially asked for, this is counted as a win for the plaintiff because the plaintiff has won a recovery from the defendant.

Note that a judgment is closely related to a "verdict." The term verdict technically refers to the findings of a jury, and so would only exist in a case where there has been a trial. Usually if the jury issues a verdict in favor of a party, then the judge would record a judgment in favor of that party because of the jury's verdict. However, a judgment can exist without a verdict and therefore is a broader term.

Judgment for Defendant Just as a court may award judgment to a plaintiff, the court can also award judgment to the defendant. Typically this means that the plaintiff does not collect from the defendant. A judgment for defendant is a win for the defendant's attorney.
Consent Judgment A consent judgment is a judgment entered for a party; however, it occurs in the special circumstance where the party against which it is being entered has consented to it. In other words, both parties have effectively agreed that a judgment be entered. This typically occurs when the parties have reached some out of court agreement. Accordingly, a consent judgment is counted as a settlement.
Dismissal

A dismissal occurs when a court ends a case without granting money or relief to the plaintiff. Dismissals can be voluntary or involuntary. A voluntary dismissal is a dismissal at the request of the plaintiff. This usually occurs when the plaintiff and defendant have reached a settlement, and the plaintiff asks the court to dismiss the case. Accordingly, a voluntary dismissal is counted as a settlement. If both parties jointly request dismissal, then this is also counted as a settlement.

An involuntary dismissal is a dismissal at the request of the defendant, where the plaintiff opposes or at least does not consent to the dismissal. When the court grants the dismissal at the request of the defendant, without the consent of the plaintiff, then this is counted as a win for the defendant.

Outcomes in Criminal Cases

Court Outcome Explanation
Guilty

Defendants have the opportunity to plead innocent or guilty to a charge. If a defendant pleads innocent, he is effectively challenging the prosecutor to prove his guilt via trial. If the defendant pleads innocent and is ultimately found guilty of a charge, then the case is counted as a win for the prosecutor. When a defendant is found guilty, this person is said to be "convicted."

Alternatively, a defendant may plead guilty, in which case he is admitting guilt. Defendants generally plead guilty after making a plea bargain with the prosecutor to obtain a lighter sentence than would be obtained if they went to trial and lost. In this sense, it is similar to a settlement in other legal cases. If a defendant pleads guilty, then the case is recorded as a "plea bargain" for both the prosecutor and defending attorney, which is neither a win or loss.

Not Guilty If the defendant is found not guilty of all charges, then the case is counted as a win for the defendant's attorney. Not guilty verdicts usually result from a trial.
Dismissal In a criminal case, a dismissal means that the court has ended the case without finding the defendant guilty. Courts may sometimes dismiss cases at the request of the prosecutor. Dismissals in criminal cases are treated as a win for the defendant's attorney because the defendant is effectively not guilty and suffers no punishment.
Decline to Prosecute The prosecutor may decide to "decline to prosecute" a case for various reasons, possibly a lack of evidence, a belief that he will not win at trial, or other reasons. The case is effectively dismissed at the request of the prosecutor. This situation is treated as a win for the defendant's attorney.
Diversion Diversion programs divert criminal defendants away from the traditional criminal justice system and towards counseling, treatment, and probationary programs. These are usually used for low-level, minor offenses. The defendant is typically required to meet certain conditions such as staying off drugs and not committing any crimes. If the defendant completes the program, the criminal case is typically dismissed without a conviction (i.e., without finding the defendant guilty). Successful completion of diversion is a good outcome for a defendant and so is treated as a win for the defendant attorney. Diversion is counted as an "other" outcome rather than a loss for the prosecutor since it is, in some sense, an exercise of mercy by the prosecutor.

A few other terms used on this site, like "Success" and "Unsuccessful," are also worth explanation. "Success" and "Unsuccessful" are used for case types that are typically uncontested. An uncontested case means that no one is opposing the request made by the person filing the lawsuit (i.e., the plaintiff). It makes sense to classify these cases in a separate category from "win" because these cases will largely be successful since no one is opposing them. Sometimes, another person could find out about an uncontested case and then file with the court to oppose the case; if the opposition succeeds, then the outcome is counted as "unsuccessful." Uncontested cases can be, for example, uncontested divorces where both parties want the divorce or a wills cases, where a will is filed with the court, but no one contests the will. The following table explains "success," "unsuccessful," and other terms used on this site.

Other Outcomes

Term Explanation
Success The attorney achieved a successful result in an uncontested case.
Unsuccessful The attorney represented a client in a case type that is typically uncontested. However, the case ended without the client's request being granted, possibly due to another party entering the case to contest the requested relief.
Transferred The case was transferred to another court.
Consolidated The case was combined with another case. This can occur if two cases are related. The outcome of the consolidated case would then be found in the case to which it was consolidated.
Sent to Arbitration Arbitration is an out of court process for parties to resolve their disputes. A case that is sent to arbitration is removed from the court system and resolved through an arbitration process. The winner and loser of the arbitration is generally not recorded in court documents.
Other An "other" outcome was obtained. For example, if a party dies before the case can be resolved, this would be counted as an "other" outcome.
N/A The case type does not lend itself to being called a "win," "loss," "success", or "unsuccessful" outcome. This is used for some simple cases that are almost like a formality, for example filing formal papers to collect on a legal win in another state.
Ongoing The case is still ongoing.
Undetermined The outcome of the case could not be determined.
Case Not Initiated / Case Initiated in Error These outcomes are recorded from court documents that generally indicate some error occurred in the court records system, possibly due to a misfiling or other reason.
Resolved The case was resolved.

We have done extensive testing of our AI but keep in mind that there is no AI program in the world that is 100% accurate. Our AI program runs without manual verification. Even if we had an attorney read five million cases, it is unlikely he would get through them all with no mistakes. The wins and losses reported by our AI are estimates and are not infallible.

FAQ

Q1: Should consumers have access to this information?

I. Empowering People

We believe that consumers should have more information about attorneys so that they can make informed decisions. Having more information empowers consumers to make the best decisions for themselves in their own legal cases.

The value in giving people more objective data about attorneys appears self-evident to us. However, occasionally, we come across people who argue that consumers should not have win rate information. They argue that the data is too hard for non-lawyer consumers to interpret or that there are other factors when choosing an attorney that are more important than win rate.

We believe it is better to empower people by giving them more information, than try to hide information in the belief that we need to "protect people from themselves." Consumers are adults. They may not be legal experts, but they have jobs, run households, and make important life decisions all the time. At the end of the day, they are the one who has to make the best decisions for their own legal case. It's the client's skin in the game, whether it be money, property, or liberty at stake, and the client is really the best person to look out for his or her own interests. Lawyers, the government, and other entities can be helpful, but the person who will most diligently guard one's own interests has got to be oneself.

Indeed, in the recent past, there were a number of laws that restricted the free flow of information to consumers on the theory that restricting information was "protective" of consumers, but those laws have been repeatedly struck down by the United States Supreme Court as violative of the Freedom of Speech guaranteed by the First Amendment. For example, a Virginia state law in the 1970s prevented pharmacists from advertising the prices of drugs. Virginia State Bd. of Pharm. v. Virginia Citizens Consumer Council, 425 U.S. 748 (1976). The theory behind the law was that price competition would damage the so-called "professionalism" of the pharmaceutical establishment. Supposedly, allowing the advertising of prices would cause a price war among pharmacists that would make it impossible for them to maintain their standards of professionalism. Of course, consumers were the ones hurt because they were subject to widely varying prices depending on where they shopped, and they could not compare prices between pharmacists to see who was overcharging them. Id. at 763-64.

This Virginia case contains some of our favorite quotes from the Supreme Court on the importance of the free flow of information, even if it be information about commercial services: "As to the particular consumer's interest in the free flow of commercial information, that interest may be as keen, if not keener by far, than his interest in the day's most urgent political debate." Id. at 763. "There is, of course, an alternative to this highly paternalistic approach [of suppressing price advertising]. That alternative is to assume that this information is not in itself harmful, that people will perceive their own best interests if only they are well enough informed, and that the best means to that end is to open the channels of communication rather than to close them." Id. at 770. The Supreme Court struck the price advertising ban down.

We agree with these wise words of the Supreme Court and believe in giving consumers access to information so that they can make their own best choices, rather than trying to paternalistically hide information from the public.

II. There is a Need for Better Information about Lawyers

On a practical level, people are currently at a complete loss as to how to find a good lawyer. Our founder, Bryant Lee, was inspired to create this company when attorneys at his five-hundred person law firm would regularly send firm-wide emails asking for advice on good lawyers for everyday issues like car accidents and divorces. If even lawyers do not know how to find a good lawywer, what hope does the general public have?

Frankly, if you ask most people how they would find a lawyer if they had a legal emergency, they would have no idea. Many people would resort to trying to get a personal referral (akin to the firm-wide emails at Mr. Lee's firm). The downsides of referrals are that it is slow and labor intensive to collect referrals, the pool of referred lawyers is likely to be small and limiting, and the consumer has no objective data about the performance of the referred lawyers.

The referral process also breaks down considerably when it comes to underserved, minority communities. Minorities are heavily underrepresented in the legal field and many minority communities have few or no lawyers. Folks in these communities may not have contacts with any lawyers at all. Thus, the most vulnerable communities are probably the least well-served by the status quo.

III. Consumers are Researchers

Moreover, those who would criticize the public release of lawyer win rates as unduly influencing consumer opinions, just like those who thought consumers would make pharmaceutical decisions based solely on price in the Virginia State Bd. of Pharm. case, fail to recognize that consumers are smart enough to research their options. Consumers compare options every time they go to the grocery store. They do not always buy the product that advertises the lowest price or claims to have the best results, especially if they find that the product does not work for them.

A well-respected personal injury lawyer in Virginia, Ben Glass, did some research on the topic and found that consumers spent months after an accident considering their options before calling his firm. Ben Glass, Great Legal Marketing (2012). We live in the Google era, where consumers research every major decision that they make, including their choice of lawyer.

Finally, it is typically impossible for a consumer to hire an attorney without speaking to the attorney first. Attorneys have to screen each case to ensure it is in their area of expertise, it does not conflict with another client, and to ensure a good personality fit with client. Thus, with or without win rate data, a consumer's final hiring decision will be made after speaking with the attorney anyway.

In conclusion, we believe in empowering consumers to make their own best decisions. We do that by releasing one of the few objective sources of information about attorney performance that is freely available to the public.

Q2: How should I use this data when selecting an attorney?

Lawyers are specialists, just like doctors. If you had a heart problem, you would want to go to a doctor who specialized in heart surgeries and had obtained succcessful results for past heart patients. The same concept is true with lawyers.

This site lists the number of cases each lawyer has taken per case type and also the lawyer's estimated win rate in the case type. On an attorney page, you can see attorney's results per case type in the "Attorney's Top Practice Areas" section. Click on a practice area to see results specific to that area. You can also search in the search bar by case type to find more attorneys who practice in that area.

Once you've identified a set of attorneys who have a lot of experience and good prior results in your type of case, it makes sense to see who would be a good fit for you personally. For example, if you find multiple doctors who have very good success rates, you don't necessarily have to choose the doctor who is 1% better than the others. Rather, you probably want to at least consider other factors, like who is going to listen to you and understand your specific issue, who will be responsive, who is a good personality fit, and so on.

Thus, once you have your list of "candidates" that you are considering, we recommend that you do as much additional research on the lawyers as possible. Read their web pages. Do some research on them on other legal websites. Use the information you find to narrow down the list.

Finally, you will want to call the attorneys you are interested in. The attorneys will want to know the details of your case. It's best to try to get the facts of your case straightened out in your mind before you call so that you can explain the situation clearly. Attorneys will be best able to help you if they clearly understand the situation. By calling an attorney, you are not obligated to hire them. You are absolutely allowed to call multiple attorneys to see which one you like best, and it is often advisable to do so.

After you've hired your attorney, keep in mind that it isn't a marriage. You aren't stuck with them. Generally, you can fire an attorney at almost any time.

Q3: Are win rates a fair measure if some cases are harder than others?

The fact that some cases are harder than others doesn't reduce the value of win rates because, over time, the statistics result in a correlation between win rate and attorney quality. In the long run, variation in cases tends to even out, and better attorneys will tend to win more often.

This site also groups together cases of the same type and by the type of client involved (a business or consumer). By grouping cases by type and comparing cases within the type, this also reduces the variation in easy and hard cases, which allows correlations between win rate and attorney quality to be observed more quickly.

Q4: Aren't some adverse rulings really a win if the client gets off with lower liability or is convicted of a much lesser charge?

In the main body of this page, we discussed the advantages of using an objective measure of wins and losses based on who won the ruling of the court. It's a clear-cut measure that leads to consistent results and corresponds well with how a neutral, third-party would view the case. Moreover, it is easy for everyone to understand.

That said, there is some logic to calling a case a win if someone is facing millions of dollars of liability and ends up paying only a few thousand dollars, or if someone is charged with murder but ends up only being convicted of marijuana possession. Still, one must be careful in evaluating these cases because the so-called "millions of dollars of liability" might just be an unrealistic and excessive demand by the plaintiff, and the murder charge could just be a prosecutor throwing the book at a defendant without any real hope of the charges sticking. We are thinking about ways in which we could present more detailed types of outcomes than just a win and loss, though care is needed to get it right and not make the measure too subjective.

Finally, lawyers can sometimes be too quick to declare a case "impossible." Many lawyers believed that O.J. Simpson would be found guilty. Whether you believe O.J. was guilty or not, the weight of the evidence was clearly against him. O.J.'s lawyer, Johnnie Cochran, could have have given up and made a plea deal. If he had gotten O.J. a few years in jail, many lawyers would probably have called that a victory. Instead, Johnnie pulled off an indisputable and undeniable win by obtaining a not guilty verdict.

Q5: Will this site encourage lawyers to avoid hard cases?

This concern is probably not realistic. In numerous law firms, attorneys take any case that comes in the door that is in their practice area and they believe they can do a good job on. Moreover, attorneys know that most cases settle, so an attorney would have no expectation that any particular case will result in a loss. This is especially true at the beginning of a case when not all the facts of a case are known, and it is often not clear which side has the better argument.

In addition, consider whether a lawyer would have an incentive to turn down a hard case to avoid adding one loss to his record, when most legal matters in court will earn a lawyer thousands of dollars in fees.

Q6: Don't most cases settle rather than leading to a win or loss?

As discussed in the main section of this web page, wins and losses are not limited to trials. Every lawyer knows that there are many common ways to be victorious in a case without going to trial, including summary judgment, dismissals, the prosecutor dropping charges, and so on as described in detail above.

Looking at our own data, over 30% of the cases in our database have a win or loss outcome according to the criteria described above. Including "success" and "unsuccessful" outcomes increases that number even higher.

Q7: Don't other factors matter for an attorney besides win / loss rate?

There are other factors that are important for an attorney-client relationship, such as attorney's responsiveness, communication skills, personality fit, and so on. However, at the end of the day, clients are in a legal case to get a result. We want to help consumers find the attorney that gets them the result that they want.

Consider a sports analogy: is the better team the one that "played the best" or the one that won the game? We want to help consumers find attorneys that win.

Some people might say that "ability to settle a case" is an important lawyer skill that is not measured in win rate. However, ability to settle a case often comes from being in a strong legal position, which means being on the winning side of the case. Moreover, this site reports settlement statistics for lawyers as well.