The Secret Algorithm That Could Send You to Prison, Take Your Kids, or Crash Your Car

By Joe Jarvis

In 1603, English officials arrested Sir Walter Raleigh. They charged him with treason. He was allegedly involved in a plot to overthrow and kill the new King, James I.

At his trial, prosecutors entered just one piece of evidence. It was a confession letter written by the man who allegedly planned the plot, Lord Cobham. The letter named Raleigh a co-conspirator.

Sir Walter Raleigh said the confession alone was not reliable. If he could bring Cobham to testify in court, he would at least have a chance to cross-examine the only witness against him.

But the English courts at the time were not particularly interested in due process. They convicted Raleigh without allowing him to confront or examine the witness.

He was imprisoned, and eventually executed.

Fast forward to USA, 2017. A man faces murder charges. And the main witness against him is a computer algorithm.

The algorithm analyzes complex DNA samples. It decides if the DNA of the accused is present in a sample.

Naturally, the defendant wants to know how exactly the algorithm works.

And he has good cause to wonder. His earlier conviction for the same murder had relied on a different DNA testing machine.

But scientists later decided the algorithm wasn’t quite right. The evidence that convicted him was changed from positive to inconclusive.

He had been sitting in prison, convicted of murder, based on a faulty algorithm!

So at the new trial, the defense asks to examine the new machine’s algorithm. But the company that designed the algorithm doesn’t like that idea.

The algorithm is a trade secret. It was costly to develop, and they don’t want that secret getting out.

We’re living in the age of algorithms.

An algorithm chooses which music you listen to. And Spotify does a pretty good job… A faulty algorithm might force you to listen to a gospel song instead of gangster rap. Not the biggest deal…

But what happens when an algorithm for a self-driving car has to decide between hitting a group or pedestrians, or plunging itself—and you—off a cliff?

The algorithm becomes a matter of life and death.

Before you get in a self driving car, you’ll want to know a little bit about the algorithm.

What kind of split-second decisions has it been programmed to make?

What data inputs are considered?

And who exactly wrote the algorithm? Remember, algorithms still have a human component. And humans are inherently biased. Their subjective opinions will inevitably make it into the algorithm.

When an algorithm convicts the defendant in a murder trial… that too is life or death.

Some child protective services use an algorithm to decide which kids to take. The algorithm assigns a risk score based on inputs like how many calls the department has received, and if the parents are hostile towards caseworkers.

Other courts already use an algorithm to figure out the recidivism risk—if a criminal is likely to re-offend. The higher the risk, the longer the sentence.

If I smoked a joint or got in a fight in high school, am I 40% more likely to commit a crime?

Will I be assumed guilty based on probability alone? How can I be sure the algorithm isn’t biased?

When ProPublica analyzed the recidivism algorithm, they found that it was racially biased.

O’Neil Risk Consulting & Algorithmic Auditing is another organization looking into how fair the algorithms used by the justice system are. But they admit that even the audits are subjective to a degree.

That’s pretty scary to have algorithms control the criminal justice system. Due process might be replaced by computer processors.

But algorithms aren’t necessarily a bad thing. They can analyze vast and complex sets of data. It takes a lot of work to build a good algorithm. You can understand why a company wants to protect its secrets.

But the benefits of algorithms cannot take precedence over due process.

The American system of justice was designed to be open and transparent, to avoid the type of show trial that convicted Sir Walter Raleigh.

Whether the witness against you is a human or an algorithm, transparency–and due process–is non-negotiable.

You can read more from Joe Jarvis at The Daily Bell.