The promotional exam was supposed to replace the corrupt patronage system of former probation commissioner John O'Brien, who was convicted last year of illegally handing out jobs to politically connected candidates whether they were qualified or not.
But then 55 percent of the current employees who took the new, objective test received a failing grade, putting some of them in danger of demotions and pay cuts.
Now, the union is questioning the fairness of the test and how it was scored, while some experts doubt that a written exam is the right way to judge a probation officer's leadership skills. Still others wonder whether the high failure rate, even among veteran employees, says something bad about the agency's workforce.
Who knew meritocracy could be so hard?
"They fixed what was a horrible problem with an equally horrible solution," said Harold Lichten, a Boston employment lawyer who represents one of the employees who failed the exam. "Instead of trying to determine who are the very best candidates for the job by assessing the person's career and performance," he said,"they did a pen and paper test" that had a particularly adverse impact on minority job candidates.
Carl Wicklund, executive director of the American Probation and Parole Association, said it's unlikely that so many Massachusetts probation employees are unqualified for a promotion.
"I know a lot of people who are working at the agency and they're no dummies. Many have graduate degrees. I would say there is something wrong with the test," Wicklund said.
Court and probation officials say they were required to administer the test as part of the Legislature's reforms in 2011 after the Globe documented the hiring scandal and Jack O'Brien resigned as commissioner. Under the reforms, candidates for probation jobs must pass a qualifying exam, which was devised and administered by Industrial/Organizational Solutions Inc., an Illinois-based consulting firm.
But they didn't anticipate that employees seeking promotions or to make their acting position permanent would do so poorly. Of 362 employees who took the test in March, 199 failed, according to the state court system, which includes the Office of the Commissioner of Probation. A dozen of those who failed are already serving as assistant chief probation officers on a provisional basis, and face a demotion and loss of pay.
"The results have raised concerns with the union that represents probation officers and assistant chief probation officers," according to a statement by court officials. "The Trial Court has asked I/O Solutions to conduct an audit of the test development, administration, and scoring to ensure that industry standards and procedures were met throughout."
For veteran employees, many of whom were repeatedly passed over for promotions as O'Brien chose friends and relatives of politicians and court officials instead, the new test felt like a sucker punch.
"When I received my results I was in shock," said one veteran probation officer, who didn't want his name used out of fear of retaliation. "I gave them everything I needed to know to become a supervisor."
The employee suggested probation's current leadership made the test and scoring especially difficult to drive out long-term employees, viewed as vestiges of the O'Brien days.
"It's like they lump you in with the O'Brien administration," said the employee. "It leaves individuals who were victimized by O'Brien with no voice, no support. I never benefited from the O'Brien administration. But this administration still treats you like crap. They embarrass and humiliate you."
Court officials declined to provide a copy of the test, but union officials said it was divided into five sections containing multiple-choice and essay questions. They said that the failure rate was substantially higher on the essay portion of the test, and that candidates had to pass all five sections to be considered for promotion. The applicants are allowed to appeal the multiple-choice results, but not the essay portion of the test.
The court system has expressed "every confidence that the exam was unbiased and fair."
However, in response to union concerns, court officials agreed to delay filling the jobs until I/O Solutions can review the essay portions of the test.
Company president Chad Legel declined to comment on the test's high failure rate.
Wicklund, with the Kentucky-based probation and parole association, said most probation agencies around the country do not rely on written tests to decide promotions.
"That's typically more of an interview kind of situation, where you have a group of people do interviews with set questions," he said. "You're looking for someone with experience, identifiable skills, and competencies highlighted in the job description."
David Holway, president of the National Association of Government Employees, the union that represents the probation officers, said he wants to make sure the tests were handled fairly, especially the essay portions.
"The number-one reaction was that some of the people scoring the essays were longtime managers of the probation department who were there under Jack O'Brien. We want to make sure any prior bias didn't affect the fairness of the test," he said.
Despite the problematic test, Holway said, he is "generally pleased that the probation department is trying to move forward and fill positions in a manner that hopefully will take the politics out of the process.
"Under the previous administration, many careers were ruined if the commissioner didn't like you."