Matt Mullenweg, founder of WordPress and the CEO of its parent company Automatic, got loads of attention in 2020 when the whole world went remote. The reason was simple: Mullenweg’s companies had been near-full remote for years by that point and they were thriving. However, Mullenweg’s managerial ingenuity goes beyond mobile work. Perhaps most interesting is Automatic’s long standing hiring methodology.
Let me backtrack a minute. If you follow the academic literature on best hiring practices, you’ll know that the conventional interview is useless at best. More likely, however, is that it’s worse than useless because it leads us to hire incompetent workers over competent ones. This mostly boils down to unconscious bias. Human bias is an inherent part of the interview process, even when interviewers strive to be impartial. Unconscious biases, such as affinity bias or confirmation bias, can influence decision-making, leading to potential disparities and missed opportunities for diverse talent.
What’s the alternative? According to many studies on the subject, cognitive tests are really better indicators of future success on the job.
Perhaps the most famous is a 1998 research paper by University of Iowa professors Frank Schmidt and John Hunter. The meta-analysis examined over 85 years of research and concluded that cognitive ability tests were the single best predictor of job performance across different occupations and job levels. They found that cognitive tests had a higher validity (predictive accuracy) than other selection methods, including interviews.
Another 2001 study out of University of Manchester’s school of management compared cognitive ability tests with structured interviews as predictors of job performance. They found that cognitive tests had a stronger relationship with future job performance than interviews, indicating that cognitive abilities played a more significant role in job success.
Not only are cognitive tests more accurate predictors of job performance, they also save companies a ton of time.
If you find this depressing, you’re not alone. It seems reductive and impersonal, which probably explains why so many have hung on to the interview. Luckily, Automatic provides a template for a highly personal and engaging hiring process that avoids the bias pitfalls intrinsic to the old-fashioned interview. They call it “auditions”.
Here’s how the process is described in a Harvard Business Review article from 2014:
“After an initial screening, promising candidates are required to work with the company for three to eight weeks (with pay), performing real tasks that are closely related to the jobs they’re applying for, and working alongside the people who will be their colleagues if they’re hired. They can size up the company as it evaluates and provides feedback to them, benefiting all concerned.”
Mullenweg expands on the process and filosophy behind it in more detail in the interview:
“The most significant shift we’ve made is requiring every final candidate to work with us for three to eight weeks on a contract basis. Candidates do real tasks alongside the people they would actually be working with if they had the job. They can work at night or on weekends, so they don’t have to leave their current jobs; most spend 10 to 20 hours a week working with Automattic, although that’s flexible. (Some people take a week’s vacation in order to focus on the tryout, which is another viable option.) The goal is not to have them finish a product or do a set amount of work; it’s to allow us to quickly and efficiently assess whether this would be a mutually beneficial relationship. They can size up Automattic while we evaluate them.”
“When the process is over, all the people involved have a great sense of whether they want to work together going forward. At that point the final step is an interview with me. (Even as our company has grown, I continue to spend at least a third of my time on hiring.) I’ve changed my interview style dramatically over the years, to stay in the spirit of the tryout process. I conduct interviews via text-only Skype chats or instant messaging. I don’t know the gender or ethnicity of anyone I interview; I see only the words on the screen. It’s as close to a double-blind process as you can get. I’m looking mainly for passion and cultural fit. Of the people who make it to the final interview, 95% get a job offer—a testament to the effectiveness of our approach.”
Automattic’s hiring approach seems to provide the best of both worlds, for the most part. One area it fails where simple cognitive tests succeed is time. The “audition” or “tryout” process is obviously time intensive. Although I’m sure Mullenweg would insist that it saves the company time longterm by finding more talented and dedicated candidates. Another thing is that this format works best for remote companies. Candidates who are still working full-time jobs can find time to do their tryout work at any time. This isn’t the case at many companies, including utilities.