Why is it so hard to recruit top data scientists?

Posted by | June 28, 2017 | Recruiting

It’s some combination of these reasons :

1. You, or your client aren’t paying enough
2. Your, or your client’s expectations are too high or are disjointed from reality
3. You, or your client’s aren’t willing to let anyone learn things on the job

Ask yourself :

1. What qualifies someone as a “top” data scientist?

My guess is that everyone has a different answer here because there isn’t any agreed on definition of what a data scientist does, nor is there any
agreed on way of valuing a data scientist.

Consider academic research for a moment. How do you value a researcher? They may have 10 failed experiments for every 1 success. The 10 failures teach you a lot about your problem, and the 1 success can open whole new avenues of research. In science sometimes things that aren’t immediately applicable open up whole new industries later.

That’s sort of how data scientists operate–on the fringe of what your company knows and what they think is possible. There will be long stretches of time where a data scientist is researching solutions to a problem before they solve it and the research can be just as valuable as the end product or problem solved.

You can’t apply the same metrics to a data scientist that you apply to a developer if you want to get a good data scientist.

2. Do you actually need such a person?

A lot of value can get added to your data just by munging it and/or using simple models. That can be done by a data analyst or statistician. You don’t necessarily need someone with a PhD in machine learning to answer business questions or create new data products.

Many companies don’t even have the foundation needed to benefit from a data scientist fully, and instead might be better off hiring an analyst and/or a solid “big data” engineer first. Or, they might want to shift focus and build a data science team.

3. Does such a person actually exist?

Data scientists are not usually expert database developers, software engineers, statisticians and PhD academic researchers. You can’t replace a “data science” team with one data scientist. Likely you wont even find such a person.

Further to that, a data scientist is usually a generalist developer with well above average mathematical knowledge. The skill set is general and abstract. You probably aren’t going to easily find a data scientist that knows 5 different frameworks and 5 languages inside and out. In fact, that would be close to the opposite of what I’d expect out of a good data scientist and more of what I’d expect out of a higher-tier engineer.  On the scale of pure application to pure theory, data scientists are halfway in-between.

Data scientists have an abstract framework to think about problems in, namely mathematics. In math you can scale in to, or out of smaller problems within a larger picture using a common set of tools. You won’t necessarily remember every detail because it’s impossible, but you have the tools to dive into the details if you need to. A good data scientist will retool their brain for the problem at hand.

Basically, I’d say that you shouldn’t look for someone with 10 years of experience in A, B, and C languages and X, Y, and Z frameworks. Look for mathematical aptitude and understanding instead. Look for academic research experience. Look for someone that knows data structures and algorithms rather than language B or framework Z. They can learn the rest when they solve problems, and you aren’t hiring an engineer. If your job description is closer to that of an engineer, then change the title and hire an engineer.

4. Would it be possible to relax your expectations and let someone develop into the candidate you want over time?

Too many companies want to hire someone that is ready to hit the ground running or has a very specialist skill set. That’s effectively the same thing as saying “I want to poach someone from my competitor and pay more money, or I’ll just wait X years for someone to catch up to my expectations”.

People meeting many employer’s expectations either don’t exist, or they already have jobs and they are paid well. You won’t find such a person until your competitors make such a person, and/or you’ll have to poach them from a competitor (in the labor market) which means you have to offer better incentives.

Those that exist may not even be looking for new opportunities which makes it harder to find them. To add to that problem, good “data science” candidates may not even have that title even if they have most of the qualifications.

Simply put, you could probably hire someone that doesn’t meet all your requirements and then train them to be the candidate you want over 3 months, rather than wait for 6 months to a year to fill a position and also have to pay a larger salary on top of that. Keep in mind part of data science is research–they will learn what they need to learn to solve a problem.

As I mentioned before, you might also have better luck hiring 2-3 people to make up a data science team. Hire an engineer and a statistician for close to the same price you’d pay a PhD with 5 years of data science experience and get the added benefit of having twice as many person-hours available to solve a problem.

5. Are you offering compensation that someone fitting your expectations would find attractive?

A person with 5 years of experience and a PhD in Machine Learning probably already has a sweet deal going with their current employer. You need to beat it, or they won’t be interested. It’s unfortunately not an employer’s market right now.

672 total views, 1 today

Leave a Reply

Your email address will not be published. Required fields are marked *