80% of the data scientist jobs in 2011 have not been filled due to the data scientist skills shortage.
A McKinsey study estimates that there will be 490,000 data science jobs in the US by the end of 2018 but there will be less than 200,000 skilled data scientists to fill these open data science jobs. According to this study, the demand for data scientist skills will exceed the supply by 50%.
On an average, there are only 23 students in a data science graduate class. With only 110 universities offering data science programs, the increasing demand for data scientists will pressurize the already deficient supply of talent, in the US.
According to a report by Expert Group of Future Skills Needs, there will be 21,000 data science jobs in Ireland by end of 2020.
UK has only 25 Master degree on data science programs available but the data science job market is accelerating at a gazelle’s pace, with 3000 open data science jobs available now.
CLICK HERE to get the Data Scientist Salary Report for 2016 delivered to your inbox!
The above facts and figures clearly depict that there is a talent crunch in the field of data science. With the shortage of data scientist skills worldwide, it is difficult for an organization to fulfil its dream of capitalizing on data science, as it is hard to find the exceptionally skilled person - who is a machine learning expert, a data engineer, a developer, a storyteller and a business analyst. Here’s a valuable piece of advice from industry experts at DeZyre, on how to build a data science team with candidates, who individually lack the comprehensive data scientist skills, but together as a team possess them all.
“You can find a great developer and a great researcher who has a background in statistics, and maybe you can find a great problem solver, but to find that in the same person is hard.”- said Stan Humphries, chief economist at Zillow.
Executive recruiters are struggling hard to find candidates who possess all the skills of a data scientist - Advanced Analytic Capabilities, Business Acumen, Creativity, Communication, Storytelling, Data Integration, Software Programming, and System Administration. As businesses struggle to hire the talent they need, an alternative to solving the data science skills gap is - building an effective data science team.
If you would like more information about Data Science careers, please click the orange "Request Info" button on top of this page.
It is important for the organizations to look beyond the definition of a UNICORN data scientist because the number of professionals who qualify for every skill requirement of the data science job specification - are few and far! With the scarcity of data scientists expected to increase in the coming years, the culture of building world-class data science teams is on the rise. 2016 is the best time, when organizations should focus on building a data science team instead of looking for a single person with the entire data scientist skills set.
“Rather than seeking out rare individuals who excel in all the areas that encompass data science, CIOs should build data science teams with complementary talents.”- said Bob Rogers, Data Scientist at Intel.
Organizations are looking to hire the brightest minds to support their big data endeavours. Unless it is a bigwig Silicon Valley organization, finding that one single person who possesses all the data scientist skills, in this field is difficult. A realistic approach is to – divide the job and conquer big data with a best-in-class data science team. For example, when a physicist undertakes a big data endeavour, he builds a team of people to design the required equipment, run experiments and do data analysis. Similarly, the responsibilities of a data scientist can be divided by building a data science team, instead of a searching for one superhuman who can do it all.
Here’s a good word from the DeZyre faculty and industry experts for candidates looking to pursue a career in data science. ‘Do not be put off if you see a data science job specification that has a huge qualification checklist, like Hadoop, Python, R, Spark, SQL, Tableau, Matlab, Machine Learning, etc. It does not cost anything to take a chance and apply for the job, even if you do not have any practical experience in these disciplines. If you are from a math or statistics background, are driven by data and keen about research, then go for it without a second thought.’
Data Scientists at organizations are made up of teams of people with diverse skill sets who outdo the crowd, in their respective areas of skill specialization. You don’t need to be a “unicorn” to land a top gig as a data scientist, as companies today are looking for candidates with major skill sets and basic comfort level in their own areas of expertise for working together as a team.
Even the biggest IT giants and small tech companies are adopting the culture of building effective data science teams, rather than hunting for that single person with data scientist skills -
- The largest professional network LinkedIn had one of the first data science teams, which helped build innovative data products like – “Who’s viewed your profile”, “People you may Know”, “People also Viewed”, “Jobs You Might Be Interested in”, etc.
- 12 Lab’s Applause, is a data science powered weight loss application, that uses behavioural psychology and data science algorithms to make personalized suggestions and predictions to lose weight. They did not have any data scientists when they started building Applause. Instead of looking for ‘unicorns’ in the world of data science, they built an in-house data science team. In one year, 12 Labs has built scores of data products that even the top organizations in the fitness space have not been able to.
- American International Group Inc. has a data science team of business analysts, project managers, engineers, statisticians and systems architects for its Property and Casualty Business.
- Monster Worldwide has built a data science team of statisticians, computer scientists, data crunchers, navigators and business analysts who can explain the outcomes to managers.
- Data Science Team at Facebook consist of PhD’s in Natural Language Processing and Machine Learning to entry-level Web Product Engineers. Each member in the team applies some technique of data science, to enhance the dataset with the ultimate goal of building a better data-driven user experience for all the users.
The size of a data science team varies based on the projects –it can be a handful of people for small tactical data science projects and can extend to 20+ people for longer, ongoing, analytic projects. A small data science team can consist of one or more software engineers and quantitative analysts who possess expertise in Hadoop, for big data processing and writing scripts in any of the data science programming language, like Python and R for data preparation, data integration, data cleaning and data analysis. It can also have a data/systems architect to host data on various systems and make sure that the systems communicate in an uninterrupted manner.
Larger data science teams can have specialized and skilled professionals for each role – quantitative analysts who can dig deep into the data to find insights, software programmers to write programs for data preparation, that can be used for analysis, data visualization experts who can tell stories from data insights in an easy and understandable format and project managers who can mastermind the efforts of all these skilled personnel.
Building an Effective Data Science Team
Here are some of the job roles and responsibilities that are a must have in any data science team –
The “Sherlock Holmes” i.e. the data detective of a data science team who has knowledge of data science programming language like Python and R, SQL, Excel, NoSQL, etc. These skilled personnel have intimate understanding of the data which leads to asking right questions for the business. The deep insights provided by a data analyst helps the data science team drive enterprise wide decision making, for profitability.
Having data analysts with relational database experience in a data science team, adds value as they easily adapt to working with new distributed tools like Spark SQL and Hive. If the organization’s data science team, demands the analyst to be engaging with the stakeholders, then having a data analyst on the data science team with knowledge of Microsoft Excel is a plus.
Data Science Product Manager
A data scientist is to discoveries as a product manager is to innovation. Product management is vital in data science to stay at-par with the market requirements. Having a good product manager in the data science team, helps connect the business problems with the data science findings. It is a known fact that there is a shortage of data scientists but there is an even more dire shortage of product managers, who have a good business acumen and understanding of data science concepts, to establish a successful connection. Having a member with good product sense on the team, who can give a clear picture on what the product will look like, will take the organization a long way, in its data science investments.
Data Science Product Managers helps prioritize projects to maximize ROI and eliminate any hindrances coming in the way of the data science team so that they operate efficiently. Data Science Product Managers defines the product vision, translate business problems into user stories and emphasize the developers to build data products depending on these stories.
Data Science QA
Software Quality Assurance Engineers are used to test the quality of a product or a service but analytics QAs are a little different. They need to ask “Is this number correct?” or “It could have been this number instead of that?” It is better to hire a specialist who has strong math skills and is able to test data quality. Even a minor data quality issue can lead to meaningless results and it is extremely important to have a specialized QA in the data science team, who can understand the data and engineer it throughout the modelling process. Knowledge of RDBMS, JSON, working knowledge of RESTful API data sources and HTTP are some of the essential skills for a data science QA.
“The bottom layer, the foundation. They are the ones who play with Hadoop, MapReduce, HBase, and Cassandra. These are people interested in capturing, storing, and processing this data… so that the algorithm people can build models and derive insights from it.”- said Dr. Michael Wu is chief scientist of Lithium Technologies.
A data engineer is a software engineer by trade with experience in diverse technologies like Java/Scala/Python, Unix Scripting and SQL. He/she is an asset to a successful data science team. Data engineers should possess knowledge of various distributed programming frameworks like Hadoop or Spark Clusters, great understanding of data infrastructure and architectural concepts.
They are many other job roles that can be added to the data science team - UI designer, Machine Learning Experts, etc. based on the organizational goals. A Data science team’s composition and structure along with a well-built strategy for prioritizing high-value data science projects, can boost the ROI of a business. Every organization can modify the above approach of building an effective data science team, based on their goals and organizational culture to make the most out of their data science investments.
Having built a data science team, it is good to inculcate the habit of encouraging team members to pick up skills from others in the team. This helps the organization create flexibility within the data science team, that is resistant to attrition. When each person in the data science team learns new skills from their peers, fostering their careers - have more reasons to stay put.
Ken Blanchard, a popular American author once said “None of us is as smart as all of us.” Businesses who build multi-skilled data science teams are likely to achieve more from that synergy. Data Scientists or a Data Science Team? The answer to this completely depends on business requirements and how much it can afford to capitalize on the growth of the data science team.
With increasing demand for data scientist skills and continued talent crunch in the industry, organizations are more inclined towards building effective data science teams but the question as to who is the best fit to manage a data science team still remains unanswered.
We invite the big data community to chime in with their comments on who can be an ideal fit for managing a data science team effectively.