Desktop version

Home arrow Business & Finance

  • Increase font
  • Decrease font

<<   CONTENTS   >>

IT Manager Interviews

Eight interviews took place w'ith subjects responsible for, or part of, the hiring of big data resources. The intent was to gather depth around the big data hiring process, identify themes in the recorded conversations, compare the data collected to the survey data, and to understand what the experience is in this phenomena through the words of those living it. The interview' questions were similar to the survey questions but were worded to draw' out greater detail. This allowed the researcher to compare the answers, as a form of triangulation, and capture more detail around those questions.

An example of this was w'here the survey question asked what challenges the subjects had finding big data specialists. While a survey response may have been, “gap on skill sets,” the interview question asked the subjects to describe their challenges, which resulted in a longer explanation and examples of candidates w'ho had limited training or experience with technologies or interpersonal skills important to the organization that w'as hiring. These examples added to a greater understanding of the challenge to find big data specialists, which the survey questions were valuable for identifying themes but lacked in clarity of understanding w'hat w'as specifically meant by “skill sets.”

Specialist Interviews

Five big data specialists w'ere interviewed as w'ell. This third source of data added to the triangulation by verifying the data collected of the hirers of big data specialists by those who w'ere the seekers of big data specialist positions. New structures and themes came out of these interviews, particularly around the attainment and development of skills to be successful. The answers to what each of the big data specialists did were specific to the business and industry they worked in, but similar in nature, with some having a wider breadth of responsibility and others being very focused on a couple of roles. Each verified that they came to the businesses they are currently working for with a limited amount of knowledge in either their big data technical skills or the industry the business was in. All five grew their skill sets while on the job and received access to the software they wanted or needed to learn more about from their current employers.

Key Analysis & Findings

The purpose of this study was to learn what IT Managers are doing to meet the big data demands of their business as well as what quality criteria they are considering in big data specialist candidates. With ambiguity of the term, “big data,” and a growing number of technologies, methods, and approaches to big data analytics, trying to understand exactly what businesses are looking for and how they are meeting their big data demands was not understood from the previous literature. The findings also explain what hiring managers are looking for, what types of big data initiatives they are engaged in, what are the qualities difficult to find in big data specialists, and what they still need.

Theme 1: “ Lacking”

“Lacking,” “Dearth,” “Shortage,” “Scarcity”: The key finding from this study is that there are deficiencies in nearly every component of big data resources. Through thematic analysis, the research identified, in general, candidates either lack experience with the big data technologies or they lack business acumen. There is a lack of industry familiarity, and a lack of interpersonal, communication, or social skills often referred to as emotional intelligence. Advanced math and statistic skills are hard to find, but to find that data scientist looking for a job who can provide the strong math, coding, strong communication skills, and emotional intelligence that someone with that title is expected to have is nearly impossible.

Table Ю.7 highlighted the shortage of candidates with the skills or experience businesses are looking for as well as just a lack of candidates overall. A lack of effective recruiting was another that added to 19 of the 22 responses around the challenge of finding qualified candidates to fill the big data roles they have open as due to something lacking. As listed back in Table 10.1, there is a lack of educational institutions providing specific big data or data analytic degrees. There are limited certification programs and insufficient skill validation avenues.

Theme 2: “ Passion”

If there was ever a trend that espoused the phrase, “Where there is a will, there is a way,” big data and the people involved in it definitely have the will. Though many are lacking in either skill or experience, big data specialists are passionate about learning and improving. Identified through structural analysis, the emotions and level of excitement around solutions to the big data talent gap were evident across the board. More data were volunteered in this area than any other topic area. Big data specialists are tasked with providing answers and solving problems, so it is no surprise that when discussing a solution to the data scientists’ shortage, they were engaged and ready to provide their opinions.

Passion is also a key quality that hiring managers are looking for. One subject with a job opening listed “highly motivated” as the very first quality he is looking for in a candidate with “deep interest,” and “obsession” listed high on the candidate qualification list.

Theme 3: Soft Skills

While “big data” is defined by the data and the technologies surrounding it, knowing how to work with big data; managing, cleansing, developing statistical algorithms to run against the data, and analyzing the findings are only parts of what qualifies a data scientist for his or her role. When asked what qualities are most difficult to find and most ideal in big data specialist candidates, the answers that were provided included: story tellers, people who can speak in business terms, good communication skill, emotional intelligence, teaching skills, and leadership skills.

Theme 4: Technical Skills

The list of hardware, software, and services being added to the market around big data analytics does not look to be slowing down anytime soon. During the data collection process, many technologies were mentioned as valuable to the subjects’ big data initiatives. Candidates with such technology experience and expertise are in high demand: Java programming, Hadoop, machine learning, Scala, DevOps concepts, cloud computing, Python, R, Hive, Hbase, and Spark.


We are in the infancy of this fast-growing trend. The promise of such great insights and life-changing breakthroughs has organizations moving forward before a pool of trained and experienced experts are able to fill the desired roles. Education is by far the top opportunity but not just for technical information. Communication, statistics, coding, technology infrastructure, software, industry knowledge, business acumen, and critical thought are individual and collective areas for growth that educational institutions can focus on to develop more qualified big data candidates to the work force.

AI is another area of opportunity for businesses to consider. If someone with coding, communicating, and statistic expertise is not able to be found, perhaps computers can be programmed to do the coding and statistics, leaving the communication of the results left to those who can verify and validate them. Hundreds of businesses have started to provide software to the big data analytic world to provide answers faster and easier than manual activities.

Big data has forced a change in the traditional separation between technology- focused specialists and business leaders. The untraditional makeup of the data, from which businesses are trying to find value, requires someone who has a strong understanding of the business, its industry, and the type of information that can provide new benefits. Having that business-level understanding then requires the technical knowhow to process the data and glean the value from it. Understanding, collecting, and processing that data then require the ability to communicate it to business leaders so better qualified decisions can be made and actions can be taken. This study reveals the primary answer given around filling big data positions has been to find and train internal resources to become data scientists. With what is in demand, this makes sense: They know the business since they work for it, they have a familiarity with their employees, and many already have a proven maturity in a couple of the big data disciplines, whether that is coding, mathematics, or communication skills.

The second most common response to the question of where are IT Managers finding their big data resources is referrals. If internal resources fall under the umbrella of people known to the business, referrals are people trusted by people known to the business. In the beginnings of a new revolutionary change in how businesses make decisions, it is not a far-fetched concept that those who will be easiest to find, most trusted, and valuable will be those best known to the business’ leadership.

Is the data scientist someone who has a business acumen and picks up the technology skills allowing him or her to find business insights, or is the data scientists someone who has a technical focus and then learns the business? This question will be pondered by those who wish to be a part of developing future data scientists. Do universities make big data a part of their business programs, their information systems programs, or math and statistics programs? Perhaps, this is a new program all together. Regardless of where the big data specialists are developed, the communication, critical thinking, and emotional intelligence skills must be central to the development of these future data scientists.

<<   CONTENTS   >>

Related topics