5 major challenges faced by data scientists

When the Harvard Business Review proclaimed that data scientists were the sexiest job of the 21st century, they did not say anything about the job is free of challenges. In that case, they would have called it the easiest job of the century. Every job comes with its own set of challenges and data scientists face a bunch of challenges that are often insurmountable.

Born of a challenge

Jonathon Goldman, the person whose work at LinkedIn is extensively featured in the Harvard Business Review’s famous article on data scientists, a Ph.D. in physics from Stanford, was faced with a challenge. The people joining LinkedIn were not really visiting much of it or building any significant network. This concerned him and he decided to use background data entered by the members to find people they may know. He implemented networking ideas like triangle closing, where if person A knows person B and person C, then there is a good chance that B and C know each other. He worked on these ideas to formulate ‘people you may know’ ads. This worked like a charm.

Data science always starts with a problem, a challenge. We can say there is hardly a need for data scientists without challenges. While this sort of challenge is welcome, some challenges can really threaten the work of data scientists.

Dirty data

A survey reports that every data scientist faces three major problems on average. 36% of all the responding data science professionals claimed that dirty data is their worst challenge.

The amount of unstructured data that a data scientist has to handle is quite huge. Moreover, data can be misleading, biased, situation-specific, and largely counter-intuitive. These are factors that can make or break a data science venture depending on the correct interpretation. In fact, a significant change in collective lifestyle, like the one experienced during the lockdown, can render predictive models dysfunction as the existing parameters of interpreting data to become obsolete.

Lack of skilled professionals

30% of respondents have marked this as a serious challenge. Now, you may think that a lack of skilled professionals means less competition for those who are already placed in the industry. Well, of course, it does not work like that. Data science is a team sport. In order to find accurate results with efficiency, you need wingment who can not only help you clean the data and prepare the models but also provide a fresh perspective.

Young data science professionals need to work with seniors to learn faster and be more effective. Similarly, if you are working in a team where every member uses Python for data science, things will automatically be better for you.

Results being ignored by stakeholders

Well, this is just as frustrating as it sounds. 18% of the respondents face this issue. It is hard work, preparing a model, cleaning data, running algorithms, and drawing insights. When the management does not use the results and take action in accordance, the work is lost. It results in a mindless waste of resources including time.

Lack of a clear goal

Setting up a data science team without being sure about the purpose of the same has become a corporate trend. Data scientists cannot work without specific questions being pointed at them. It is like going to a doctor and asking her what your problem is. This challenge is faced by 22% of the respondents. This sure is one of the major reasons why most companies implementing data science end up being disappointed with the results.

Explaining their work to others

There is a reason why communication skills are held so highly in the list of skills that data scientists should possess. As a data science professional, you will have to interact effectively with your work and plans with people who are completely unfamiliar with the data science acumen. 16% of respondents are bugged by this challenge. This is exactly why data literacy across the company is needed. A lot of companies help their employees upskill with big data training to avoid this issue. A data science professional with excellent data visualization skills can somewhat mitigate this challenge.

Well, these are just some of the many challenges faced by data scientists. There are other serious issues like unfavorable policies, corporate politics, and lack of funding. The good part is that none of these problems are related to the technology itself, these are mostly caused by faulty company policies and lack of awareness.  When it comes to business-related challenges data scientists thrive among them; as I said earlier, there would be no data science without challenges.

Leave a Reply

Your email address will not be published. Required fields are marked *