Lately, it seems that Big Data and its next of kin “BI” pop up in nearly every industry conversation about business opportunities and technology challenges. Yet with every megatrend, there is a gap between perception and reality, and that’s clearly the case with Big Data.
On one hand, many organizations think that simply by harnessing Big Data, they’ll finally be able to answer elusive questions about their business and unlock the value of their information. At the same time, others are so fixated on assembling all the Big Data technologies, tools and resources they think they need, that they’ve lost sight of where and how to get started.
Crossing the Big Data divide between truth and reality first requires some myth busting. Quentin Hardy’s recent New York Times blog, “Why Big Data is not Truth,” asserts that while the term suggests the assembling of many facts to create greater, previously unseen truths, there are plenty of opportunities for misinterpretation or miscalculation.
Hardy shares insight from a speech by Microsoft researcher Kate Crawford where she cites the “six myths of Big Data,” including the misconceptions that Big Data is new, objective and anonymous. According to Crawford, a paper visualizing Big Data surfaced in 1997—well before it became a widespread social phenomenon. She also addresses the importance of applying context references to data sets, particularly where people are concerned, while also recognizing the potential of human error in creating those data sets in the first place.
In addition to Crawford’s Big Data myths, I have a few of my own. Last month, I participated in a podcast with analyst Dana Gardner, the summary of which appeared on ZDNet, “Want a data-driven business culture? Start sorting out the BI and big data myths now.”
Let’s start with what I think is one of the biggest misconceptions: that Big Data applies only to massive volumes of data. Companies of every size are experiencing monumental data growth, yet many are getting hung up on sheer data volume as defining what is and isn’t Big Data. I would counter that it’s not about size—highly valuable answers to burning business questions can be found in small elements of small data sets by looking at combinations and patterns within the data.
Also worth busting is the belief that Big Data is limited to only large organizations. No way. Because large or small, one fact remains: it’s better to make decisions based on data than intuition or gut feelings. While we’re preoccupied with Big Data, the takeaway is to become more data-driven. Even the smallest companies can apply a data-driven mindset to leverage methodologies behind Big Data and answer important business questions.
Another reality check is that companies don’t need a full bench of data scientists to become more data driven. Sure, taking advantage of Big Data can involve complex algorithms and sophisticated analysis. But, there are varying degrees and different ways to tackle Big Data problems, especially if you apply a combination of programming skills, some packaged software offerings and open-minded thinking to accomplish your objectives.
There’s also confusion about whether an organization needs to be super-proficient in Business Intelligence to benefit from Big Data. Not so. Today, BI is much more about recognizing patterns and relationships within data. You can become data driven and find patterns in your data without mastering traditional BI. Think of it as a new way to look at your data. The key is to approach from the business and IT perspectives together.
Finally, don’t think that using Hadoop guarantees Big Data success. To be successful, you have to start with the business objectives. Sure, using new technologies like Hadoop and predictive analytics can accelerate finding the answers you’re seeking, but it still comes back to having a mindset that data can help you make better business decisions. It’s imperative to start with the business requirement instead of the technology or you might find yourself perpetuating some of today’s most common Big Data myths and misconceptions.
Have any Big Data myths you would like to try on me? If so, drop a line to darin.bartik@software.dell.com.