What Is Big Data ?

Big data is data that contains greater variety arriving in increasing volumes and with ever-higher velocity. This is known as the three Vs (What is big data, n.d.). 

Put simply, big data is larger, more complex data sets, especially from new data sources. These data sets are so voluminous that traditional data processing software just can’t manage them. But these massive volumes of data can be used to address business problems you wouldn’t have been able to tackle before. Big data is analyzed for insights that lead to better decisions and strategic business moves (What is big data, n.d.). 

The Three Vs of Big Data

1. Volume 

The amount of data matters. With big data, you’ll have to process high volumes of low-density, unstructured data. This can be data of unknown value, such as Twitter data feeds, clickstreams on a webpage or a mobile app, or sensor-enabled equipment. For some organizations, this might be tens of terabytes of data. For others, it may be hundreds of petabytes (What is big data, n.d.).

2. Velocity 

Velocity is the fast rate at which data is received and (perhaps) acted on. Normally, the highest velocity of data streams directly into memory versus being written to disk. Some internet-enabled smart products operate in real time or near real time and will require real-time evaluation and action (What is big data, n.d.).

3. Variety 

Variety refers to the many types of data that are available. Traditional data types were structured and fit neatly in a relational database. With the rise of big data, data comes in new unstructured data types. Unstructured and semistructured data types, such as text, audio, and video require additional preprocessing to derive meaning and support metadata (What is big data, n.d.).

The Value and Truth of Big Data

The value and truth of Big Data Two more Vs have emerged over the past few years: value and veracity (What is big data, n.d.). 

Data has intrinsic value. But it’s of no use until that value is discovered (What is big data, n.d.).

A large part of the value tech companies offer comes from their data, which they’re constantly analyzing to produce more efficiency and develop new products (What is big data, n.d.).

Finding value in big data isn’t only about analyzing it. It’s an entire discovery process that requires insightful analysts, business users, and executives who ask the right questions, recognize patterns, make informed assumptions, and predict behavior (What is big data, n.d.).

Big Data Use Cases

1. Product Development

Companies like Netflix and Procter & Gamble use big data to anticipate customer demand. They build predictive models for new products and services by classifying key attributes of past and current products or services and modeling the relationship between those attributes and the commercial success of the offerings. In addition, P&G uses data and analytics from focus groups, social media, test markets, and early store rollouts to plan, produce, and launch new products (What is big data, n.d.).

2. Predictive Maintenance 

Factors that can predict mechanical failures may be deeply buried in structured data, such as the equipment year, make, and model of a machine, as well as in unstructured data that covers millions of log entries, sensor data, error messages, and engine temperature. By analyzing these indications of potential issues before the problems happen, organizations can deploy maintenance more cost effectively and maximize parts and equipment uptime (What is big data, n.d.).

3. Customer Experience 

The race for customers is on. A clearer view of customer experience is more possible now than ever before. Big data enables you to gather data from social media, web visits, call logs, and other data sources to improve the interaction experience and maximize the value delivered. Start delivering personalized offers, reduce customer churn, and handle issues proactively (What is big data, n.d.).

4. Fraud and Compliance 

When it comes to security, you’re up against entire expert teams. Security landscapes and compliance requirements are constantly evolving. Big data helps you identify patterns in data that indicate fraud and aggregate large volumes of information to make regulatory reporting much faster (What is big data, n.d.).

5. Machine Learning 

We are now able to teach machines instead of program them. The availability of big data to train machine-learning models makes that happen (What is big data, n.d.).

6. Operational Efficiency 

With big data, you can analyze and assess production, customer feedback and returns, and other factors to reduce outages and anticipate future demands. Big data can also be used to improve decision-making in line with current market demand (What is big data, n.d.).

7. Drive Innovation 

Big data can help you innovate by studying interdependencies between humans, institutions, entities, and process and then determining new ways to use those insights. Use data insights to improve decisions about financial and planning considerations. Examine trends and what customers want to deliver new products and services. Implement dynamic pricing (What is big data, n.d.).

8. Improving Healthcare

Data-driven medicine involves analysing vast numbers of medical records and images for patterns that can help spot disease early and develop new medicines (Bernard Marr, n.d.). 

9. Predicting and Responding to Natural and Man-Made Disasters 

Sensor data can be analysed to predict where earthquakes are likely to strike next, and patterns of human behaviour give clues that help organisations give relief to survivors. Big Data technology is also used to monitor and safeguard the flow of refugees away from war zones around the world (Bernard Marr, n.d.). 

10. Preventing Crime

Police forces are increasingly adopting data-driven strategies based on their own intelligence and public data sets in order to deploy resources more efficiently and act as a deterrent where one is needed (Bernard Marr, n.d.).

Big Data Challenges

First, big data is…big. Data volumes are doubling in size about every two years. Organizations still struggle to keep pace with their data and find ways to effectively store it (What is big data, n.d.).

Data must be used to be valuable and that depends on curation. Clean data, or data that’s relevant to the client and organized in a way that enables meaningful analysis, requires a lot of work. Data scientists spend 50 to 80 percent of their time curating and preparing data before it can actually be used (What is big data, n.d.).

Finally, big data technology is changing at a rapid pace. A few years ago, Apache Hadoop was the popular technology used to handle big data. Then Apache Spark was introduced in 2014. Today, a combination of the two frameworks appears to be the best approach. Keeping up with big data technology is an ongoing challenge (What is big data, n.d.).

How Big Data Works

1. Integrate 

Big data brings together data from many disparate sources and applications. It requires new strategies and technologies to analyze big data sets at terabyte, or even petabyte, scale (What is big data, n.d.).

2. Manage 

Big data requires storage. Your storage solution can be in the cloud, on premises, or both. You can store your data in any form you want and bring your desired processing requirements and necessary process engines to those data sets on an on-demand basis. The cloud is gradually gaining popularity because it supports your current compute requirements and enables you to spin up resources as needed (What is big data, n.d.).

3. Analyze 

Get new clarity with a visual analysis of your varied data sets. Explore the data further to make new discoveries. Share your findings with others. Build data models with machine learning and artificial intelligence. Put your data to work (What is big data, n.d.).

Big Data Concerns

1. Data Privacy

The big data we now generate contains a lot of information about our personal lives, much of which we have a right to keep private. Increasingly, we are asked to strike a balance between the amount of personal data we divulge, and the convenience that big data-powered apps and services offer (Bernard Marr, n.d.).

2. Data Security

Even if we decide we are happy for someone to have our data for a particular purpose, can we trust them to keep it safe? (Bernard Marr, n.d.)

3. Data Discrimination

When everything is known, will it become acceptable to discriminate against people based on data we have on their lives? We already use credit scoring to decide who can borrow money, and insurance is heavily data-driven. We can expect to be analysed and assessed in greater detail, and care must be taken to ensure no data discrimination (Bernard Marr, n.d.).

Edited by: 浪子

Bibliography

What is Big Data? (n.d.). Retrieved from https://www.oracle.com/big-data/guide/what-is-big-data.html

Bernard Marr. (n.d.). What Is Big Data? A Super Simple Explanation for Everyone. Retrieved from 
https://www.bernardmarr.com/default.asp?contentID=766