Long before the invention of modern technologies, data were written down in books or memorized, and later on, as the amount of data increased, people slowly started using the computer to store and process information. As the usage of technology and the internet rises, the quantity of data also increases exponentially, which led to the introduction of the concept of big data, where large sets of data are analyzed and processed.
Through this article, we will discuss what is big data and its characteristics also, we will see what is the future scope of big data.
Evolution of Big Data:
When we realized that data could no longer be contained in books or physical copies, we started storing information in computers, but with the increase in data production, they could no longer be stored in normal systems, So we needed means to store the produced data, process and retrieve it when needed according to the requirement. Thereby around the 1960s and 70s, the idea of RDBMS (Relational Database Management System), where data were stored, processed, and easily retrieved as needed, was introduced.
During the initial days, data was simple we could categorize those into column format, which could be managed by RDBMS, but as new types of data such as audio, video, etc. emerged, this file couldn’t be stored in RDBMS or any other traditional tools, Thus to save a huge amount of complex data the concept of big data was introduced.
Around 2005, the amount of data released by Facebook, YouTube, and other online platform was enormous. This led to the creation of Hadoop. It is nothing but an open-source framework created to analyze and store big data sets. During the same time, NoSQL also started to gain popularity.
During later years, the quantity of data also skyrocketed, and these open-source frameworks like Hadoop and the recently added spark framework made these gigantic data much simplified, easier to work with, and stored cheaply. These technologies have very less human involvement.
What Is Big Data and Its Characteristics?
Big data is data that contain different verities and a huge amount of complex data sets from multiple sources, but what decides whether the data can be defined as big data or not? For this, we follow the 5V rule
Volume
It stands for the quantity or size of data that needs to be stored. In other words, we can say that the volume of data being generated every second will be larger than terabytes and petabytes, and as the amount of data increases in the further years, the data unit considered will also be larger.
Variety
In variety there are three types structured, semi-structured, and unstructured data. Structured data are from sources like product databases, contact lists, and excel sheets. Semi-structured data have sources like HTML code, graphs and tables, and emails, and for unstructured data, sources are social media, voice files, IOT, etc.
Velocity
Velocity or speed at which data is progressing or changing is also considered. It also says about the speed at which the data need to be generated, processed, and collected. Velocity is important because if a certain issue is found or for an immediate requirement, it should be able to do the action quickly.
Value
The business value of the data, or usefulness of the data given, is termed as value.
Veracity
The terms bias, noise, and abnormality are used to describe data veracity. Incomplete data, errors, outliers, and missing numbers are also examples .it deal with the consistency and quality of data.
Types of Big Data and Its Sources
There are mainly 3 types of big data which are:
Structured:
• Here, the data would be precise, factual, and highly organized.
• They can be displayed in rows and column
• Would be as numbers, strings, or dates.
• Less storage required
• Easier to manage and protect
• Analysis would be quantitative
• Storage and searching are easier (ex: MySQL-use SQL commands to insert, select, delete, etc.)
Structured data sources are OLTP systems, spreadsheets ( excel sheets), online forms, sensors such as GPS or RFID tags, SQL databases, network, and web server logs
Semi-structured:
• Loosely organized into categories
• Would usually be emailed by inbox, sent draft, spreadsheet, etc.
• In tagged text format, it is stored in relational databases
• Abstracts and figures are mainly used to represent semi-structured data
Semi-structured data sources are CSV files (Comma separated value-saved as structured value separated by commas), HTML, NoSQL database
Unstructured:
• Data does not have a predefined structure and can have any form.
• They cannot be displayed in rows, columns, and relational databases.
• Would be as images, audio, video, emails, word processing files etc.…
• More space is needed for storage.
• More protective measures need to be taken to store and secure the data.
• Here, the analysis would be qualitative.
• Need complicated measures to store and process data (Hadoop- which is an open-source framework for storing and processing big data sets)
• Data lakes are used for storage (support various datatypes)
Unstructured data sources are audio files, social media, emails, images, video files, etc.
Challenges Faced by Big Data
The four main challenges faced by big data are storage, quality, validation, and collection of data. therefore,
• Security issues: A system can malfunction as a result of information theft, DDoS attacks, ransomware, or other malicious behaviors that can occur online or offline.
• Ethical issue: ownership and privacy need to be ensured
• Storage: Big data is difficult to store, especially due to its unstructured type.
• Collection of data: data need to be efficiently transported to the storage system without variations.
• Unintentional misuse
What Is the Future Scope of Big Data?
Big data is an ever-progressing domain that will only increase its importance in the coming future as more and more data are being created and used. Unstructured data production is increasing enormously. Social media, like Facebook (Meta), etc. are depending on bigdata as to store their massive data
Bigdata is involved in almost all verticals, including Retail, Bank, E-commerce, Healthcare, Telecom, etc. Not only these data need to be stored, but they also need to be processed, and as per the necessity and quick data retrieval should be possible
Conclusion
The importance of big data will increase day by day, and it is important to find out newer methods to find and process data. So the demand gets higher and higher, making it one of the most important domains of all time.
I hope the article helped in understanding what is big data and its characteristics also, we read about what is the future scope of big data.
