TSM - Big Data - Data Representation

Cătălin Roman - Lead Software Architect @Frequentis

The previous issue of the magazine talked about the trend of Big Data in the software industry. The current article presents the fundamentals of the technology that enables the storage and query of high volume of data. To get a better picture about the issues around Big Data, let’s take a look at an example. Not long ago there was this statistics showing up on the Internet: eBay collects about 20TB of user generated data daily. Facebook collects 20TB of user generated content and it generates another 10TB from daily analytics. The Insights service is powered by processing about 15PB of data. Google processes 20PT of data daily. It is pretty much clear that not only hardware is needed, but also a new kind of software concept is required to deal with such a data volume.