Most of the software engineers working in Big Data domain but they don’t exactly know why we are going for big data technology. So, let’s talk about Why Do We Need Bigdata in this post.

Why Big Data?

• To process huge amounts of data which traditional systems (like your pc/ laptop) are not capable of processing.
• To process huge amounts of data we need to store it first.

Example: Suppose we need to store 150 TBs of data, can our traditional system/ laptop which have 1 TB capacity store these huge amounts of data?  No Right.

Big Data System requirements:
Storage, Processing, and Scalability are the three major components to build a good Big Data system.

• Storage => To store huge amounts of data that traditional systems are not capable of storing.
• Processing =>Process huge amounts of data in an efficient and timely manner that traditional systems are not capable to process.
• Scalability => Scale easily to accommodate growing requirements that traditional systems are not capable to handle.

⭐⭐Two Ways to Build a Big Data System:⭐⭐

• 📍Monolithic => one powerful big system with lots of resources like Memory(Ram), Storage(Hard-disk), Compute(CPU))
• 📍Distributed => here many commodity systems come together to solve a purpose. each system is technically known as node and all the nodes together make a cluster.

⭐⭐⭐Monolithic or Distributed systems which one is best for Big Data Design?⭐⭐

📍Monolithic System:

1. A single powerful server.

2. This will support Vertical Scaling. Hard to add resources after a certain limit.

3. This is not horizontally scalable that is why this will not easily accommodate the growing requirements.

4. 2X(times) Resources is not providing the 2X(times) performance after a certain limit.

📍Distributed Systems:

1. Many small and cheap computer’s come together to act as a single entity.

2. This will support Horizontally scalability.

3. Easy to add resources like Ram, Hard-Disk and CPU etc.

4. 2X(times) Resources providing the 2X(times) performance

That is why all good big data systems are based on Distributed Architecture.

References:  Why Do We Need Bigdata?