Big Data

Big data refers to data collections whose volume, velocity (arrival speed), and variety exceed what a conventional database or spreadsheet on one machine can handle comfortably. Handling them usually means spreading storage and computation across many servers.

Why it matters

Clickstreams, logs, sensor feeds, and transaction records can reach billions of rows. Analyzing them reveals trends invisible in samples, but the infrastructure adds cost and complexity. There is no official size cutoff; what counts as "big" depends on your tools.

Example

A site logging every request might produce:

50 million requests/day x ~500 bytes = ~25 GB/day
~9 TB per year of raw logs

Querying that with distributed engines over a data lake or data warehouse is routine; opening it in a spreadsheet is not.

Common mistakes

• Adopting big data platforms for data that fits comfortably in one PostgreSQL or SQLite database.
• Collecting everything with no plan to use it, then paying to store it.
• Neglecting quality and governance as volume grows.

Related

• Data lake
• Data warehouse
• Data pipeline
• Data science

This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.