Big-data, data analysis, and strategic decision-making

Big data refers to extremely large and complex datasets that traditional data processing applications cannot adequately handle. The concept of big data has evolved from simple data collection to sophisticated analysis that informs strategic decisions across industries. 

Historically, organisations relied on small, structured datasets stored in traditional databases. Analysis was often retrospective and limited in scope. Advances in digital infrastructure, digital devices, internet connectivity, and IoT (Internet of Things) have dramatically increased the volume and types of data available. Today, organisations can collect vast amounts of structured and unstructured data from diverse sources, enabling more comprehensive and predictive analysis. For example: 

These examples illustrate how modern organisations harness large-scale data to drive innovation, efficiency, and customer engagement.  

Why it matters 

• Enables data-driven decision-making, replacing intuition-based approaches 
• Provides competitive advantage through pattern identification and trend forecasting 
• Supports innovation by revealing insights that might otherwise remain hidden 
• Enhances customer experience through personalisation and targeted services 
• Improves operational efficiency by identifying bottlenecks (a situation that causes delay in a process or system) and optimising processes

Big data is often characterised by 3 key dimensions: 

1. Volume: The sheer quantity of data generated and collected. Modern organisations deal with petabytes (1,024 TB/~1 quadrillion bytes) or exabytes (1,024 PB/~1 quintillion bytes) of data from various sources, including social media, sensors, transactions, and more. 
2. Velocity: The speed at which data is generated, collected, and processed. Real-time data streams from social media, financial markets, and IoT devices require rapid processing capabilities. 
3. Variety: The diverse formats and types of data, including: 
   1. Structured data i.e. relational databases, spreadsheets) 
   2. Semi-structured data i.e. Extensible Markup Language (XML) files, JavaScript Object Notation (JSON) files 
   3. Unstructured data i.e. emails, videos, social media posts, audio files 

Data is applied across various industries and functions, some examples include: 

• Business and Retail: Customer behaviour analysis, inventory management, supply chain optimisation 
• Healthcare: Patient outcome prediction, treatment personalisation, disease outbreak monitoring 
• Finance: Fraud detection, risk assessment, algorithmic trading 
• Transportation: Route optimisation, predictive maintenance, traffic management 
• Government: Policy effectiveness evaluation, resource allocation, public service improvement 
• Education: Student performance analysis, personalised learning, institutional efficiency 

After looking at these examples, can you think of any ways in which big data is used in the industries you are interested in?