I read recently that data is now considered “the new currency”. No kidding, I thought! Organisations that are seeking a competitive advantage are finding their edge in the data they capture and process.
We no longer talk about just a simple data warehouse. Instead, we’re talking about data lakes, which are a repository for all our unstructured and semi-structured data, from numerous sources.
Let’s think about a Travel company and the type of data it might store. It will capture data from customer bookings, social media activity, website logs, and industry trends. The data stored in their data lake may include:
Transactional data, such as the date and time of a booking, the destination, and the price.
Social media data, such as how many times the company has been mentioned on Twitter, and how many ‘likes’ it’s most recent Instagram post attracted.
Website logs, such as the number of visits, most popular times, user activity, and errors.
But what is the best way to store this data? You need to choose the right type of database and this depends on the type of data and how you need to process it.
As an example, social media platforms such as LinkedIn query huge amounts of connected data in order to make recommendations about people or pages that you may be interested in. Traditional relational databases, which store data in rows and columns within related tables, are not designed for these types of queries, and become slow and cumbersome to use.
Increased demand for availability, flexibility, and performance, together with a shift towards cloud providers such as AWS and Microsoft Azure, has created an eco-system consisting of many different types of database that are optimised to deal with specific types of workload.
As a result, we now have a seemingly unlimited amount of options and services when it comes to storing our data. Here’s a brief rundown of some of these options and the scenarios and common use-cases.
Relational
Key features
Referential integrity, ACID transactions, Fixed Schema
Examples
Common Use Cases
Key-value / Document
Key features
Flexible schema, high-throughput, low latency read/write, highly scalable
Examples
AWS DynamoDB, Azure Cosmos DB
Common Use Cases
Shopping cart, social, product catalogue, customer preferences, user profiles
In-memory
Key features
Microsecond latency
Examples
AWS ElastiCache, Azure Cache for Redis
Common Use Cases
Gaming leaderboards, real-time analytics, caching
Graph
Key features
Quick and easy navigation of complex relationships between data
Examples
AWS Neptune, Azure Cosmos
Common Use Cases
Fraud detection, social networking, recommendation engines
Leger
Key features
A complete, immutable, and verifiable history of all changes over time
Examples
Amazon QLDB, supported in Azure SQL and Microsoft SQL Server
Common Use Cases
Supply chain, health care, financial transactions
I hope this has given you a good introduction to the different types of database to consider for your data. If you’re interested in discussing this further, please get in touch with us at Koderly for a coffee and a chat!
