Hello friends,

In this session, we will discuss the characteristics, features, and advantages of scalarDB.

The course is mainly aimed at application developers. 

We will learn about the challenges faced by siloed databases.

We will also discuss the challenges in managing data integrity across microservices. Then, we will discuss the solutions that are possible with ScalarDB.

What is ScalarDB?

ScalarDB is a hybrid transactional and analytical processing engine that is HTAP engine for diverse databases it runs as middleware on the databases.

ScalarDB consolidates virtually any diverse database with ACID transactions and real-time analytics, simplifying the complexities of managing multiple databases.

It enables ACID transactions and real-time analytics across heterogeneous multiple databases, including NoSQL.

ScalarDB supports a variety of databases.

Relational databases that support JDBC, such as MariaDB, Microsoft SQL Server, MySQL, Oracle Database, PostgreSQL, SQLite, and their compatible databases, like Amazon Aurora, and YugabyteDB are supported.

 It also supports multiple NoSQL databases like Amazon DynamoDB, Apache Cassandra, and Azure Cosmos DB.

Challenges with Siloed Databases

Database siloing refers to a state in which a database is used only within each system, lacking integration with other systems, and becoming isolated.

Database silos occur when data is fragmented across departments or when different types of databases are used.

Systems optimized for the business processes of each department have the advantage of making it easier to focus on actions to achieve specific goals. 

As a result of each software being individually optimized, it becomes difficult to share data, and data silos arise. 

Even if a system is easy to use within a specific task or department, if it is not linked together, it can reduce the business efficiency of the entire organization.

What are the major challenges?

The first challenge is - How to access fragmented databases effectively.

If the data is not collected in one place, it is difficult to perform highly accurate data analysis using AI, so companies are forced to conduct data analysis manually or run their businesses on rules of thumb. Data analysis done manually can easily be mixed with subjective elements, and the accuracy is likely to be lower than objective analysis using a system.

How we can control different interfaces and access methods for each database?

Systems optimized for the business processes of each department are often not designed to link with other systems, and the types of databases and access methods are often different. This can make it difficult to utilize data collected by other departments.

This diagram shows an image of departments A and B managing different types of databases.

Of course, each database has a different I/F, and each system would require additional development to access the other databases.

Another challenge is - How to manage consistency among multiple databases

When considering using databases that are managed individually by each department across the organization, maintaining data consistency can be a challenge due to differences in databases and access methods.

This can make it difficult to reconcile databases and hinder collaboration between systems and departments.

Furthermore, it should not be forgotten that managing duplicate data in different departments means duplicate efforts and costs.

With ScalarDB, you can achieve the following for siloed databases:

• Consolidate different interfaces for each database

• Unify access methods to siloed databases

• Handle multiple different databases as if they were one database

• Ensure consistency among multiple different databases

ScalarDB provides a unified interface for different databases, allowing multiple types of databases to be treated as if they were a single database.

Being able to use the same interface and the same access method helps reduce development costs when the number of databases accessed increases.

ScalarDB also provides ACID functionality for non-acid compliance databases such as NoSQL.

As a result, it makes it possible to perform transactions between multiple types of databases, including NoSQL, and makes it easy to manage data consistency between departmental databases.

Consistency issues between multiple services

In a microservice-based architecture, each database is accessed via a corresponding service.

It is a challenge to ensure data consistency across the microservices.

The application needs to implement compensating transactions using Saga Pattern or TCC and this needs complicated logic to be implemented in the application.

Also, some NoSQL databases are ACID compliant to some extent, but still have restrictions.

Consider a case where there are three microservices, A, B, and C.

Microservice A manages a relational database, microservice B manages a limited ACID-compliant NoSQL database, and microservice C manages a NoSQL database that is not ACID-compliant.

In this case, it may be possible to achieve transactions between microservices A and B by making use of the SAGA pattern, etc.

However, transactions including microservice C cannot be realized.

Solution by ScalarDB

ScalarDB makes it easy to implement ACID transactions across multiple types of databases.

ScalarDB also provides ACID transaction functionality for non-acid-compliant databases.

So, you can use ScalarDB to run ACID transactions across multiple services, even if the underlying databases are different types, ACID-compliant or non-acid-compliant.

It also reduces development effort by eliminating the need to implement complex compensating transactions.

It is true that transactional systems which span multiple services can be developed using the Saga pattern or TCC.  

However, without ScalarDB microservices 

• Require compensation transaction logic

• Need to write different code for each type of database

• Become more complex and lose flexibility

• Need to consider various failure cases and those test patterns

ScalarDB supports transactions across multiple microservices, simplifying the development of microservices that require transactions.

• You can access each database using a unified interface, so there is no need to write different code for different databases.

• No compensating transaction is required, and the transaction execution code itself is simplified.

• There is no need to consider failure scenarios across multiple services and write test cases for them.

This makes application development across multiple microservices very efficient.

Now let us understand what is offered in scaler DB Enterprise Edition.

ScalarDB is available in open source and commercial licenses.

The commercial Enterprise Edition comes in two versions.

One is the standard version and the other is the premium version.

In the Enterprise Edition, ScalarDB offers a clustering solution. It has a set of cluster nodes, each of which provides ScalarDB functionality.

The ScalarDB cluster offers multiple interfaces such as gRPC, Java API, SQL and Spring Data, GraphQL, and .NET SDK.

The standard version offers only gRPC and JAVA API interfaces, whereas the premium version offers all interfaces including SQL/Spring Data, GraphQL, and .NET SDK.

Course Overview

In this training program, we will discuss more details about ScalarDB architecture and the data model used by ScalarDB. This can give you a better insight when implementing ScalarDB. We will then discuss what are the prerequisites while implementing ScalarDB. We will focus more on the practical approach. So, we will discuss different sample applications.

We will start with a simple console application.

Then we will study in application using microservice architecture 

So, let us get ready for this informative learning program and watch our next session on scalarDB architecture.