Introduction

Before we begin

Spined is in very early development phase, and many things, including the API, are going to change often and break things. Please consider it as experimental software that must not be used in production but if you do it, it's entirely at your own risks. Also, this document is a work-in-progress and incomplete, sometimes lagging with new features implementation. But if you get stuck, have questions, need help, or want to contribute reach out in the Discord chatroom.

What is Spined?

Spined is a data store (or database): a software to create and manipulate efficiently persistent data. Spined organize data as a semantic network, a flexible and high level paradigm.

Spined is extremely opinionated and focus on storing JSON type documents, using JSONSchema to enforce structure and JSONata as indexes and queries descriptor.

Spined is written in Node.js and bindings are currently only available for Node.js.

The low level storage implementation uses Redis and Spined generally preserve the high-performance of Redis and in some cases provide even faster performance by using an "in memory" application cache.

As an abstract front-end to Redis, it is part of the NoSQL database family but try bring back high level organisation of your data as in more traditional relational database. But far from a table oriented abstraction, Spined provide a graph-type model for easier and more human-understandable queries and mutation.

Spined never hide the lower level implementation and direct access to the Redis API access is always an option.

We have put a particular emphasis on implicit low level data separation security and enabling easy modern REST an GraphQL backend queries.

Thanks

Many thanks to all open source softwares projects linked above, without them Spined would not exist.

Hat tip for the Svelte and Sapper teams. They have a great framework and this documentation engine is forked from Sapper own documentation.

Install

Prerequisites

  • Node.js version 14 or later version
  • An access to a Redis store

Getting started

To use Spined, you need to install the npm spined package :

npm install spined
# or: yarn add spined

Quick start

In this section, you will learn how to use Spined in very simple cases:

  • basic operations to create and update persistent data
  • introduction to the Spined relationship paradigm (semantic network)
  • queries operations principles

Using Spined in Node.js

Before being able to use a Spined universe data handler, you first need to import the IORedis and Spined modules and create a Redis connection (check Redis documentation for the many options of a new Redis connection). For example, using the default Redis port on a local Redis installation :

import IORedis from 'ioredis'
import Spined from 'spined'

const redis = new IORedis()

IORedis comes as a peer dependency, so you may have to install it in your own project with npm install ioredis

You can now create a Spined universe data handler (by convention always named sp in this documentation) using the imported Spined module. You need to pass an object options argument containing at least the redis connection previously created:

const sp = Spined({ redis })

Many Spined operations are asynchronous and return a Promise. We will use await/async notation in this documentation but you may also use callback with .then

Adding persistent data in the store

To add our first persistent data in the store, we will use the method createSingularity. It takes a JSON compatible JavaScript Object as argument:

const john = await sp.createSingularity({ name: 'John Doe', age: '33', hobbies: ['music', 'cinema'] })

JSON is a subset of JavaScript Object. A JSON compatible Javascript Object should not contain data types which are not supported by JSON (undefined, BigInt, Symbol or function). To learn more about the differences, read What is the difference between JSON and Object Literal Notation?

A corresponding serialized JSON document is stored in Spined as an abstract representation called an Entity. An Entity contains the JSON data Object argument above, a unique id (usually following uuid format) and several meta information like the timestamps of the creation and the last update of the stored data.

A Singularity is an special Entity that has been created without any relationship with any other Entities.

Using the node.js Entity handler

The nodejs constant john in the previous § is a Spined Entity handler that can be used to perform operation on the stored data.

You can for example fetch the value of the property age:

console.log(john.age) // logs 33

You can also fetch the meta id property named $$id:

console.log(john.$$id) // logs ad1b8c32-17cf-4f1e-9265-78712295289b

And the meta timestamp corresponding to the creation of the john entity:

console.log(john.$$createdAt) // logs 2020-10-26T23:15:57.130Z

Fetching properties and meta information on an Entity handler is a synchronous operation because Spined maintain a cache data in memory when its handler in Node.js exists.

Meta properties are always prefixed with $$

Updating entity data

The method update$on the Entity handler will update or create all properties existing in its argument JSON object:

await john.update$({ age: 34, hobbies: ['music', 'painting'] })
console.log(john.name) // logs John Doe
console.log(john.age) // logs 34
console.log(john.hobbies) // logs [ 'music', 'painting' ]

Notice that to avoid collision with property names, all Entity methods like update$ are postfixed with the symbol $

Delete an entity from the store

The method delete$on the Entity handler will remove it permanently from the store:

await john.delete$()
console.log(john.name) //

Relationships

You can define for any Entity one or several named Dimensions.

Each Dimension is defining semantically designated set of relationships with other Entities.

Let's define a new Singularity Entity library with its two Dimensions Ẁriter and Musician:

const library = await sp.createSingularity({ address: '221B Baker Street, London, England' })
const WritersCollection = await libray.assignDimension$('WriterCollection')
const MusiciansCollection = await libray.assignDimension$('MusicianCollection')

To better distinguish Entities and Dimensions in your code, we encourage you to use capitalized names for Dimension name and their respective Node.js handlers constants.

Addind a new Entity into the Dimension MusicianCollection could be done using the method addEntity:

const lenon = await sp.createSingularity({ name: 'John Lenon', language: 'English' })
await MusicianCollection.addEntity(lenon)

But most of the time, you won't be creating Singularity Entities to add them afterwards into a dimension set. Instead, you can create and add a new entity directly into a Dimension in one step using the dimension method createEntity:

const shakespeare = await WritersCollection.createEntity({ name: 'William Shakespeare', language: 'English' })

There are no limits in the number of Dimensions an Entity can be member of. Let's create a new entity member of the Musician Dimension and add it to the WriterCollection dimension set:

const leonard = await MusicianCollection.createEntity({ name: 'Leonard Cohen, language: 'English' })
await WriterCollection.addEntity(leonard);

Entities created using createEntity can only be accessed through the Dimensions they are part of. This is the a base Spined principle to restrict and control access to a subset of data.

Simple queries

Spined being a JSON focused data store, we are using JSONata, a rich existing query language to search entities inside a dimension. But let's add a few more writers documents in our dimension set WriterCollection before writing our first queries:

await WriterCollection.createEntity({ name: 'Molière', language: 'French' });
await WriterCollection.createEntity({ name: 'Racine', language: 'French',  });
await WriterCollection.createEntity({ name: 'William Butler Yeats' language: 'English', nobel: true });
await WriterCollection.createEntity({ name: 'Henri Bergson', language: 'French', nobel: true });

To search all entites of the Dimension WriterCollection having English has value of the property language, you can use the dimension method filter as follow and iterate on the Resultset:

const writers = WriterCollection.filter('language = "English"');
for await (const a of writers.iterate()) {
  console.log(a.name);
}

All writers with a name containing the string ea :

const writers = await WriterCollection.filter('$contains(name, "ea")');
for await (const a of writers.iterate()) {
  console.log(a.name);
}

All writers with a nobel price : 

const writers = await WriterCollection.filter('nobel');
for await (const a of writers.iterate()) {
  console.log(a.name);
}

Queries are only limited by the the expressesiness of JSONata

TODO intro resultset and chaining...

Concepts

Graph Universe

Forget about flat and restricted bi-dimensional "table<=>record" models for your data, Spined allows multi-dimensional complex data architectures as a rich graph.

The Spined graph universe is easy to understand and allow both extremely simple model of data and conceptually rich sub-graphs of information securely isolated from each other.

The first thing to know is that the Spined graph universe is just a big collection of the same thing: Entities.

Entity

In Spined, an Entity can represent anything from a specific entry point to the graph universe to any abstract or concrete data instance you want to store (Entity could be products, customers, but also roles, permissions, etc). They are the nodes (or vertices) of the graph universe.

Simply put, an Entity is a set of data that you want to keep persistent record of as a block. There are very few restrictions on the size or the internal structuration of this data set. It just needs to be stored as JSON data object and be smaller that what the current Redis backend storage allows (512 Megabytes).

Each Entity being a JSON document decorated with a few meta information, it can have any shape allowed by the JSON standard format. For example, an "invoice" Entity would be able to store its lines in an array property and a person would be able to store many types of phone numbers and address without being contrainst by the very limited concept of "columns".

So JSON as the base data format liberates your data from the jail of the RDBMS "columns" concept but how do you manage relationships between Entities in the graph universe? There are handled in Spined using the concept of "Dimension".

Dimension

In Spined, a Dimension is not a relationship in itself but a category (you could also say a "class" or a semantic link definition) of relationships between an Entity and other Entities.

Let's say we have an Entity representing a company "ACME corp.", we could define a new Dimension "Employee" for this "ACME" Entity to store this specific semantic relationship "to be an employee" with other Entities representing persons.

TODO drawing

By default, new Entities in the graph Universe have no Dimensions, but you can add as many as you want. In our example, we could add the Dimensions "Products", "Suppliers", etc, to the "ACME" Entity.

The Spined graph universe is oriented, thus a relationship from the Entity "A" to the Entity "B" defined by the Dimension "X" doesn't imply any symetric Relationship Between Entity "B" towards Entity "A".

Each Dimension has a name and every Dimensions in the graph universe sharing the same name may be statically bound to common behaviors and schema conditions for their constituent Entities. For example, "being an employee" is not a kind of relationship only valid for the company "ACME corp." but for any company. Thus you could add the same Dimension named "Employee" for many other Entities representing companies and add a condition that any person Entity added to any of these "Employee" Dimensions being older than 16 years old before allowing a person in the Dimension "Employee".

In the graph theory, Spined Dimensions are just labels for oriented links between nodes. The decorated graph is forming a semantic network.

normalisation

When building a spined model, you often can choose two different directions to represent a set of data that can be linked to an Entity:

  • As a subdata set included in the JSON data document of the Entity itself
  • As a set of separate Entities whose relationship with the main Entity is recorded as a specific "dimension"

One example would be a person Entity owning some books. You could store the books information as an array inside the JSON document of the person Entity or create a Dimension "OwnedBook" for that person and put into it as many book Entities as necessary.

The choice will often depends on the type of application you are building but some general rules can be drawn:

  • for very small subset of data, it is often more practical and efficient to store them inside the JSON document of the main entity even at the cost of small repetitions in the backend storage.

  • for structured subset of data with more than one Entity relationship, it preferable to store them in separate entities, specially if their relationships cardinality vary.

  • finally, for subset of data that are updated or deleted independently, it is nealy always a better idea to store them as separate Entities.

So to come back with the Person/OwnedBooks example above, you would prefer store books as separate Entities if one book can be owned by more than one person. If a book information dataset needs to be updated (selling statistics or price history), you absolutely needs to store them as separate Entities to keep your life simple.

schema & model

You can modelize a Dimension in greater detail than just a name, and specify that all Entities in a specific Dimension must obey a particular JSONschema description.

You can also define model methods and trigger to get callable for all entities in a specific Dimension (for example a method sendInvoice callable on any Entity in the Dimension "invoice" of a client).

Schema, model methods and triggers are statically bound to a particular Dimension name, meaning that any Dimension with that name will inherit these (for example the "sendInvoice" method bound to all Dimensions "Invoices", for any suppliers or clients).

Resultset

A Resultset represents a set of Entities selected via differents type of query methods. Resultsets can be chained, and you can also create a new Resultset via union, intersection and exception set operations.

union, intersection and exception are not yet implemented

Index

You may define Indexes on Dimensions to allow faster search and ordering of Entities. Spined currently support 3 types of Index :

  • Set Index are grouping Entities inside a Dimension depending on the value of a simple property of each Entity JSON document or even a complex result value from a JSONata query. This type of Index allow faster search inside a big Dimension filtering by the indexed value.

  • Unique Index enforce a unicity condition on Entities belonging to a Dimension on a simple property or JSONata query result. It allows O(1) retrieval of the Entity mapped to that value. This Index is ideal to retrieve instantly an Entity with a unique identifier, as a classic example an "email" for an Dimension "Account".

  • Cursor Index are organizing all Entities of a Dimension inside a sorted set (again either on the value a simple property or a full JSONata query result). This is ideal to implement cursor-based or offset-based pagination on big dataset.

Reference Frame

All operations on Entities and Dimensions are operating in the context of a Reference Frame. A reference frame is simply a non empty list of other special Entities of the graph universe (named Reference Sources) that give a "point of view" on these all operations. The semantic to associate with a Reference Frame is not pre-determined and is enterily of your choosing.

One simple illustration of this concept would be to assign different behaviors on the method "engineOn" of an "Entity" representing a car. The method could test if the reference frame include an Entity refrence source "Account" and only allows the "engineOn" method if it matches the owner property of the car in the JSON data object.

A Reference Frame is always implicitly inherited automatically in all operations results (Entities, Dimensions, Indexes and Resultset). You usually start your software workflow with a very generic Reference Frame from a Singularity (see below) and use smaller references frames to manage different contexts (typical examples would be role permissions or specialized behaviors).

Singularity

A Singularity is a special Entity that has been created "ex-nihilo" with no prior relationship with any other Entities. They are starting points in the graph universe and the only Entities searchable by their meta id without knowing anything else about them.

Looking up for a Singularity Entity automatically defines a default Reference Frame including only the singularity itself as reference sources.

A simple query is usually a search in a Dimension of a single Entity. In this case we the Reference Frame is the Entity on which you apply the search.

But a Reference Frame can contain more than one Entities to allow advanced search...

[ TODO complete with example ]