Spined graph universe

Constructor Spined(params)

The constructor of a Spined universe data handler (by convention always named sp in this documentation) is the imported Spined module that only need one mandatory parameter: a Redis data source.

import IORedis from 'ioredis'
import Spined from 'spined'

const redis = new IORedis()

const sp = Spined({ redis })

params = { redis, extend?, extendFrom?, options? }

  • redis — redis client instance used to store data
  • extend? — dimensions model extensions
  • extendFrom? — read dimensions extensions from a directory
  • options? — TODO

Spined handler properties

sp.redis

  • Return the redis connection

sp.extend

  • Return the current model extensions (merge of constructor parameter and loaded from file)

sp.options

  • Return the constructor parameters options

Spined handler methods

sp.createSingularity(JSON?)

The method stores a JSON data object as a new singularity Entity.

This is the only way for an entity to get the SINGULARITY trait

const paris = await sp.createSingularity({
	name: 'Paris,
	country: 'FR',
	attractions: {
		'Eiffel Tower' => '48.8584° N, 2.2945° E',
		'Louvre' => '48.8606° N, 2.3376° E',
	},
	airports: [ 'CDG', 'ORL' ]
});
  • JSON — the JSON data object to be stored
  • Return — Promise of the new created Entity

sp.referenceFrame(<$$id>)

The method retrieve the implicit Reference Frame associated with the Singularity Entity having the meta id <$$id>. This type of Reference Frame have only one source, the Singularity Entity itself.

The source Entity will be accessible with alias default and 0

const ref = await sp.referenceFrame('715980bd-152d-43b8-b958-9e1fae3862a4');
  • $$id — an entity meta id (this entity need to have the SINGULARITY trait)
  • Return — Promise of a Reference Frame

sp.defineReferenceFrame([...entities] | { alias: entity1, ... })

The method can create a complex Reference Frame from one or more source Entities. At least one of the source Entities own Reference Frame sources need to have the OMNISCIENT trait.

The method can simply take as arguments a list of entities. Source aliases in this case will be the index in the list (and also default for the first source entity).

Another way, is to call the method with a one dimension object with aliases as keys and entities as values. This allow a Reference Frame with custom alias accessor.

const ref = await sp.defineReferenceFrame([ e1, e2]);
// or
const ref = await sp.defineReferenceFrame({
	monument: OneTowerEntity,
	role: OneArchitectEntity,
});

Spined handler helpers

sp.isReferenceFrame(arg)

  • Return true if the argument is a Spined ReferenceFrame

sp.isEntity(arg)

  • Return true if the argument is a Spined Entity

sp.isDimension(arg)

  • Return true if the argument is a Spined Dimension

sp.isIndex(arg)

  • Return true if the argument is a Spined Index

sp.isResultset(arg)

  • Return true if the argument is a Spined Resultset

sp.get$$id(arg)

  • Return the $$id if the argument is a Spined Entity or the arg itself

Reference Frame

Be sure to check the introduction on the Reference Frame concept

Reference Frame definition

There are only two ways to get a Reference Frame in the graph universe:

  1. by calling the sp.referenceFrame() method on the graph universe handler to retrieve the implicit reference frame of a Singularity Entity
  2. by calling the sp.defineReferenceFrame() method on the graph universe handler with one or more Entities

Reference Frame properties

ref[alias]

You can directly get a source entity of a Reference Frame using its corresponding alias. If no aliases where defined, you can access it using the index of the source entities in the list parameters of drawReferenceFrame().

ref.default

The first source entity is always accessible via the default property

ref.isOmniscient

  • Return true if the Reference Frame is said to be Omniscient. Its the case only if at least one source entities has the OMNISCIENT trait.

You can only define a new Reference Frame using the sp.defineReferenceFrame() method from entities inside an Omniscient Reference Frame.

ref.isUnbounded

  • Return true if the Reference Frame is said to be Unbounded. Its the case only if at least one source entities has the UNBOUNDED trait.

You can only change Entities traits (using ent.addTrait$() or ent.addTrait$() inside an Unbounded Reference Frame.

ref.spined

ref.aliases

  • Return the aliases name to access the source entities of the Reference Frame

ref.sources

  • Return an Object of all sources entities with aliases as keys.

Entity

Be sure to check the introduction on the Entity concept

In this documentation ent is always an Spined specific Entity

Entity creation

There are only three ways to create a new entity:

  1. by calling the sp.createSingularity() method on the graph universe handler
  2. by calling the ent.duplicate$() method describe below
  3. by adding a new entity into an existing dimension with the method dim.createEntity

Entity properties

ent.<any>

The JSON data object keys stored as an Entity automatically defines corresponding getter properties.

const ent = await john.update$({ age: 34, hobbies: ['music', 'painting'] });
console.log(john.name); // logs John Doe
console.log(john.age); // logs 34
console.log(john.hobbies); // logs [ 'music', 'painting' ]

Since all meta properties, directly hanlded by Spined are prefixed with $$, you should avoid using first level properties in the JSON data object to be stored starting with $$. Similarly, all Entity methods are ended with the '$' character, so first level properties should also avoid this pattern. Note that such data would still be saved but with not direct getter using the ent.<any> syntax. You may use the syntax ent.$$json.$$<any> to access them in these cases.

ent.$$id

A unique id is assigned for each newly created Entity. You can access this meta attribute using the handler $$id.

For this alpha relase of spined, $$id̀ follow the uuid convention.

[ TODO byt not yet implemented : seq / cuid / short uuid / etc ]

ent.$$createdAt

Any new Entity is created with the current timestamp stored in the $$createdAt meta attribute

ent.$$updatedAt

Anytime an Entity is updated, the current timestamp is retain in the $$updatedAt meta attribute

ent.$$json

Entity handlers in node.js are not standard JavaScript Object. To get a pure JSON object of the data stored of any Entity you can call the meta attribute $$json

const paris = await sp.createSingularity({
	name: 'Paris,
	country: 'FR',
	airports: [ 'CDG', 'ORL' ]
});

console.log(paris.$$json)
//
//

ent.$$json$$

Shortcut to get $$json object but with with $$id̀, $$createdAt, $$updatedAt meta attributes value included.

ent.$$traits

Return the list of all Traits of the Entity ent

ent.$$dimensions

Return a Promise of the list of all Dimensions names of the Entity ent

Entity traits

Entity can be decorated by Traits that assert non default behaviors.

SINGULARITY

Only Entities created using the sp.createSingularity() method have this trait.

Only Singularity Entities can be retrieved only by knowing their $$id to create a new Reference Frame.

A more numdate explanation is that Singularity Entities are the only direct entry points to the graph universe.

OMNISCIENT

Entities created using the sp.createSingularity() method have this trait by default. This trait can also be added or removed using the ent.addTrait$() and ent.delTrait$() methods.

At least one Omniscient Entity need to be in the current Reference Frame for a new Reference Frame being defined using the sp.drawReferenceFrame()

UNBOUNDED

This trait can only be added or removed using the ent.addTrait$() and ent.delTrait$() methods.

At least one Omniscient Entity or Unbounded Entity need to be in the current Reference Frame for ent.addTrait$() and ent.delTrait$() methods to be available.

Unbounded Reference Frame can also overwrite some meta properties ($$id, $$created, ...) if the graph universe was created with the corresponding options

Warning: NOT FULLY IMPLEMENTED YET

PERPETUAL

This trait can only be added or removed using the ent.addTrait$() and ent.delTrait$() methods.

You cannot use the ent.delete$ method on Pertetual Entities.

IMMUTABLE

This trait can only be added or removed using the ent.addTrait$() and ent.delTrait$() methods.

You cannot use the ent.update$ method on Immutable Entities.

INVISIBLE

This trait can only be added or removed using the ent.addTrait$() and ent.delTrait$() methods.

You cannot use the ent.$$jon ent.<any> properties on Invisible Entities. Custom extension properties and methods are still accessible.

Entity CRUD methods

ent.duplicate$()

ent.get$()

ent.update$()

ent.inc$()

ent.delete$()

Entity other methods

ent.be$(ent2)

Return true if ent2 is strictly the same Entity as ent (return false for two different Entities with different $$id even if the stored JSON data object is identical).

NOT yet IMPLEMENTED

ent.assignDimension$(<name>)

Assign a new dimension the the Entity ent (Return a Promise).

ent.revokeDimension$(<name>)

Revoke a new dimension the the Entity ent (Return a Promise).

ent.hasDimension$(<name>)

A Promise of a boolean, true if the dimension <name> of the Entity ent exists.

ent.dimension$(<name>)

A Promise of the dimension <name> of the Entity ent.

ent.addTraits$(<trait>)

ent.delTraits$(<trait>)

Dimension

Be sure to check the introduction on the Dimension concept

In this documentation dim is always an Spined specific Dimension

Dimension creation

There is only one way to add a new dimension to a specific Entity :

  1. by calling the ent.assignDimension$(<name>) method on an Entity

Dimension properties

dim.name

Return the name of the dimension dim.

Dimension names are unique for each Entity

dim.schema

TODO

dim.indexes

TODO

dim.extend

TODO

dim.$$id

Each Dimension has also a unique meta id. You can access it using the handler id.

Dimension mutation methods

dim.createEntity(<JSON>)

The method stores a JSON data object as a new Entity and directly add this new Entity to the Dimension dim.

dim.addEntity(ent)

Add the existing Entity ent to the Dimension dim.

dim.removeEntity(ent)

Remove the existing Entity ent to the Dimension dim.

WARNING, GARBAGE collection is not yet implemented...

dim.addIndex()

dim.removeIndex()

dim.setSchema()

Dimension query methods

dim.has(ent|<$$id>)

Return true if the Entity ent (or the meta id <$$id>) is part of the dimension dim.

dim.retrieve(<$$id>)

Return the Entity with the meta id <$$id> which is part of the dimension dim.

dim.all()

Return a new Resultset with all Entities of the dimension dim.

Dimension alias methods

Some useful shortcuts alias methods are provide as syntaxic sugar:

  • dim.filter => dim.all().filter

  • dim.order => dim.all().order

  • dim.where => dim.all().where

  • dim.iterate => dim.all().iterate

  • dim.first => dim.all().first

  • dim.ids => dim.all().ids

These last two shorcuts provide the same results as their long version but are slightly more optimized when called directly from the dim handler :

  • dim.count => dim.all().count
  • dim.has => dim.all().has

Index

Be sure to check the introduction on the Index concept

Index creation

There are only two ways to have an Index defined on a Dimension :

  1. by calling the dim.addIndex() method on a Dimension
  2. by adding an Index definition when assigning a new dimension with ent.

Index definition

In this documentation idx is always an Spined specific Index

Index types

You can define different kind of Index on Dimension and they are classified by type. Index types are currently either 'U', 'S' or 'C' which are respectively "SET", "UNIQUE" and "CURSOR" Indexes.

Index indentifier

Index identifier are formed by the concatenation of their type (U, S, or C) the separator :, an optional Index argument followed by : and a JSONata query.

Index JSONata query

Indexes are created on a specific JSONata query that map every Entity JSON document to a value from the JSONata query evaluation. In most cases, that would be just the value of a specific property inside the JSON document but any complex of JSONata query is possible.

You can also use the meta properties $$createdAt and $$updateAt in the JSONata query.

complex JSONata query may slightly impact performance when you add a very large number of new Entities in batch to an indexed Dimension.

Index type S (SET)

Type S Indexes, are just a subset of the Dimension that verify a specific JSONata query (i.e. evaluate to truthy value).

This type of index is well suited to query really fast a Dimension subset that verify a particular criteria. For example, you could set a Index on the "city" property and be able to search or count Entities efficiently on any specific city subset.

idx.all(<value>)

Return a new Resultset of Entities that verify <value> for the Index idx.

S alias methods

  • idx.ids(<str>) => idx.all(<str>).ids
  • idx.count(<str>) => idx.all(<str>).count
  • idx.first(<str>) => idx.all(<str>).first
  • idx.iterate(<str>) => idx.all(<str>).iterate

Index type U (UNIQUE)

You can declare that all Entities in a Dimension verify a unicity condition for a specific JSONata query result (typically used to guarantee that a property value is unique for each Entities of the Dimension like email in a "Account" Dimension, but any query can be used).

idx.find(<value>)

Return a Promise of the Entity for which is indexed unique value is <value>. Returns null if no entity is found.

idx.findElseThrow(<value>)

Same as find but throw if no Entity is found.

Index type C (CURSOR)

Cursor Indexes are used create an ordered set of all Entities in a Dimension. The JSONata query is used to calculate the key to sort all their Entities. The sort order can be alphanumeric or numeric depending the identifier argument, respectively AlphaNum or Num.

If you don't mention the sort order mapping, it default to AlphaNum except on the meta attributes $$createdAt and $$updatedAt for which it is Num

Cursor Indexes are primaryly used as argument to the rs.order() method.

idx.rank(id)

Return a promise of the rank the Entity with id id in the ordered by set.

Resultset

Be sure to check the introduction on the Resultset concept

In this documentation rs is always an Spined specific Resultset

Resultset query methods

Query methods are always returning a Resultset, so you can chained them

rs.where(<S:idx>, <value>)

Return a new Resultset of Entities that are included in the Set Index <S:idx> for the value <value>.

rs.order(<C:idx>)

Return a new Resultset of Entities using the order defined with the Cursor Index <C:idx>.

rs.filter(<JSONata>)

Return a new Resultset of Entities that verify the <JSONata> query. See https://jsonata.org/ for all the possibilities.

For example, to filter only Entities with a property primaryName containing the substring Jean, you shoudl write :

rs.filter('$contains(primaryName, "Jean")')

The filter methods has an O(N) complexity (all Entities of the called Resultset need to be tested). You should avoid to use it for very large Resultset. Use Set Indexese instead for faster queries.

TODO note chaining

rs.sort(<fct>)

TODO

Resultset solving methods

Resultset solving methods always return a promise or an AsyncGenerator.

rs.has(<$$id>)

Return a promise of a boolean. True if Entity with id <$$id> is part of the Resultset rs.

const has = await rs.has('5236e2ab-71f2-4bce-9d39-2ae4d3ca0ab8')
console.log(has) // true

rs.ids()

Return a promise of a list of all Entities meta ids part of the Resultset rs.

const ids = await rs.ids()
console.log(ids) // ['5236e2ab-71f2-4bce-9d39-2ae4d3ca0ab8', '47bd05a9-10e5-4b5d-aa8f-3a1f955dfd07' ]

rs.count()

Return a promise of a number of Entities part of the Resultset rs.

const count = await rs.count()
console.log(count) // 2

rs.first()

Return a promise of the first Entity part of the Resultset rs.

const ent = await rs.first()

rs.iterate()

Return a async generator of all Entities part of the Resultset rs.

Return AsyncGenerator of all results

for await (const ent of all.iterate()) {
	console.log(ent.$$json)
}

rs.connection({ <args> })

Return a promise of a connection closely following the specifications (https://relay.dev/graphql/connections.htm).

<args> is an Object with either first or last as mandatory properties with the integer value <n>. The connection will return an Object with properties edges and fieldsproperties, respectively the first or last <n> "edge" in the Resultset.

An "edge" is just an Object ecapsulating an Entity and its "cursor".

If first is used, you can pass a cursor in the parameters after to get the edges after the Entities cursor. Respectively you can pass a cursor in before when you use last.

desc is a boolean (by default false) to indicates that the ordered resulset is understood as descendant. Finally you can use afterRank that take the absolute rank instead of a cursor.

// in Graphql resolvers 
books: async (parent, args, ctx) => {
  const rs = parent.dimension$('book')).all()
  return connection(rs.order('C:$$createdAt'), args)
}

Resultset alias methods

Some useful shortcuts alias methods are provide as syntaxic sugar:

  • rs.ids(<JSONata>) => rs.filter(<JSONata>).ids()
  • rs.count(<JSONata>) => rs.filter(<JSONata>).count()
  • rs.first(<JSONata>) => rs.filter(<JSONata>).first()
  • rs.iterate(<C:idx>) => rs.order(<C:idx>).iterate()
  • rs.first(<C:idx>) => rs.order(<C:idx>).first()

Dimension extensions

A dimension name can be statically extended with schema constraints, triggers and behavioral methods.

Any concrete dimension of an Entity having that identical name would be bound with these extensions.

TODO extensions are not yet fully documented

Loading extensions

The current most convenient way to add extensions is to use the extendFrom option of the Spined() Constructor.

The option take a directory as argument. Every <dimension name>.js files in this directory with a default Object export will inject all extensions under the properties schema, triggers, getters, methods, options for all Dimensions whith that name.

Very simple extension file :

const extensions = {
  schema: {
	title: 'Account',
	type: 'object',
	required: ['email'],
	properties: {
	  email: { type: 'string' }
	}
  },
  triggers: {
	beforeDelete: ($ref, attributes) => !attributes.isActive
  },
  getters: {
	fullName: ($ref, self) => () => self.firstName + ' ' + self.lastName,
  }
  methods: {
	toString: ($ref, self) => () => {
	  return `${self.slug}`
   }
  }
}
export default extensions

schema extension

You can bind a JSONschema to a dimension name. Entities JSON data will need to verify the JSONschema to be able to added to the dimension.

triggers extensions

beforeCreate: ($ref, attributes) => attributes

beforeUpdate: ($ref, attributes) => attributes

beforeDelete: ($ref, attributes) => boolean

getters extensions

getterName: ($ref, self) => () => Any

methods extensions

methodName: ($ref, self) => (Any args) => Any

options extensions

inMemoryTtl

Debugging

TODO

Format

Redis Storage format

Low level storage of data is handled by Redis. Data is stored with a predicatable and open sourced simple format that doesn't depends on the current actual implementation of Spined (though they are both optimized to work well together).

  • Every Entity data and meta data is a serialized JSON object value stored at the key <entity id>
  • Every Dimension data and meta data is a serialized JSON object value stored at the key <dimension id>
  • A Redis hash with key <entity id>.D is storing all <dimension id> and their names of an Entity.
  • A Redis set <entity id>.M is recording all <dimension id> in which an Entity is included
  • A Redis set <dimension id>.E is recording all <entity id> belonging in a Dimension
  • A Redis set <dimension id>.I is recording all indexes of a Dimension

Other specialized set with key <dimension id>.<long hash> are used to optimize resultset requests but they are not necessary to restore the data stored with Spined.