Towards a New Data Modelling Architecture

By Athanassios I. Hatzis, PhD, R&D Software Engineer
(C) 17th of March 2015

In Part 1 of this series we talked about the main fundamental constructs of the Entity-Relationship data model, i.e. the record and the table. We continue our discussion here introducing the Atomic Information Resource (AIR) unit of AtomicDB database management system and we describe the principles of R3DM, a conceptual framework based on semiosis that can offer a solid theoretical background and basis for this approach in Part 3.

Part2: The Atomic Information Resource Unit (AIR)

Background Information

Origins of the AIR Data Model

The data model we describe has a long history behind it. Searching at Google for patents with the title “Data Base and Knowledge Operating System” or with the title “Data management architecture associating generic data items using reference” we find several documents that are filed in the year 2003 from the inventor Ron Everett. The same year a “proof of concept” for this invention was successfully demonstrated with a use case of managing a 50+ milion records of spare/repair part requirements for US Navy ships. Today AtomicDB is a fully fledged database management system that is based on Ron Everett’s patented associative data items architecture. Several pilots are currently running under big corporations and University research establishments.

AtomicDB API Evaluation

The author of this article is an evaluator of AtomicDB and has been given full access to test GUIs, APIs and web services of their system. In particular he is exploring their C# API functionality for developers. In order to make tests interactive and efficient most functions of their API have been ported to Wolfram Language using the NETLink C # interconnectivity package. In this part function calls and other expressions that you see are based on this AtomicDB Mathematica API that is not included with the publication of this article.

Introduction to the Problem

The Table ‘Silo’ Structure

Needs["DatabaseLink`"]
conn=OpenSQLConnection[JDBC["Microsoft Access(ODBC)","C:\\Temp\\SuppliesCatalogDB.mdb"]];

Extracts from the Inventor of Technology Ron Everett

The following extracts are from a recent article titled “Introduction to Associative Information Systems”

“When we attempt to use Tables as a storage paradigm for Information we discover that Tables are a namespace bound, non-dynamic, 2-D, structured storage paradigm that has a different structure for every Table in every Database. Each application is developed with unique and special queries written to each specific database design, table layout and named tables, columns and keys.”

This is the main reason that noSQL databases appeared on the scene recently, key-value store, hierarchical semi-structured documents, column based and graph-property data structures are all attempts to provide a solution to this problem.

Table Example

This is the Parts table from a Microsoft Access relational database that we have used in Part 1

(partsList=SQLExecute[conn,"select pid,pname from Parts where pid<994","ShowColumnHeadings"->True])//TableForm

"Some have hailed XML (RDF and triple stores) as the means to solve the n-dimensional relationship problem, because with it, meta-information can be captured, but XML is plagued with other problems, not the least of which are namespace binding requiring semantic accord, massively replicated tags and data, the heavy overhead of text based processing, the necessity of searching and indexing all the text in every possible XML document for each and every key/ value-tag/data match sought and the distribution of the tagged datasets across innumerable XML documents, stored in 2-D table-referenced 2-D file structures. Add to that list the overhead imposed by using Semantic Web languages and ontologies and the PhD level specialists required to develop and maintain these 'knowledge' oriented systems and you get even more namespace entrenchment and hence specialization of the applications developed with it all."

Table Example in XML format

This is the same table with records as above but now it is serialized in XML

partsXML="<?xml version="1.0" encoding="UTF-8" ?>
	<dataroot>
		<Parts>
			<pid>991</pid>
			<pname>Left Handed Bacon Stretcher Cover</pname>
			<pcolor>Red</pcolor>
		</Parts>
		<Parts>
			<pid>992</pid>
			<pname>Smoke Shifter End</pname>
			<pcolor>Black</pcolor>
		</Parts>
		<Parts>
			<pid>993</pid>
			<pname>Acme Widget Washer</pname>
			<pcolor>Red</pcolor>
		</Parts>
	</dataroot>";

Table Example in JSON format

Same records of the table serialized in JSON format

partsJSON="{
  \"dataroot\": {
    \"Parts\": [
      {
        \"pid\": \"991\",
        \"pname\": \"Left Handed Bacon Stretcher Cover\",
        \"pcolor\": \"Red\"
      },
      {
        \"pid\": \"992\",
        \"pname\": \"Smoke Shifter End\",
        \"pcolor\": \"Black\"
      },
      {
        \"pid\": \"993\",
        \"pname\": \"Acme Widget Washer\",
        \"pcolor\": \"Red\"
      }
    ]
  }
}";

The Alternative Solution - From Data Items in Table to Information Atoms with NO Table

“Every table is a silo. Every cell is an atom of data with no awareness of its contexts, or how it fits in to anything beyond its cell.It can be located by external intelligence but on its own it' s a "dumb" participant in the system - the ultimate disconnected micro - fragment accessible only by knowing the column and the record it exists in. The alternative is to replace the data elements with information at the atomic level of the system. Instead of a data atom in a table, we have an information atom with no table. Information atoms exist in a multi-D vector space unbounded by data structures and know their context, such as a “customer” or a “product”, just like atoms in the physical world “know” they are nitrogen or hydrogen items and behave accordingly. Information atoms also know when they were created, when they were last modified, and what other information atoms of other types are associated with them. They know their parents, their siblings, and their workplace associates. They are powerful little entities and most certainly NOT fragments. Nor are they triple statements requiring endless extraneous indexing.”

Therefore at its core AtomicDB is datatype and namespace agnostic, always fully contextualized, and structure free.

All these sound perfect in theory but in practice everyone is accustomed to the use of tables. The table is already the favourite manageable structure and it is most convenient as a medium of exchanging datasets. Therefore the challenge is that any alternative solution on data architecture should provide the means to view data in tables with the minimum effort no matter what is the underlying structure.

The Conceptual Model Perspective

The Entity-Relationship model is universally accepted as the means to extend the relational model in order to give meaning to the relationships among the relations. In our database example we have the following ER diagram.

Import["C:\\DataIO\\accdb\\Examples\\SuppliesCatalogDB-ER.jpg","jpg"]

The diagram above depicts the kind of relationships among the three entities of our database, Parts, Suppliers and Catalog, and shows the datatypes of their attributes. This is a classic many to many relationship between Parts and Suppliers where the Catalog is the associative entity, also known as the junction table, bridge table, join table, etc. Primary and foreign keys are also specified for the names of the attributes that play that role.

The Entity-Attributes ‘Silo’ Structure

The problem here is that from a semantic point of view, similar diagrams are in need from users that want to express business processes but when we reach the implementation stage software engineers have to marry business requirements with the technical constrains of the database system hence the ER diagram you see. Generally speaking this is known as "The Model", a conceptual view of the user on data. The ER version of the model has several limitations, due to the architecture of RDBMS. The main one is that each attribute remains enclosed in the table structure and in the case the same attribute appears in another table, the dataset that it represents has to be repeated. In our example above, the primary key (pid) of "Parts"  is repeated as a foreign key (catpid) in Catalog. The difficulties that arise in data aggregation due to this limitation are substantial.

How to Break Free from the Entity-Attribute-Value Paradigm

The relational and the entity-relationship model made a huge impact in the IT world for nearly half a century. But during this long period of standardization it meant also one thing, everyone had to comply with the rules and requirements of the model. Everyone had to think in terms of Entity-Attribute-Value or Subject-Predicate-Object as it is known in the RDF semantic model. Programming languages have been affected to from this monolithic way of thinking. Although it proved to be advantageous to program with classes and objects, it created an artificial problem of how to map these onto persistent data structures on the disk, also known as the object-relational impedance mismatch problem. Knowledge representation frameworks did not escape from this too. Ontologies expressed in OWL followed the same paradigm with classes, attributes, and values. Serialization methods such as JSON (object-name-value) and XML (element-attribute-value) also came after the same rationale.

The Signified - Sign - Signifier Alternative Paradigm

The aforementioned Entity-Attribute bond and distinction plays its role here too. But most important another concept, ‘value’, is added to make this triplet even more difficult to handle in our digital world. This is mainly due to the fact that three perspectives, the conceptual, the representative and the physical layer encoding are mixed in such a way that it is very hard to separate and work with them at distinct levels of abstraction. The R3DM, or S3DM, conceptual framework that we discuss in Part 3 is based on the natural process of semiosis where the signified, i.e. concept, entity, attribute and the signifier, i.e. value are referenced through symbols, i.e. signs, at discrete layers. The same philosophy is shared in the architecture of this database management system and we demonstrate this with the following example.

AtomicDB Model and Concepts

There are many ways one can start working with AtomicDB, and even this flexibility demonstrates how vigorous is the whole data modeling process. But in order to continue our discussion on the signified, sign, and signifier principle we will design first a simple concept map using the cmap tools from IHMC that corresponds to the ER diagram  above.

Design a Concept Model with CMAP Tools

Import["C:\\DataIO\\accdb\\Examples\\PartsSuppliersCatalogCMAP.jpg","jpg"]

This demonstrates fully the point we have already made about Entities and Attributes. Entities in this diagram are simply formed by grouping Attributes together. One or more Attributes are shared between two or more Entities. According to the AtomicDB terminology, shared attributes are called bridge concepts and this is the equivalent of the relationship that is implemented with primary and foreign keys on two tables but here we are completely independent to mix and match them. For example PartColor, could be merged with another attribute from a different table in another relational database.

Model and Concepts

Import the Concept Map

AtomicDB can read a CMAP cxl file and get the structure

Add The Model and Concepts Programmatically

Alternatively we can add the model and the concepts by using commands from our API. The functions we use have been defined on top of those we ported from the C# API of AtomicDB. Now, we assume that we have already established a connection with the AtomicDB database server. Let us first create our model and give it a name.

modelName="Parts-Suppliers Catalog Model";

addModel[modelName]

We can get back the model we have just added to the system and at the same time print also a user-friendly output

(model=getModelByName[modelName])//printKVP

The key is a reference 4D vector that we use to access the model item (13=models, 289=our model) and the value is the string we assigned as the name of the model. Everything that is stored in AtomicDB has a key and a value. This makes AtomicDB fully symmetrical in terms of values, structures and relationships. It a unique property that allows to build everything on top of it in a symmetric way, e.g. the commands of the API that we will discuss in details in another part of this series. Now let us add the concept names in groups. The first name in the list signifies a Nexus concept, i.e. a concept that is used to associate the rest of the concepts in the list.

catalogConceptNames={"NX_Catalog","SupplierIdentifier","PartIdentifier","PartCost"};
partConceptNames={"NX_Part","PartIdentifier","PartName","PartColor"};
supplierConceptNames={"NX_Supplier","SupplierIdentifier","SupplierName","SupplierAddress"};

(catalogConcepts=addConceptsByName[modelName,catalogConceptNames])//printKVPL2

As we expected, four concepts were added to our model. Notice the last two dimensions of the keys, i.e. (context, item). It is using the same dimension as the instance of the model in the previous reference to denote that these concepts belong to that model. Actually all the reference vectors in AtomicDB are cleverly inter-related and there is never a chance for a collision when adding items. Another observation we can make is that these items have been created in a different Environment, Repository, the first two dimensions, than the model item. More about the reference vector system of AtomicDB in another part. We will continue to add the rest of our concepts to the model.

(partConcepts =addConceptsByName[modelName,partConceptNames])//printKVPL2

(supplierConcepts =addConceptsByName[modelName,supplierConceptNames])//printKVPL2

In these concept groups you may notice that two of them PartIdentifier and SupplierIdentifier are the same as in the first group. These concepts are called bridge concepts and play the same role as the primary and foreign keys in relational data sets. The main difference and a great advantage of this approach is that this time data sets are not dublicated. The same collection of items, i.e. data set of the attribute can be referenced by many concepts.

Get The Concepts from a Model

Time to verify that all the concepts have been added to our model

getConceptsFromModelName[modelName]//printKVPL2

AtomicDB Collections and Records

Terminology

Data Item is a particular type of item that holds an atomic piece of data (an atomic value).

Collection (data set) is a generic container for data items with no duplicates. A collection is similar to the notion of attribute (column) data set in the relational model.

Nexus item is a special type of data item whose role is to keep associations with the other data items in a record. Nexus item plays a similar role to that of a record in the relational model.

Nexus Collection is a special type of collection which holds nexus items only. Nexus collection act similarly to the primary key column in the relational data model.

Record is a set of data items from different collections each associated to the same nexus item (exactly one per record)

Group refers to several collections and associates them. The group is not a container for collections. Each group has one and only one nexus collection.

Bridge Collection is a certain type of collection that can be associated with more than one group. Bridge collection act similarly to the foreign key column in the relational data model

Concept is a special type of item that represents uniquely one collection of items. A collection can have one or more representative concepts. A concept can be though as a reference to collection.

Model is a generic container for unique concepts that are associated to form higher constructs and relations. Model is similar to a database schema, or view.

Collections and Records

Add Collections with Automap and AutoGroup options

With the following addCollectionsAutoMapGroupByName command Collections are automatically associated with the concepts and a group is added.

Add Catalog Collections

(catalogCollections=addCollectionsAutoMapGroupByName[modelName,catalogConceptNames])//printKVPL2

Add Part Collections

(partCollections=addCollectionsAutoMapGroupByName[modelName,partConceptNames])//printKVPL2

Add Supplier Collections

(supplierCollections=addCollectionsAutoMapGroupByName[modelName,supplierConceptNames])//printKVPL2

Get Collections From the Model

Verify that we have added all the collections to the system

getCollectionsFromModelName[modelName]//printKVPL2

Add Records

Read Relational Datasets

catalogDataSet=SQLSelect[conn,"Catalog","ShowColumnHeadings"->False];
partDataSet = SQLSelect[conn,"Parts","ShowColumnHeadings"->False];
supplierDataSet=SQLSelect[conn,"Suppliers","ShowColumnHeadings"->False];

Add Catalog Records

addRecordsByName[modelName,catalogConceptNames,catalogDataSet]//printKVPL2

These are the items added to the NX_Catalog nexus collection

Add Part Records

addRecordsByName[modelName,partConceptNames,partDataSet]//printKVPL2

These are the items added to the NX_Part nexus collection

Add Supplier Records

addRecordsByName[modelName,supplierConceptNames, supplierDataSet]//printKVPL2

These are the items added to the NX_Supplier nexus collection

Get Items from The Collections of a Model

Get Items from the Catalog Concepts

getItemsFromConceptNames[modelName,catalogConceptNames]//printKVPL2

Get Items from the Part Concepts

getItemsFromConceptNames[modelName,partConceptNames]//printKVPL2

Get Items from the Supplier Concepts

getItemsFromConceptNames[modelName,supplierConceptNames]//printKVPL2

The Table and Record in AIR

Tables with Records from the Microsoft Access Database

tableNames=SQLTableNames[conn]

Map[TableForm, SQLSelect[conn,#,"ShowColumnHeadings"->True]&/@tableNames,{1}]

Assimilation of Parts Table in AtomicDB with Ouput in Wolfram Language

Execute Select Query to Read the Records

SQLExecute[conn,"select * from Parts","ShowColumnHeadings"->True]//TableForm

Same Result From AtomicDB

Get the Nexus Items

(nexusMemberItems=getItemsFromConceptNames[modelName,"NX_Part"])//printKVPL2

Get Part Concepts

partConcepts[0]//printKVPL2

Get Records

records =getAnything[model,partConcepts[0],nexusMemberItems,setType->enGridType]

Format the Output

res=coreKVPL1ToRules/@records@ToArray[];

res[[1]]

res[[2]]

res[[3]]

res[[4]]

res[[5]]

res[[6]]

res[[7]]

res[[8]]

res[[9]]

parts={
{{2,8,262,1}->1,{2,259,259,1}->"991",{2,263,263,1}->"Left Handed Bacon Stretcher Cover",{2,264,264,1}->"Red"},
{{2,8,262,2}->2,{2,259,259,2}->"992",{2,263,263,2}->"Smoke Shifter End",{2,264,264,2}->"Black"},
{{2,8,262,3}->3,{2,259,259,3}->"993",{2,263,263,3}->"Acme Widget Washer",{2,264,264,1}->"Red"},
{{2,8,262,4}->4,{2,259,259,4}->"994",{2,263,263,3}->"Acme Widget Washer",{2,264,264,3}->"Silver"},
{{2,8,262,5}->5,{2,259,259,5}->"995",{2,263,263,4}->"I Brake for Crop Circles Sticker",{2,264,264,4}->"Translucent"},
{{2,8,262,6}->6,{2,259,259,6}->"996",{2,263,263,5}->"Anti-Gravity Turbine Generator",{2,264,264,5}->"Cyan"},
{{2,8,262,7}->7,{2,259,259,7}->"997",{2,263,263,5}->"Anti-Gravity Turbine Generator",{2,264,264,6}->"Magenta"},
{{2,8,262,8}->8,{2,259,259,8}->"998",{2,263,263,6}->"Fire Hydrant Cap",{2,264,264,1}->"Red"},
{{2,8,262,9}->9,{2,259,259,9}->"999",{2,263,263,7}->"7 Segment Display",{2,264,264,7}->"Green"}
};

parts//TableForm