Showing revision 6.0

Advanced JData Examples

1. Complex-number and complex-valued arrays
2. Sparse arrays
3. Complex-valued sparse arrays
4. Special matrices
5. Tables

The goal of this section is to show how one can use the lightweight and portable JData annotations to conveniently represent advanced data structures, including complex-valued arrays, sparse arrays, tables, trees, linked lists and graphs. The notation used in this section is similar to those used in the basic examples.

1. Complex-number and complex-valued arrays

Complex numbers are supported in many scientific programming languages, such as MATLAB, python and R, but there is no standardized format to store complex-valued data and share among these programming languages. Using the portable JData annotation system, we can conveniently represent complex-valued data and make sharing such data possible.

A complex-valued data record must be stored using the "annotated array format" as shown in the basic examples. This is achieved via the presence of "_ArrayIsComplex_" keyword and the serialization of the complex values in the "_ArrayData_" constructs in the order of [[serialized real-part values], [serialized imag-part values]]

Native data text-JData/JSON form binary-JData(UBJSON)

a=10.0+6.0j

 =>

{
"a":{
"_ArrayType_":"double",
"_ArraySize_":[1,1],
"_ArrayIsComplex_":true,
"_ArrayData_":[[10.0],[6.0]]
}
}

[{]
[U][1][a]
[{]
[U][11][_ArrayType_] [S][U][6][double]
[U][11][_ArraySize_] [[] [U][2][U][3] []]
[U][16][_ArrayIsComplex_] [T]
[U][11][_ArrayData_] [[] [[][D][10.0][]] [[][D][6.0][]] []]
[}]
[}]

a=[
1+2j,3+4j
5+6j,7+8j
]

 =>

{
"a":{
"_ArrayType_":"uint8",
"_ArraySize_":[2,2],
"_ArrayIsComplex_":true
"_ArrayData_":[[1,3,5,7],[2,4,6,8]]
}
}

[{]
[U][1][a]
[{]
[U][11][_ArrayType_] [S][U][5][uint8]
[U][11][_ArraySize_] [[] [U][2][U][2] []]
[U][16][_ArrayIsComplex_] [T]
[U][11][_ArrayData_] [[]
[[] [$][U][#][4] [1][3][5][7]
[[] [$][U][#][4] [2][4][6][8]
[]]
[}]
[}]

2. Sparse arrays

Sparse array is an important data type widely used in computational models. A small number of scientific programming languages provide built-in support to sparse arrays, such as MATLAB; others support sparse arrays via 3rd party libraries, such as scipy.sparse for Python, Blas for C/C++, Lapack for FORTRAN etc.

Similar to the above case, a sparse array must be stored using the "annotated array format" as shown in the basic examples. This is achieved via the presence of "_ArrayIsSparse_" keyword and the serialization of the non-zero sparse data elements and indices in the "_ArrayData_" constructs in the order of [[first index i1], [second index i2], ... [last index iN], [serialized non-zero values]]

Native data text-JData/JSON form binary-JData(UBJSON)

a=sparse(5,4);
a(1,1)=2.0;
a(2,3)=9.0;
a(4,2)=7.0;

 =>

{
"a":{
"_ArrayType_":"double",
"_ArraySize_":[5,4],
"_ArrayIsSparse_":true,
"_ArrayData_":[[1,2,4],[1,3,2],[2.0,9.0,7.0]]
}
}

[{]
[U][1][a]
[{]
[U][11][_ArrayType_] [S][U][6][double]
[U][11][_ArraySize_] [[] [U][2][U][3] []]
[U][16][_ArrayIsSparse_] [T]
[U][11][_ArrayData_] [[]
[[][$][U][#][3] [1][2][4]
[[][$][U][#][3] [1][3][2]
[[][$][D][#][3] [2.0][9.0][7.0]
[]]
[}]
[}]

3. Complex-valued sparse arrays

Combining the "_ArrayIsComplex_" and "_ArrayIsSparse_" tags, we can easily support complex-valued sparse arrays using JData annotations. The "_ArrayData_" stores the information regarding the non-zero values in the order of [[first index i1], [second index i2], ... [last index iN], [serialized real-part of non-zero values], [serialized imag-part of non-zero values]]

Native data text-JData/JSON form binary-JData(UBJSON)

a=sparse(5,4);
a(1,1)=2.0+1.2j;
a(2,3)=9.0-4.7j;
a(4,2)=7.0+1.0j;

 =>

{
"a":{
"_ArrayType_":"double",
"_ArraySize_":[5,4],
"_ArrayIsSparse_":true,
"_ArrayData_":[[1,2,4],[1,3,2],
[2.0,9.0,7.0],[1.2,-4.7,1.0]]
}
}

[{]
[U][1][a]
[{]
[U][11][_ArrayType_] [S][U][6][double]
[U][11][_ArraySize_] [[] [U][2][U][3] []]
[U][16][_ArrayIsSparse_] [T]
[U][11][_ArrayData_] [[]
[[][$][U][#][3] [1][2][4]
[[][$][U][#][3] [1][3][2]
[[][$][D][#][3] [2.0][9.0][7.0]
[[][$][D][#][3] [1.2][-4.7][1.0]
[]]
[}]
[}]

4. Special matrices

The light-weight data annotation mechanism provided by JData specification can further enable efficient storage of special matrices, such as diagonal, triangular, Toeplitz matrices etc. This is enabled via the "_ArrayShape_" keyword. For example

Native data text-JData/JSON form binary-JData(UBJSON)
diagonal matrix

a=diag([1.0,1.0,2.1,2.2,5.3])

 =>

{
"a":{
"_ArrayType_":"double",
"_ArraySize_":[5,5],
"_ArrayShape_":"diag",
"_ArrayData_":[1.0,1.0,2.1,2.2,5.3]
}
}

[{]
[U][1][a]
[{]
[U][11][_ArrayType_] [S][U][6][double]
[U][11][_ArraySize_] [[] [U][5][U][5] []]
[U][12][_ArrayShape_] [S][U][4][diag]
[U][11][_ArrayData_] [[] [$][U][D][#][U][5]
[1.0][1.0][2.1][2.2][5.3]
[}]
[}]

upper triangular matrix

11 12 13 14 15
0 22 23 24 25
a= 0 0 33 34 35
0 0 0 44 45
0 0 0 0 55

 =>

{
"a":{
"_ArrayType_":"int8",
"_ArraySize_":[5,5],
"_ArrayShape_":"upper",
"_ArrayData_":[11,12,13,14,15,22,23,...,55]
}
}

[{]
[U][1][a]
[{]
[U][11][_ArrayType_] [S][U][4][int8]
[U][11][_ArraySize_] [[] [U][5][U][5] []]
[U][12][_ArrayShape_] [S][U][5][upper]
[U][11][_ArrayData_] [[] [$][U][#][U][15]
[11][12][13][14][15][22][23]...[55]
[}]
[}]

symmetric matrix- store upper

11 12 13 14 15
12 22 23 24 25
a= 13 23 33 34 35
14 24 34 44 45
15 25 35 45 55

 =>

{
"a":{
"_ArrayType_":"int8",
"_ArraySize_":[5,5],
"_ArrayShape_":"uppersymm",
"_ArrayData_":[11,12,13,14,15,22,23,...,55]
}
}

[{]
[U][1][a]
[{]
[U][11][_ArrayType_] [S][U][4][int8]
[U][11][_ArraySize_] [[] [U][5][U][5] []]
[U][12][_ArrayShape_] [S][U][9][uppersymm]
[U][11][_ArrayData_] [[] [$][U][#][U][15]
[11][12][13][14][15][22][23]...[55]
[}]
[}]

real Toeplitz matrix

11 12 13 0 0
21 11 12 13 0
a= 31 21 11 12 13
41 31 21 11 12
0 41 31 21 11

 =>

{
"a":{
"_ArrayType_":"int8",
"_ArraySize_":[5,5],
"_ArrayShape_":"toeplitz",
"_ArrayData_":[[11,12,13 0],[11,21,31,41]]
}
}

[{]
[U][1][a]
[{]
[U][11][_ArrayType_] [S][U][4][int8]
[U][11][_ArraySize_] [[] [U][5][U][5] []]
[U][12][_ArrayShape_] [S][U][8][toeplitz]
[U][11][_ArrayData_] [[]
[[] [$][U][#][U][4] [11][12][13][0]
[[] [$][U][#][U][4] [11][21][31][41]
[]]
[}]
[}]

complex diagonal matrix

a=diag([1+2j,4-j,5j,10,12+7j])

 =>

{
"a":{
"_ArrayType_":"int8",
"_ArraySize_":[5,5],
"_ArrayShape_":"diag",
"_ArrayIsComplex_": true,
"_ArrayData_":[[1,4,0,10,12],[2,-1,5,0,7]]
}
}

[{]
[U][1][a]
[{]
[U][11][_ArrayType_] [S][U][4][int8]
[U][11][_ArraySize_] [[] [U][5][U][5] []]
[U][12][_ArrayShape_] [S][U][4][diag]
[U][16][_ArrayIsComplex_] [T]
[U][11][_ArrayData_] [[]
[[] [$][i][#][U][5] [1][4][0][10][12]
[[] [$][i][#][U][5] [2][-1][5][0][7]
[]]
[}]
[}]

5. Tables

Native datatext-JData/JSON formbinary-JData(UBJSON)
a table without row-name 
Name    Age   Degree  Height
----  ------- ------  ------
Andy    21     BS      69.2
William 21     MS      71.0
Om      22     BE      67.1
=> 
{
  "_TableCols_": ["Name", "Age", 
                 "Degree", "Height"],
  "_TableRows_": [],
  "_TableRecords_": [
    ["Andy",    21, "BS", 69.2],
    ["William", 21, "MS", 71.0],
    ["Om",      22, "BS", 67.1]
  ]
}

[{]
 [U][11][_TableCols_] [[]
   [S][U][4][Name] [S][U][3][Age] 
   [S][U][6][Degree] [S][U][6][Height] 
 []]
 [U][11][_TableRows_] [[] []]
 [U][14][_TableRecords_] [[]
   [[] [S][U][4][Andy] [U][21] [S][U][2][BS] [d][69.2] []]
   [[] [S][U][7][William] [U][21] [S][U][2][MS] [d][71.0] []]
   [[] [S][U][2][Om] [U][22] [S][U][2][BS] [d][67.1] []]
 []]
[}]
specifying column data types=> 
{
  "_TableCols_": [
    {"DataName":"Name",
     "DataType":"string"
    },
    {"DataName":"Age",
     "DataType":"int32"
    },
    {"DataName":"Degree",
     "DataType":"string"
    },
    {"DataName":"Height",
     "DataType":"single"
    }
  ],
  "_TableRows_": [],
  "_TableRecords_": [
    ["Andy",    21, "BS", 69.2],
    ["William", 21, "MS", 71.0],
    ["Om",      22, "BS", 67.1]
  ]
}

[{]
 [U][11][_TableCols_] [[]
   [{] [S][U][8][DataName] [S][U][4][Name]
       [S][U][8][DataType] [S][U][6][string]
   [}]
   [{] [S][U][8][DataName] [S][U][3][Age] 
       [S][U][8][DataType] [S][U][5][int32]
   [}]
   [{] [S][U][8][DataName] [S][U][6][Degree] 
       [S][U][8][DataType] [S][U][6][string]
   [}]
   [{] [S][U][8][DataName] [S][U][6][Height] 
       [S][U][8][DataType] [S][U][6][single]
   [}]
 []]
 [U][11][_TableRows_] [[] []]
 [U][14][_TableRecords_] [[]
   [[] [S][U][4][Andy] [U][21] [S][U][2][BS] [d][69.2] []]
   [[] [S][U][7][William] [U][21] [S][U][2][MS] [d][71.0] []]
   [[] [S][U][2][Om] [U][22] [S][U][2][BS] [d][67.1] []]
 []]
[}]
Powered by Habitat