No matter if you are a believer or not that we are living in the era of big data, we all have to deal with data and files from time to time, and we all know that there are numerous file formats that require specific software to open them. In most cases, you can't just open a file using a text editor and make sense out of it because those files are mostly binary and not directly human-readable.
This is bad for scientists -- people who generate a lot of complicated data and process them to understand what's going on in our world -- because they have to deal with increasing types of data files from increasingly complicated experiments. This is also bad for software developers -- people who write software to read/process/display certain type(s) of data and help you understand and process them -- because they have to constantly add libraries and parsers to deal with newly emerged formats and, if the format is updated, their future revisions. This is a lot of work and overhead to keep-up, and costly too.
The result of this is that software becomes bulky, with large number of dependencies, and increasingly difficult to maintain and improve. On the other hand, the numerous language-/application-/device-specific data files, sometimes storing precious experimental data that costed a fortune to collect, are difficult to be shared, reused and understood, because you need a lot of dependent software tools to open them and understand. Even worse, many existing data formats utilizes rigid binary structures that are nearly impossible to extend (such as adding or removing something) without breaking backward-compatibility! Once a data format is revised, most data stored in the old format inevitably become outdated because no developer is interested in maintaining a parser of an outdated format.
People have noticed this problem, and had come up with a number of solutions, but each solution has met their challenges as well. For example:
On top of this, most popular programming languages that are used to make data processing software had developed a series of their own complex data types to efficiently process diverse input data and represent sophisticated relations, but it is currently lacking a common interface for these software to exchange such complex data types from software written in one language to another. For example, the data types data scientists or researchers may often use include
Data types | Python | MATLAB | Perl | JavaScript | C++ |
---|---|---|---|---|---|
flexible array | list() | cell() | array | array | - |
hash/map | dict() | struct()/containers.Map() | %hash | object, Map() | std::map |
packed-typed-arrays | numpy.NDArray | matrices | PDL::matrix | TypedArray, numjs.NdArray | std::array |
tables | pandas.DataFrame | table | Data::Table | - | - |
linked lists | - | - | - | - | std::list |
graph | - | graph()/digraph() | - | - | - |
blob | bytes/bytearray | uint8/int8/char vector | File::BLOB | Blob() | void* |
complex | complex/numpy.dtype=complex | complex | Math::Complex | numjs.ndarray(dtype='complex') | std::complex |
sparse array | scipy.sparse | sparse | - | Map() | - |
special matrices | scipy.sparse | sparse | - | - | - |
To solve this problem, we need a data representation (and/or) storage format that is:
The above mentioned general-purpose data formats - XML, JSON, binary JSON or HDF5 - all have potentials, but all need improvements in order to address their limitations. In addition, we need standardization - the representation of a complex data type, event though there are numerous ways to do it, need to have a clearly defined syntax/specification so that they can be unambiguously encoded, shared, and exchanged among software and programming languages.
That is what we are trying to achieve with JData. JData is a standardized, portable middleware-like data annotation protocol that allows one to store complex data types, ranging from simple numerical arrays and values to sparse, complex, hierarchical graph-like data, easily, efficiently, and unambiguously. Such data annotation can also be stored as compact, human-readable files and share among different programs.