Please believe me, JData is here to help! It is not a new format, and is very simple, easy-to-adopt. Read below to see why.
No matter if you are a believer or not that we are living in the era of big data, we all have to deal with data and files from time to time, and we all know that there are numerous file formats that require specific software to open them. In most cases, you can't just open a file using a text editor and make sense out of it because the files are mostly binary and not directly human-readable.
This is bad for scientists -- people who generated a lot of complicated data and process them to understand what's going on in our world -- because they have to deal with increasing types of data files from increasingly complicated experiments. This is also bad for software developers -- people who write software to read/process/display certain type(s) of data and help you understand and process them -- because they have to constantly add libraries or parsers to deal with newly emerged formats and, if the formats are evolving, their future revisions. This is a lot of work and overhead to keep-up, and costly too.
The result of this is that software becomes bulky, with large number of dependencies, and increasingly difficult to maintain and improve. On the other hand, the numerous language-/application-/device-specific data formats, sometimes storing precious experimental data that costed a fortune to collect, are difficult to be shared, reused and understood, because you need a lot of dependent software tools to open them and understand. Even worse, many existing data formats utilizes rigid binary structures that are nearly impossible to extend (such as adding or removing something) without breaking backward-compatibility! Once a data format is revised, most data stored in the old format inevitably become outdated because no developer is interested in maintaining a parser of an outdated format.
People have noticed this problem, and had come up with a number of solutions, but each solution has met their challenges as well. For example:
On top of this, most popular programming languages that used to make data processing software had developed a series of their own complex data types to efficiently processing complicated inputs, but it is currently lacking a common interface for these software to exchange such complex data types from software wrote in one language to another. For example, the data types data scientists or researchers may often use include
|associative array||dict()||struct()/containers.Map()||%associative array||object, Map()||std::map|
|linked lists||not built-in||-||-||-||std::list|
To solve this problem, we need a data representation (and/or) storage format that is:
The above mentioned general-purpose data formats - XML, JSON, binary JSON or HDF5 - all have potentials, but all need improvements in order to address their limitations. In addition, we need standardization - the representation of a complex data type, event though there are numerous ways to do it, need to have a clearly defined syntax/specification so that they can be unambiguously encoded, shared, and exchanged among software and programming languages.
That is what we are trying to achieve with JData. JData is a standardized, portable middleware-like data annotation protocol that allows one to store complex data types, ranging from simple numerical arrays and values to sparse, complex, hierarchical graph-like data, easily, efficiently, and ambiguously. Such data annotation can also be stored as compact, human-readable files and share among different programs.
JData is our solution to help data creators and users, especially scientific researchers who routinely deal with complex data, to easily store, read, understand, share, combine, process, exchange and integrate their data and enable new discoveries. They can save all the extra works to deal with numerous diverse file formats, while focusing on answering the more important questions. Their hard-work generated data files can be also be shared with a large research community by making them FAIR - findable, accessible, interoperable and reusable, helping other researchers to create bigger studies and grander discoveries.
We have developed the JData framework to specifically address the above mentioned challenges. This framework includes
Simplicity and human-readability are among the top priorities in our design of this framework because we understand that for a data standard targeted for general audiences, it has to be simple! The format has to be simple and intuitive, the serialization, reading/writing of the file has to be easily and widely acceptable without much programming overhead, and future extension must be inherent - once a data file is created, it must be easily modified without worrying about breaking the parsers, so it can stay for a long time. Human-readability is harder to achieve, because it typically conflicts with speed and data size. However, our solution attempts to strike a balance between readability and efficiency, and ensures that all "semantic components", such as data item names, strings, and data types, are easily readable and understood, while the binary data payload can be stored with compact size with compression or filters.