Data vizualisation

I receive many data reprensented as json. Each of these json can have fields like time, fieldA,fieldB,fieldC,fieldD,fieldE etc… I want to have a graph representation to detect json linked together. By linked, it means have a correlation on one or multiple fields. The stronger the json are, the closest they have to be. I was thinking about having a view where the node are json, the edges are the similarity distance. The json arrive in real time.

1. Preprocessing the Data
• Calculate Similarities: For each pair of json , compute a similarity score based on the fields.

2. Defining Nodes and Edges
- Nodes: Each json is a node in your graph.
- Edges: Create an edge between two nodes if their similarity score exceeds a certain threshold. The similarity score can also define the weight of the edge.

3. Calculating Edge Lengths
- The edge length can be inversely proportional to the similarity score: stronger similarity (higher score) results in shorter edges.

4. Visualization
- Choose a Visualization Tool: Tools like D3.js, Cytoscape.js, or Gephi can be used for this purpose.
- Custom Visualization Logic: Implement logic to convert your JSON data into a graph format that your chosen tool can understand. This will involve:
- Creating a data structure for nodes (json) and edges (similarities).
- Applying the similarity calculations to determine which json are connected.
- implementing a layout algorithm that reflects the edge lengths based on similarity scores.

5. Interactive Features
- Implement interactive features like zooming, panning, and tooltips to provide more information about each node and edge.
- Allow users to filter the time or adjust the threshold for similarity to see different levels of correlation.

6. Performance Considerations

For large numbers of json, you might need to implement performance optimizations, like clustering or simplifying the graph.