Group events in (live) data view

[johanstokking]

Summary

Group events in (live) data view.

Why do we need this?

With multiple active devices, the application and gateway live traffic view becomes quite overwhelming soon.

What is already there? What do you see now?

One gateway uplink message emits three events: the receive, and two identical forwards (NS and PB, but that isn't shown). The gateway status also comes in threefold, one receive and two forwards:

One data uplink message emits at least six events, including storage. We may emit more events from integrations and packages.

With data downlink message we also have a bunch:

Same with join-request and join-accept.

What is missing? What do you want to see?

I would like to see grouping of events. Suggesting the following:

1. Gateway live data

   1. Group events gs.{up,status}.{receive,forward,drop} by correlation ID gs:uplink:XXX
   2. We cannot group gs.down.* events yet. Needs Send Tx events upstream #76, specifically that GS needs to correlate the correlation ID of the downlink message to the TX acknowledgment

2. Application and end device live data

   1. Group activation events ns.up.join.{receive,drop,process,accept.forward}, ns.up.join.{cluster,interop}.{attempt,success,fail} and ns.down.join.schedule.{attempt,success,fail} by correlation ID ns:uplink:XXX
   2. Group uplink message events ns.up.data.{receive,process,drop,forward}, as.up.data.{receive,drop,forward}, as.up.service.forward and as.up.data.decode.{fail,warning} events by correlation ID ns:uplink:XXX
   3. Group downlink message events as.down.data.{receive,forward,drop}, as.down.data.encode.{fail,warning} and ns.down.data.schedule.{attempt,success,fail} events by correlation ID as:downlink:XXX

Notes:

• This doesn't cover all events, but that's fine for now
• Grouping under (2) only works if both NS and AS are in the cluster. If it's NS or AS only, don't group

How do you propose to implement this?

For each of the groupings above, show one row, with icons and text to show what happened. For example, for each gs.up.forward event, put a forward icon in the row, and same for a failure.

It should still be possible to somehow expand or list the grouped events.

Implementation notes:

• This really needs to be implemented in the front-end; I don't see a role for the Event Server for this, as that is a proprietary component
• Historical events may be loaded, on which grouping should be applied in retrospect
• Events may arrive out-of-order, so grouping may happen in retrospect

How do you propose to test this?

Integration testing.

@rvolosatovs @adriansmares can help with all the scenarios and see if everything works as expected.

Can you do this yourself and submit a Pull Request?

Can review, test and provide input here if needed.

[kschiffer]

In the console meeting today, we concluded that we will want to solve the immediate issue by adding event filters first (which will be set by default, based on the number of entities in the event scope). Also we need to look into solving some performance issues of the event stream that might warrant some store and display logic refactors (see #2887). After this we should be good to take a look into grouping, which is unfortunately not straightforward and needs some proper planning both in terms of UX and implementation-wise.

[kschiffer]

Related: #3817

[johanstokking]

Planning gets tracked in #3817

[NicolasMrad]

Given the new verbose filter, which addresses some of the issues, this is not in the current plan