Event stream filters in console

[kschiffer]

Summary

We should provide filtering features for our data view / event stream.

Why do we need this?

• Closing v2 feature gap
• Supporting advance use cases for data monitoring
• Mitigate event flooding

What is already there? What do you see now?

Event streaming for all our entities.

What is missing? What do you want to see?

Filtering options to improve utility of the event stream

I've done a wireframe about this a while ago:

Some related discussion can be viewed here: https://github.com/TheThingsIndustries/lorawan-stack/issues/1071

We should discuss which forms of filters are viable.

Environment

v3.7.0

How do you propose to implement this?

I think we can make use of a special store selector selectFilteredEvents(filter) which selects event based on a defined filter object.

Can you do this yourself and submit a Pull Request?

Anyone please feel free to pickup.

[kschiffer]

With regards to #3713 (comment) we need to expand the functionality here a bit. There are scenarios where applications have hundreds or thousands of end-devices. This will crash the Console altogether when the event view is open. Hence, it is necessary to enforce some filter for heavy-load situations.

Such filters could be:

• Limit the entity scope for application event streams to max 20 (or so) end devices
• Only show the forward event type for uplinks downlinks and similar outfiltering of auxiliary event

We could use different levels of enforced filtering and use it for some predefined "events per second"-thresholds.

Ideas welcome cc @bafonins @htdvisser @johanstokking

[htdvisser]

The number of registered end devices isn't necessarily an indication of the amount of traffic, so I don't think it would be a good idea to do different things depending on the number of registered end devices.

My suggestion would be to extend the API to allow filtering on event name. Every events view in the Console would then give the user some UI where they can check/uncheck events they're interested in. The event names that are selected by default can then be adapted to show the most relevant events for a given entity.

For the Application overview, we can then filter on application.** and as.**, so that we only see application layer events, not all Network Server MAC events (for those, you'd have to open a specific end device)

[kschiffer]

Blocked on #3817

[kschiffer]

I had originally planned to fix this with a general overhaul of the event stream via #3817, but I think it's necessary to make some quick adjustments now to make the event stream less verbose. A simple solution here is to add a verbose switch to the event stream, which will enable users to filter out many events that only add noise to the event output.

I propose the following list of events to be shown by default, when verbosity is deactivated:

as.down.data.decode.fail
as.down.data.encode.fail
as.down.data.forward
as.packages.loraclouddmsv1.fail
as.packages.loracloudglsv3.fail
as.pubsub.fail
as.up.data.decode.fail
as.up.data.forward
gs.down.send
gs.down.tx.fail
gs.gateway.connect
gs.gateway.disconnect
gs.status.fail
gs.status.forward
gs.up.fail
gs.up.receive
js.join.accept
js.join.reject
ns.down.data.schedule.fail
ns.down.join.schedule.fail
ns.up.join.cluster.fail
ns.up.join.interop.fail
organization.*
user.*
gateway.*
application.*
end_device.*

I've selected these events based on what is likely of direct interest to users. This includes

• any failure event
• one event per received data message, which is the as.up/down.data.forward
• one event per join request/reject
• one event per received message for gateways
• gateway metrics (though i find that these are often sent many times repeatedly)
• any entity registry event (these are usually not very frequent)

Please review this list @htdvisser @johanstokking @neoaggelos @adriansmares, so I can finalize the list and coordinate implementation.

[neoaggelos]

The list of events LGTM in general with one exception: I would swap gs.status.forward with gs.status.receive, since the status messages are important for the GS, not the upstream hosts (at the moment at least). This also gives us 1 instead of 2 events per status message (we get one gs.status.forward per upstream host).

[htdvisser]

I think it would also be useful to have these:

• *.end_device.* -- end device updates on the NS/AS/JS are mostly useful when we also have history, because then you can compare behavior before/after the update.
• as.{up,down}.data.decode.warning -- we shouldn't ignore warnings
• as.{up,down}.data.drop -- dropped uplinks/downlinks in the AS are almost always abnormal
• as.up.join.drop -- neither are dropped joins in the AS
• as.down.data.queue.invalid/as.down.data.queue.lost (maybe, @adriansmares?)
• *.api-key.* -- these are security events
• *.collaborator.* -- these too
• ns.mac.*.answer.reject -- these indicate a problem in the MAC state, shouldn't be hidden
• ns.up.data.process -- this is the first event that contains a de-duplicated uplink

[kschiffer]

as.{up,down}.data.decode.warning -- we shouldn't ignore warnings

It's not ignoring them, just hiding them by default. I decided to filter warnings out by default, since this also what you'd expect when not being in verbose mode.

• as.{up,down}.data.drop -- dropped uplinks/downlinks in the AS are almost always abnormal
• as.up.join.drop -- neither are dropped joins in the AS

👌

• *.api-key.* -- these are security events
• *.collaborator.* -- these too

I guess that's fine since they also shouldn't be very frequent.

• ns.mac.*.answer.reject -- these indicate a problem in the MAC state, shouldn't be hidden

👌

• ns.up.data.process -- this is the first event that contains a de-duplicated uplink

I don't know I'd really have just one event for uplink messages, which is the one from the AS which also contains the decoded payload.

Remember that we are really trying to reduce the noise with this filter, so I think we should hide anything that is not typically of interest when viewing the event stream. Anything else can be viewed as usual when activating verbose mode.

[htdvisser]

As discussed with @kschiffer, we'll add support for filtering events by name on the backend.

I'll pick up the API and backend implementation after which @kschiffer will update the Console.

[htdvisser]

@kschiffer I just pushed the backend work to feature/2231-event-filter.

I can imagine that for the frontend it can be useful to know the full list of event names. In pkg/events/grpc/grpc.go you'll see how the EventsServer builds that list. We could write a little tool to extract the names into a JS/JSON file (similar to the one that creates messages.json).

[htdvisser]

@kschiffer did you still want to implement the rest of this or did you only want to pick up the part in #4831?

[kschiffer]

I can stay assigned here. I've removed the milestone for now.

[kschiffer]

Let's schedule this and aim for completion in Q3.

[KrishnaIyer]

We have basic filtering of events (verbose). The proposal here is great if we have a General "events" page on the console and we can then do some filtering there. If we have such a page, then indeed we can look at this generic filtering system.

However, the way most users (including me) use the console is to navigate to the target gateway, application or end device and look into the events there. Here, we can make a few informed choices about the filtering in each of those pages (contexts). That is handled here: #7342