☂️ Handle all Zulip events to keep our data up to date

[gnprice]

We'll want to handle essentially all the events that can appear through the Zulip event system:
https://zulip.com/api/get-events#events

Some of these are correctness issues in the early prototype: we're tracking the data they would update, but are not handling the update, so our data goes stale. Others are not correctness issues because we're not yet tracking that data in the first place, but are for data we do intend to track before launch. Both those categories are in scope for this issue.

Out of scope for this issue: A few event types are for data that is only needed for areas of UI that we don't have in zulip-mobile and don't intend to have in zulip-flutter before launch. E.g., attachment, which is for web's #settings/uploaded-files screen.

Event types

• Edit this description to have a complete list of event types. (Include the types that are already handled or are out of scope; just mark them as such.)

For alpha

• heartbeat
• custom_profile_fields
• user_settings/update
• realm_user/update
• realm_user/add
• realm_user/remove
• stream/create Handle created/deleted streams (stream op: create and stream op: delete events) #181
• stream/delete
• message
• reaction/add Add Reaction class; uncomment Message.reactions and update on events #256
• reaction/remove
• update_message for edits (not moves) Handle edited messages (update_message events, 1/2) #118
• update_message_flags/add
• update_message_flags/remove

For beta

• subscription/add Subscription api events #382
• subscription/remove
• subscription/update still need msglist: Update message list on muting/unmuting a channel #1255
• user_topic Handle muted streams and topics #346, msglist: Handle UserTopicEvent, hiding/showing messages as needed #822
• submessage polls: Read-only support #165

For launch

• restart Handle restart events #1271
• realm_emoji/update store: Add realmEmoji, initialized from initial snapshot and updated with its event #394
• muted_users Mute muted users #296
• stream/update Handle updated streams (stream op: update events) #182
• delete_message Handle deleted messages (delete_message events) #120
• update_message for moves (vs. edits) Handle moved messages (update_message events, 2/2) #150
• typing/start Show "typing" status #665
• typing/stop Show "typing" status #665

For post-launch period

• realm/update Track realm settings #668
• realm/update_dict Track realm settings #668
• user_group/add Store user groups #662
• user_group/update
• user_group/add_members
• user_group/remove_members
• user_group/add_subgroups
• user_group/remove_subgroups
• user_group/remove
• presence Presence data, and report presence #196
• user_status Track user status #197
• realm_linkifiers (part of Local echo when sending messages #576)
• alert_words Handle alert words (pre-"watched phrases") #663
• drafts/add (cf Basic implementation of message drafts, to preserve composing progress #1217 but that's for local drafts)
• drafts/update
• drafts/remove
• has_zoom_token

For beyond post-launch

Other than realm settings UI

• saved_snippets/add
• saved_snippets/remove
• scheduled_messages/add
• scheduled_messages/update
• scheduled_messages/remove
• attachment/add
• attachment/update
• attachment/remove
• subscription/peer_add (cf Handle subscription peer add/remove events #374)
• subscription/peer_remove
• realm/deactivated

For realm settings UI

• default_stream_groups
• default_streams
• realm_playgrounds
• realm_domains/add
• realm_domains/change
• realm_domains/remove
• realm_export
• realm_export_consent
• realm_user_settings_defaults/update
• invites_changed
• realm_bot/add
• realm_bot/update
• realm_bot/remove
• realm_bot/delete

Never

Obsolete

• muted_topics Support muted_topics format for pre-Zulip Server 6 #422
• realm_filters
• update_display_settings
• update_global_notifications

Specific to web

• onboarding_steps
• web_reload_client

No longer exist

• hotspots

[gnprice]

I've given this the "beta" milestone, but there may be specific events we decide can wait for launch or even beyond. What's needed for beta is really just that we should look at the list and make any such choice explicitly. -> Filed #308 for this part.

[gnprice]

I've just gone through the list of all events in the server API, triaged them, and updated the issue description.

That was as of Zulip feature level 219, commit zulip/zulip@37b261e, for reference if updating the list for future server API changes.

For entertainment and future convenience, here's the pair of commands I used to get the list:
python -c 'import yaml, json; print(json.dumps(yaml.safe_load(open("zerver/openapi/zulip.yaml"))))' >tmp.json

<tmp.json jq '.paths["/events"].get.responses["200"].content["application/json"].schema.allOf[1].properties.events.items.oneOf[] | {type: .properties.type.allOf[1].enum[0], op: .properties.op?.enum[0]} | .type + if .op == null then "" else "/" + .op end' -r

The first command converts zerver/openapi/zulip.yaml (from the zulip/zulip tree) from YAML to JSON. The second command pulls out the event types. It gets tripped up on a couple of entries which use allOf differently, printing blank lines instead, so one can fix those up by hand. (Namely reaction/add and reaction/remove.) A robust version of this would have the allOf handled up front before trying to use jq.

And here's a complete list of the events, reordered lightly to group them logically, before triage:

All event types, grouped logically

Infra:
heartbeat
restart

Realm/server settings:
has_zoom_token
custom_profile_fields
default_stream_groups
default_streams
realm_linkifiers
realm_filters
realm_playgrounds
realm_emoji/update
realm_domains/add
realm_domains/change
realm_domains/remove
realm_export
realm/update
realm/update_dict
realm/deactivated
realm_user_settings_defaults/update

User settings:
alert_words
update_display_settings
update_global_notifications
user_settings/update
hotspots

Users, groups:
realm_user/update
realm_user/add
realm_user/remove
invites_changed
presence
user_status
muted_users
user_group/add
user_group/update
user_group/add_members
user_group/remove_members
user_group/add_subgroups
user_group/remove_subgroups
user_group/remove
realm_bot/add
realm_bot/update
realm_bot/remove
realm_bot/delete

Streams, subs:
stream/create
stream/delete
stream/update
subscription/add
subscription/remove
subscription/update
subscription/peer_add
subscription/peer_remove
muted_topics
user_topic

Messages:
message
reaction/add
reaction/remove
attachment/add
attachment/update
attachment/remove
submessage
delete_message
update_message
typing/start
typing/stop
update_message_flags/add
update_message_flags/remove
drafts/add
drafts/update
drafts/remove
scheduled_messages/add
scheduled_messages/update
scheduled_messages/remove

[gnprice]

I've updated the list for the current complete list of events in the server API, as of Zulip feature level 337, commit zulip/zulip@4385005.

I used essentially the same commands as last time, and compared the results between the commit recorded last time and the current commit. This variation on a previous command got the zerver/openapi/zulip.yaml from the old commit and converted it to JSON:
git show 37b261e:zerver/openapi/zulip.yaml | python -c 'import sys, yaml, json; print(json.dumps(yaml.safe_load(sys.stdin)))' >api-old.json

Then ran the same second command as last time, twice with two different inputs, and took the diff:
diff <(<api-old.json jq '.paths["/events"].get.responses["200"].content["application/json"].schema.allOf[1].properties.events.items.oneOf[] | {type: .properties.type.allOf[1].enum[0], op: .properties.op?.enum[0]} | .type + if .op == null then "" else "/" + .op end' -r) <(<api.json jq '.paths["/events"].get.responses["200"].content["application/json"].schema.allOf[1].properties.events.items.oneOf[] | {type: .properties.type.allOf[1].enum[0], op: .properties.op?.enum[0]} | .type + if .op == null then "" else "/" + .op end' -r)

The diff was:

35c35
< hotspots
---
> onboarding_steps
55a56
> realm_export_consent
62a64
> web_reload_client
67a70,71
> saved_snippets/add
> saved_snippets/remove

[gnprice]

OK, and I've reviewed the list of not-yet-handled events and re-grouped it for our current milestones. Filed one issue #1271 for an event I'd like to handle for the M5 Launch milestone; the full list remaining for launch are:

• msglist: Update message list on muting/unmuting a channel #1255
• Handle restart events #1271
• Mute muted users #296

With that, I'll move this umbrella issue out to the next milestone M6 Post-launch, to be reviewed again in that period.