NewMessage event callback taking too long to be triggered on big channels

[victorbrjorge]

I have an app that uses Telethon to listen to messages from some big channels (2k+ subscribers) and relay them. Since version 0.17 I've been noticing some delays in it. So I made the code below to test if the delay was on Telethon and discovered that telethon has been taking over 20s average to run the callback function.

    client = TelegramClient('session', api_id, api_hash, update_workers=1, spawn_read_thread=False).start(phone)

    try:
        client.get_me()
    except RuntimeError:
        print('Couldn\'t login to Telegram. Exiting...')
        sys.exit(0)

    channel= client.get_input_entity('A BIG CHANNEL')
    

    @client.on(events.NewMessage(chats=channel, incoming=True))
    def callback(event):
        print('Message at {} UTC'.format(event.message.date))
        print('Delay was {}s\n'.format((datetime.utcnow() - event.message.date).seconds))

    print('Listening for messages...')
    client.idle()

OUT:

Message at 2018-02-28 03:50:33 UTC
Delay was 52s

Message at 2018-02-28 03:54:23 UTC
Delay was 26s

When listening from a test channel with just me as a subscriber, there is no delay. I've been thinking about rolling back my Telethon version, but maybe you guys can help me with it.

UPDATE:

I rolled back the version of the library to 0.16.2 and the issue still persists. I guess it has something to do with the Telegram API itself. Anyway, can anyone help me with this, regardless?

[Lonami]

client.idle() is pretty much a while True loop:

Telethon/telethon/telegram_bare_client.py

Lines 635 to 649 in 229cd78

	while self._user_connected:
	try:
	if datetime.now() > self._last_ping + self._ping_delay:
	self._sender.send(PingRequest(
	int.from_bytes(os.urandom(8), 'big', signed=True)
	))
	self._last_ping = datetime.now()
	if datetime.now() > self._last_state + self._state_delay:
	self._sender.send(GetStateRequest())
	self._last_state = datetime.now()
	__log__.debug('Receiving items from the network...')
	self._sender.receive(update_state=self.updates)
	except TimeoutError:

No sleeps. As little delay as possible. The .receive method is as straightforward as possible:

Telethon/telethon/network/mtproto_sender.py

Lines 132 to 134 in 229cd78

	try:
	body = self.connection.recv()
	except (BufferError, InvalidChecksumError):

As soon as an update arrive it gets processed on the (updates) state:

Telethon/telethon/network/mtproto_sender.py

Lines 258 to 259 in 229cd78

	if state:
	state.process(obj)

Then one of many workers poll these queue:

Telethon/telethon/update_state.py

Lines 105 to 111 in 229cd78

	def _worker_loop(self, wid):
	while True:
	try:
	update = self.poll(timeout=UpdateState.WORKER_POLL_TIMEOUT)
	if update and self.handler:
	self.handler(update)
	except StopIteration:

If you can spot the bottleneck (is it in the code, or is Telegram itself actually taking long to deliver the updates?) and you can think of a fix, I'd appreciate :)

[realiza]

@victorbrjorge
Did you find a solution for this problem?

[SvehlaJan]

Bump. Any update on this? I'm averagely getting ~5 sec delay. Even on semi-big channels (6k users). Unfortunately I didn't find any bottleneck in Telethon. Thanks.

[Lonami]

Do you guys constantly receive updates during that time and, by the time update X arrives it's been delayed by a certain amount of time, or you receive no updates at all and then X arrives delayed? If the former, note that deserializing also takes a while so maybe that's the reason. Other than that like I stated I'm constantly reading from the socket already. Although cryptg makes a notable difference when working with files maybe that also helps when decrypting updates.

[SvehlaJan]

@Lonami For me it seems like the first variant.
I'm constantly receiving updates. Mostly one, max 10 chats:

@telegram_api.on(events.NewMessage(chats=target_channels))
def update_handler(update):
	if update.is_channel:
		handle_message(update.message)
			
telegram_api.start()

How I measure the delay:

def handle_message(message):
	...
	message_date = pytz.utc.localize(message.date)
	td = datetime.now(timezone.utc) - message_date
	print(str(td))
	...

Messages are mostly text-only. 200 char max. They contain some emoticons from time-to-time.
Let me know if I can help any more.

[SvehlaJan]

Update:
It might be caused by some batch-processing. This is log from my watcher of all channels:

22.190 Channel ID: - td: 0:00:02.190593
33.798 Channel ID: - td: 0:00:04.798141
04.907 Channel ID: - td: 0:00:00.907096
04.929 Channel ID: - td: 0:00:19.929582
05.004 Channel ID: - td: 0:00:14.004144
05.076 Channel ID: - td: 0:00:12.076281
05.133 Channel ID: - td: 0:00:03.133624
15.144 Channel ID: - td: 0:00:03.144891
50.108 Channel ID: - td: 0:00:14.108735
50.149 Channel ID: - td: 0:00:01.149451
22.287 Channel ID: - td: 0:00:05.287781
19.110 Channel ID: - td: 0:00:07.110851
19.145 Channel ID: - td: 0:00:05.145616

The first number is current time (seconds.millis)

Another:

21.753 Channel ID: - td: 0:00:01.753291
21.759 Channel ID: - td: 0:00:19.759108
21.960 Channel ID: - td: 0:00:01.960183
21.979 Channel ID: - td: 0:00:13.979178
22.124 Channel ID: - td: 0:00:11.124285
22.149 Channel ID: - td: 0:00:39.149634
43.162 Channel ID: - td: 0:00:15.162681
43.197 Channel ID: - td: 0:00:09.197945
43.234 Channel ID: - td: 0:00:08.234156
43.589 Channel ID: - td: 0:00:15.589270
43.620 Channel ID: - td: 0:00:05.620091
54.638 Channel ID: - td: 0:00:02.638004
43.561 Channel ID: - td: 0:00:06.561872
43.689 Channel ID: - td: 0:00:40.689757

[Lonami]

Some words from @delivrance and ed (adapted for this issue):

(regarding this issue)
<ed> Happens with bot callbacks too
<ed> I asked telegram support, they told me they broadcast messages from big channels in chunks
<delivrance> Guessed
<delivrance> It's not a lib problem
<delivrance> Telegram has to send a single message to thousand, maybe millions of users at once
<delivrance> It's just how telegram works, a delay would be normal and it's still pretty fast imo
<delivrance> Groups might be different than channels, groups probably have a higher priority due to live conversations

So probably not much we can do. Feel free to investigate further to check whether the server is just slow sending the updates or what's going on. Maybe create a clean account and simply call client.sender._receive() in a loop, early return to avoid decrypting the messages and just print them when they arrive in an isolated environment. If they're received quickly then the bottleneck is in the library, otherwise Telegram just takes long to send the updates.

[Lonami]

I might add, I haven't noticed any delays, but I'm not in big groups or channels either besides @Telegram, which hardly ever posts.

[waifwaif]

hello,

I believe I am experiencing a similar issue. Is there any fix or is it telegram's API? notifications for my own channel go through instantly but Binance (huge obv) has a 30 second delay.

[Lonami]

There's a timeout somewhere returned by the API which indicates how often one should call getDifference. I guess you can manually "watch" channels you're interested in with it, but I've never tried this because I've never seen any delays, so while I think it works I'm not sure. Do not ask how to use it here, we have a group for that and StackOverflow.

[darek292]

Hi @Lonami - I apologise for resurrecting an old thread. I've been running in the same issues and have noticed Pyrogram is much faster using the exact same code to receive new messages, this would suggest that there is something in the library?

For moderate channels, I'm seeing results such as:
| --- Message from -100xxxx59 ---
| > PG Delay of 0.6098659038543701 seconds

| --- Message from 146xxxx59 ---
| > TT Delay of 4.365129709243774 seconds

I'm not 100% sure if this is because they pull the difference more often or not, but just thought I'd let you know. This is consistent over a lot of messages, not a single sample size (using the same channel with approx 4-5000 users).

Code for Telethon:

@client.on_message()
def my_handler(client, event):
    print(time() - event.date.timestamp())

Code for Pyrogram:

@client.on(events.NewMessage())
async def my_event_handler(event):
    print(time() - event.date)

[Lonami]

The same delay has been observed in the opposite direction (Telethon fast, other libraries slow). I can't speak for what others are doing. Telethon relies on Telegram to send updates as soon as possible, and if it doesn't, then well, delays occur. I believe I have already checked and there is almost no delay between "network packet" and "handle update" (so the slow part is Telegram delivering the update).

[KrishJodu761]

I have an app that uses Telethon to listen to messages from some big channels (2k+ subscribers) and relay them. Since version 0.17 I've been noticing some delays in it. So I made the code below to test if the delay was on Telethon and discovered that telethon has been taking over 20s average to run the callback function.

    client = TelegramClient('session', api_id, api_hash, update_workers=1, spawn_read_thread=False).start(phone)

    try:
        client.get_me()
    except RuntimeError:
        print('Couldn\'t login to Telegram. Exiting...')
        sys.exit(0)

    channel= client.get_input_entity('A BIG CHANNEL')
    

    @client.on(events.NewMessage(chats=channel, incoming=True))
    def callback(event):
        print('Message at {} UTC'.format(event.message.date))
        print('Delay was {}s\n'.format((datetime.utcnow() - event.message.date).seconds))

    print('Listening for messages...')
    client.idle()

OUT:

Message at 2018-02-28 03:50:33 UTC
Delay was 52s

Message at 2018-02-28 03:54:23 UTC
Delay was 26s

When listening from a test channel with just me as a subscriber, there is no delay. I've been thinking about rolling back my Telethon version, but maybe you guys can help me with it.

UPDATE:

I rolled back the version of the library to 0.16.2 and the issue still persists. I guess it has something to do with the Telegram API itself. Anyway, can anyone help me with this, regardless?

I am also facing same issue but less delay like 1 or 2 or 3 or 5 seconds, But i need to get 0 sec delay means it should arrive when it sent, Is that delay problem solved? please provide some information or code regarding to that, so that will very helpful for those who are facing this issue.

[pedroota]

Hi, guys! I'm facing the exact same issue. The NewMessage event is taking too long, anyone haves a solution?

[t00ts]

+1

Been noticing this for a couple weeks now and it's quite suboptimal to say the least. Surprised to see this issue from 6y ago with no clear findings.

[yushilong]

Hi, guys! I'm facing the exact same issue. The NewMessage event is taking too long, anyone haves a solution?

[ghost]

+1

[ibrahimuzunj]

I have the same issue. Anyone can confirm that Pyrogram is better with delays than Telethon?

[manojsitapara]

same issue I am facing

[francestu96]

Same here!
If there is no way to make the event triggered automatically, can someone post the code to manually poll the channels?

I cannot find how to make it since by looking for "poll" in Telegram I found so many ""survey"" stuff instead which make a lot of noise...

[Lonami]

#4345.