[BUG] Token Counting Error Despite DISABLE_TELEMETRY OR METRICSSetting

[Hbmr9000]

Describe the bug
A clear and concise description of what the bug is:
When executing Flowise with DISABLE_TELEMETRY=true, there is an error occurring. Initially, it was assumed to be related to node-usage-telemetry, as the IPs belong to Cloudflare/US, suggesting a connection to Flowise Cloud. However, the issue persists even with telemetry disabled. Concerns arise regarding how token counting is performed, as it might imply that prompt content is sent to an external service. Alternatively, it could be related to Flowise Cloud rejecting requests due to the absence of an enterprise license.

---

To Reproduce
Steps to reproduce the behavior:

1. Set DISABLE_TELEMETRY=true in the environment variables.
2. Start Flowise (e.g., using Docker or another installation method).
3. Execute a flow with token counting or a feature that seems to trigger the error.
4. Observe the error in the logs or output.

---

Expected behavior
Token counting and related operations should function locally without external dependencies, especially when telemetry is disabled. This behavior should be an opt-in, to prevent unnecessary/unwanted communication with third-parties.

---

Screenshots
Failed to calculate number of tokens, falling back to approximate count TypeError: fetch failed
at node:internal/deps/undici/undici:13178:13
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async RetryOperation._fn (C:\Users\USERNAME\AppData\Local\npm-cache_npx\8b95a19c5c9c8708\node_modules\p-retry\index.js:50:12) {
[cause]: ConnectTimeoutError: Connect Timeout Error (attempted addresses: 104.21.32.1:443, 104.21.64.1:443, 104.21.48.1:443, 104.21.96.1:443, 104.21.80.1:443, 104.21.16.1:443, 104.21.112.1:443, timeout: 10000ms)
at onConnectTimeout (C:\Users\USERNAME\AppData\Local\npm-cache_npx\8b95a19c5c9c8708\node_modules\undici\lib\core\connect.js:237:24)
at Immediate._onImmediate (C:\Users\USERNAME\AppData\Local\npm-cache_npx\8b95a19c5c9c8708\node_modules\undici\lib\core\connect.js:188:35)
at process.processImmediate (node:internal/timers:483:21)
at process.topLevelDomainCallback (node:domain:160:15)
at process.callbackTrampoline (node:internal/async_hooks:128:24) {
code: 'UND_ERR_CONNECT_TIMEOUT'
}
}

Setup

• Installation: npx flowise start --DEBUG=true --DATABASE_PATH="D:\flowise" --DISABLE_FLOWISE_TELEMETRY=TRUE
• Flowise Version: 2.2.4
• OS: Win10
• Browser: Edge

[Hbmr9000]

I additionally tried with a docker-compose setup .. it seems that the env-variables (ENABLE_METRICS=false, METRICS_INCLUDE_NODE_METRICS=false) dont have an effect. no behavior is changed and also in the running container I dont see those listed. trying the same with npx i receive actually an error about unknown variables...

[swami-sanapathi]

looks like the issue is related to #3484

[HenryHengZJ]

Im not sure if this is related to metrics/telemetry as there is no token involved there.

Are you seeing this on specific model? For example, when I use o3 mini, I can see this error because tiktoken is not yet updated with the latest model

[Hbmr9000]

For me it's for both GPT4o & mini the case actually..! And it's also certainly related to our enterprise firewall that have blocked those ports. I suppose it would be great to have some env-variables to deactivate this..