Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat_: init logs in InitializeApplication #6117

Merged
merged 8 commits into from
Nov 29, 2024

Conversation

igor-sirotin
Copy link
Collaborator

@igor-sirotin igor-sirotin commented Nov 22, 2024

Needed for status-im/status-desktop#16414

Description

  1. Moved logic from InitLogging endpoint to InitializeApplication.
    No need for a separate endpoint.
    InitLogging is now deprecated.

  2. Simplified arguments for logging initialization:

    • LogDir (and fallback to DataDir if empty)
    • LogEnabled
    • LogLevel
    • APILoggingEnabled (renamed from LogRequestGo, because we don't just log requests, we also log responses. And might add signals there as well)

    Log filenames is defined in status-go now:

    • geth.log for main log
    • api.log for API log
  3. LogSettings.MobileSystem is now deprecated as redundant, any value will be overridden with false.
    It was always set to false on status-mobile. And was never set in status-desktop.

  4. Extracted some lower-level functions in mobile/callog/status_request_log.go
    Needed for custom cases like InitializaApplication: we enable the API logging in this endpoint, so we need to attempt logging AFTER the call finishes.

  5. Added a functional test for this 🎉

@igor-sirotin igor-sirotin self-assigned this Nov 22, 2024
@igor-sirotin igor-sirotin changed the title feat: init logs in InitializeApplication feat_: init logs in InitializeApplication Nov 22, 2024
@status-im-auto
Copy link
Member

status-im-auto commented Nov 22, 2024

Jenkins Builds

Click to see older builds (67)
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ c3909f7 #1 2024-11-22 16:31:47 ~4 min macos 📦zip
✔️ c3909f7 #1 2024-11-22 16:31:54 ~4 min linux 📦zip
✔️ c3909f7 #1 2024-11-22 16:32:20 ~5 min ios 📦zip
✔️ c3909f7 #1 2024-11-22 16:32:23 ~5 min macos 📦zip
✔️ c3909f7 #1 2024-11-22 16:32:26 ~5 min tests-rpc 📄log
✔️ c3909f7 #1 2024-11-22 16:33:10 ~5 min windows 📦zip
✔️ c3909f7 #1 2024-11-22 16:33:18 ~6 min android 📦aar
✖️ c3909f7 #1 2024-11-22 17:00:58 ~33 min tests 📄log
✔️ f782827 #2 2024-11-23 17:44:19 ~4 min windows 📦zip
✔️ f782827 #2 2024-11-23 17:44:40 ~4 min linux 📦zip
✔️ f782827 #2 2024-11-23 17:45:02 ~4 min macos 📦zip
✖️ f782827 #2 2024-11-23 17:45:12 ~5 min tests-rpc 📄log
✔️ f782827 #2 2024-11-23 17:45:20 ~5 min android 📦aar
✔️ f782827 #2 2024-11-23 17:45:20 ~5 min ios 📦zip
✔️ f782827 #2 2024-11-23 17:47:11 ~7 min macos 📦zip
✖️ f782827 #3 2024-11-23 18:06:05 ~4 min tests-rpc 📄log
✔️ f782827 #4 2024-11-23 18:10:55 ~4 min tests-rpc 📄log
✔️ f782827 #2 2024-11-23 18:14:11 ~34 min tests 📄log
✖️ 2102cc2 #3 2024-11-25 21:36:16 ~2 min tests 📄log
✖️ 2102cc2 #5 2024-11-25 21:37:18 ~3 min tests-rpc 📄log
✔️ 2102cc2 #3 2024-11-25 21:37:40 ~4 min windows 📦zip
✔️ 2102cc2 #3 2024-11-25 21:37:53 ~4 min linux 📦zip
✔️ 2102cc2 #3 2024-11-25 21:38:10 ~4 min macos 📦zip
✔️ 2102cc2 #3 2024-11-25 21:38:21 ~5 min ios 📦zip
✔️ 2102cc2 #3 2024-11-25 21:38:32 ~5 min android 📦aar
✔️ 2102cc2 #3 2024-11-25 21:38:47 ~5 min macos 📦zip
79ef9ac #4 2024-11-25 23:06:46 ~1 min ios 📄log
79ef9ac #4 2024-11-25 23:06:59 ~2 min android 📄log
✖️ 79ef9ac #6 2024-11-25 23:07:30 ~2 min tests-rpc 📄log
79ef9ac #4 2024-11-25 23:07:34 ~2 min linux 📄log
79ef9ac #4 2024-11-25 23:07:34 ~2 min macos 📄log
✖️ 79ef9ac #4 2024-11-25 23:07:39 ~2 min tests 📄log
79ef9ac #4 2024-11-25 23:07:44 ~2 min windows 📄log
79ef9ac #4 2024-11-25 23:08:42 ~3 min macos 📄log
✖️ 5a18c97 #5 2024-11-25 23:29:22 ~2 min tests 📄log
✖️ 5a18c97 #7 2024-11-25 23:30:24 ~3 min tests-rpc 📄log
✔️ 5a18c97 #5 2024-11-25 23:30:42 ~3 min windows 📦zip
✔️ 5a18c97 #5 2024-11-25 23:31:19 ~4 min linux 📦zip
✔️ 5a18c97 #5 2024-11-25 23:31:51 ~5 min android 📦aar
✔️ 5a18c97 #5 2024-11-25 23:32:03 ~5 min macos 📦zip
✔️ 5a18c97 #5 2024-11-25 23:33:18 ~6 min ios 📦zip
✔️ 5a18c97 #5 2024-11-25 23:34:11 ~7 min macos 📦zip
✖️ 268277a #6 2024-11-26 14:38:08 ~3 min tests 📄log
✖️ 268277a #8 2024-11-26 14:38:49 ~3 min tests-rpc 📄log
✔️ 268277a #6 2024-11-26 14:39:03 ~4 min windows 📦zip
✔️ 268277a #6 2024-11-26 14:39:50 ~5 min linux 📦zip
✔️ 268277a #6 2024-11-26 14:39:50 ~5 min macos 📦zip
✔️ 268277a #6 2024-11-26 14:40:15 ~5 min ios 📦zip
✔️ 268277a #6 2024-11-26 14:40:28 ~5 min macos 📦zip
✔️ 268277a #6 2024-11-26 14:40:32 ~5 min android 📦aar
✖️ 494726d #9 2024-11-26 15:18:30 ~3 min tests-rpc 📄log
✔️ 494726d #7 2024-11-26 15:18:46 ~3 min windows 📦zip
✔️ 494726d #7 2024-11-26 15:19:23 ~4 min linux 📦zip
✔️ 494726d #7 2024-11-26 15:19:50 ~5 min android 📦aar
✔️ 494726d #7 2024-11-26 15:19:50 ~5 min macos 📦zip
✔️ 494726d #7 2024-11-26 15:20:02 ~5 min ios 📦zip
✔️ 494726d #7 2024-11-26 15:20:02 ~5 min macos 📦zip
✖️ 494726d #7 2024-11-26 15:21:59 ~7 min tests 📄log
✔️ 42772eb #8 2024-11-26 18:34:29 ~4 min linux 📦zip
✔️ 42772eb #8 2024-11-26 18:34:36 ~4 min windows 📦zip
✔️ 42772eb #10 2024-11-26 18:35:16 ~5 min tests-rpc 📄log
✔️ 42772eb #8 2024-11-26 18:35:24 ~5 min macos 📦zip
✔️ 42772eb #8 2024-11-26 18:35:24 ~5 min android 📦aar
✔️ 42772eb #8 2024-11-26 18:36:10 ~6 min ios 📦zip
✔️ 42772eb #8 2024-11-26 18:36:22 ~6 min macos 📦zip
✖️ 42772eb #8 2024-11-26 19:02:18 ~32 min tests 📄log
✔️ 42772eb #9 2024-11-26 20:28:54 ~31 min tests 📄log
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ e30c6aa #9 2024-11-28 11:22:33 ~4 min windows 📦zip
✔️ e30c6aa #9 2024-11-28 11:22:42 ~4 min linux 📦zip
✖️ e30c6aa #11 2024-11-28 11:23:17 ~5 min tests-rpc 📄log
✔️ e30c6aa #9 2024-11-28 11:23:42 ~5 min macos 📦zip
✔️ e30c6aa #9 2024-11-28 11:24:02 ~5 min ios 📦zip
✔️ e30c6aa #9 2024-11-28 11:24:07 ~5 min macos 📦zip
✔️ e30c6aa #9 2024-11-28 11:24:18 ~6 min android 📦aar
✔️ e30c6aa #10 2024-11-28 11:48:41 ~30 min tests 📄log
✔️ 831a743 #10 2024-11-28 11:54:46 ~3 min macos 📦zip
✔️ 831a743 #10 2024-11-28 11:55:24 ~4 min windows 📦zip
✔️ 831a743 #12 2024-11-28 11:55:52 ~4 min tests-rpc 📄log
✔️ 831a743 #10 2024-11-28 11:56:30 ~5 min linux 📦zip
✔️ 831a743 #10 2024-11-28 11:56:42 ~5 min android 📦aar
✔️ 831a743 #10 2024-11-28 11:56:53 ~6 min macos 📦zip
✔️ 831a743 #10 2024-11-28 11:57:57 ~7 min ios 📦zip
✖️ 831a743 #11 2024-11-28 12:20:23 ~29 min tests 📄log
✖️ 831a743 #12 2024-11-28 13:07:43 ~28 min tests 📄log
✔️ 831a743 #13 2024-11-28 13:37:25 ~29 min tests 📄log

@igor-sirotin igor-sirotin force-pushed the feat/InitializeApplication-init-logs branch from c3909f7 to f782827 Compare November 23, 2024 17:39
@igor-sirotin igor-sirotin marked this pull request as ready for review November 23, 2024 18:00
Copy link

codecov bot commented Nov 23, 2024

Codecov Report

Attention: Patch coverage is 41.79104% with 39 lines in your changes missing coverage. Please review.

Project coverage is 60.96%. Comparing base (991d5df) to head (831a743).
Report is 7 commits behind head on develop.

Files with missing lines Patch % Lines
mobile/status.go 0.00% 32 Missing ⚠️
mobile/callog/status_request_log.go 85.18% 2 Missing and 2 partials ⚠️
cmd/status-backend/main.go 0.00% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #6117      +/-   ##
===========================================
- Coverage    60.98%   60.96%   -0.02%     
===========================================
  Files          827      827              
  Lines       109734   109764      +30     
===========================================
- Hits         66924    66922       -2     
- Misses       34970    34992      +22     
- Partials      7840     7850      +10     
Flag Coverage Δ
functional 14.18% <7.46%> (-0.02%) ⬇️
unit 59.97% <43.75%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
api/defaults.go 77.50% <ø> (ø)
api/geth_backend.go 54.09% <ø> (-0.03%) ⬇️
cmd/statusd/main.go 39.52% <ø> (-0.19%) ⬇️
logutils/override.go 92.85% <ø> (-0.70%) ⬇️
nodecfg/node_config.go 65.27% <100.00%> (-0.91%) ⬇️
params/config.go 70.83% <ø> (ø)
protocol/messenger_store_node_request_manager.go 41.60% <ø> (ø)
protocol/requests/initialize_application.go 0.00% <ø> (ø)
wakuv2/history_processor_wrapper.go 100.00% <ø> (ø)
cmd/status-backend/main.go 0.00% <0.00%> (ø)
... and 2 more

... and 31 files with indirect coverage changes

mobile/status.go Show resolved Hide resolved
Copy link
Contributor

@qfrank qfrank left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, LGTM, minor comments left

logutils/override.go Outdated Show resolved Hide resolved
mobile/callog/status_request_log.go Outdated Show resolved Hide resolved
@igor-sirotin igor-sirotin force-pushed the feat/InitializeApplication-init-logs branch from 5a18c97 to 268277a Compare November 26, 2024 14:34
@igor-sirotin igor-sirotin force-pushed the feat/InitializeApplication-init-logs branch from 268277a to 494726d Compare November 26, 2024 15:14
@igor-sirotin
Copy link
Collaborator Author

Waiting for status-backend tests to be enabled to pass the coverage requirements.
Right now the added test is not executed on CI

@igor-sirotin igor-sirotin force-pushed the feat/InitializeApplication-init-logs branch from 42772eb to e30c6aa Compare November 28, 2024 11:17
@igor-sirotin
Copy link
Collaborator Author

TestInitializeLogging requires tests and status-backend to be running at same host/container. This is being implemented now, for now I'm skipping the test.

Below is the test passing locally:

TestInitializeLogging.test_init_logging

Launching pytest with arguments --status_backend_urls=http://localhost:12345 test_init_status_app.py::TestInitializeLogging::test_init_logging --no-header --no-summary -q in /Users/igorsirotin/Repositories/Status/status-go/tests-functional/tests

============================= test session starts ==============================
collecting ... collected 1 item

test_init_status_app.py::TestInitializeLogging::test_init_logging 
-------------------------------- live log call ---------------------------------
INFO     root:status_backend.py:40 Sending POST request to url http://localhost:12345/statusgo/InitializeApplication with data: {
    "apiLoggingEnabled": true,
    "dataDir": "/private/var/folders/g4/8wp8vfgs7lq_0s48d0ljqbzr0000gn/T/pytest-of-igorsirotin/pytest-79/test_init_logging0/data",
    "logDir": "/private/var/folders/g4/8wp8vfgs7lq_0s48d0ljqbzr0000gn/T/pytest-of-igorsirotin/pytest-79/test_init_logging0/logs",
    "logEnabled": true
}
INFO     root:signals.py:75 Connection opened
WARNING  websocket:_logging.py:66 websocket connected
INFO     root:status_backend.py:42 Got response: b'{"accounts":null,"centralizedMetricsInfo":{"enabled":false,"userConfirmed":false}}'
INFO     root:status_backend.py:48 Got response: b'{"accounts":null,"centralizedMetricsInfo":{"enabled":false,"userConfirmed":false}}'
PASSED

============================== 1 passed in 0.18s ===============================

Process finished with exit code 0

TestInitializeLogging.test_no_logging

Launching pytest with arguments --status_backend_urls=http://localhost:12345 test_init_status_app.py::TestInitializeLogging::test_no_logging --no-header --no-summary -q in /Users/igorsirotin/Repositories/Status/status-go/tests-functional/tests

============================= test session starts ==============================
collecting ... collected 1 item

test_init_status_app.py::TestInitializeLogging::test_no_logging 
-------------------------------- live log call ---------------------------------
INFO     root:status_backend.py:40 Sending POST request to url http://localhost:12345/statusgo/InitializeApplication with data: {
    "apiLoggingEnabled": false,
    "dataDir": "/private/var/folders/g4/8wp8vfgs7lq_0s48d0ljqbzr0000gn/T/pytest-of-igorsirotin/pytest-81/test_no_logging0/data",
    "logDir": "/private/var/folders/g4/8wp8vfgs7lq_0s48d0ljqbzr0000gn/T/pytest-of-igorsirotin/pytest-81/test_no_logging0/logs",
    "logEnabled": false
}
INFO     root:signals.py:75 Connection opened
WARNING  websocket:_logging.py:66 websocket connected
INFO     root:status_backend.py:42 Got response: b'{"accounts":null,"centralizedMetricsInfo":{"enabled":false,"userConfirmed":false}}'
INFO     root:status_backend.py:48 Got response: b'{"accounts":null,"centralizedMetricsInfo":{"enabled":false,"userConfirmed":false}}'
PASSED

============================== 1 passed in 0.14s ===============================

Process finished with exit code 0

Copy link
Contributor

@ilmotta ilmotta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a very minor comment/suggestion @igor-sirotin. Thanks for bringing yet another improvement!

accs, err := statusBackend.GetAccounts()
if err != nil {
return makeJSONResponse(err)
}
centralizedMetricsInfo, err := statusBackend.CentralizedMetricsInfo()

// Read centralized metrics info
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code speaks for itself without comment.

@@ -80,38 +97,69 @@ func initializeApplication(requestJSON string) string {
return makeJSONResponse(err)
}

// Initialize logs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Untested suggestion, but this log initialization section could be a separate function. Usually catches my attention whenever I see comments describing the what because the more useful comments are about the why and what comments can often times be made irrelevant by good function names and good function composition. Unfortunately in Go, function composition can be cumbersome without some abstractions.

func initLogs(request requests.InitializeApplication) error {
	if request.LogDir == "" {
		request.LogDir = request.DataDir
	}
	logSettings := logutils.LogSettings{
		Enabled: request.LogEnabled,
		Level:   request.LogLevel,
		File:    path.Join(request.LogDir, api.DefaultLogFile),
	}

	var err error
	if err = logutils.OverrideRootLoggerWithConfig(logSettings); err == nil {
		logutils.ZapLogger().Info("logging initialised",
			zap.Any("logSettings", logSettings),
			zap.Bool("APILoggingEnabled", request.APILoggingEnabled),
		)
	} else {
		return err
	}

	if request.APILoggingEnabled {
		logRequestsFile := path.Join(request.LogDir, api.DefaultAPILogFile)
		return requestlog.ConfigureAndEnableRequestLogging(logRequestsFile)
	}

	return nil
}
err = initLogs(request)
if err != nil {
	return makeJSONResponse(err)
}

@igor-sirotin
Copy link
Collaborator Author

@ilmotta thanks for the review 👍
I agree with both of your comments, but the tests seem to be quite flaky these days, so if you don't mind I'd apply your suggestions in a follow-up PR.

@igor-sirotin
Copy link
Collaborator Author

Coverage is there with TestInitializeLogging, but it's currently skipped, details here: #6117 (comment)

@igor-sirotin igor-sirotin merged commit e477269 into develop Nov 29, 2024
17 of 20 checks passed
@igor-sirotin igor-sirotin deleted the feat/InitializeApplication-init-logs branch November 29, 2024 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants