Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix rsyslog permission error in github ubuntu tests from apparmor #15717

Open
wants to merge 5 commits into
base: devel
Choose a base branch
from

Conversation

AlanCoding
Copy link
Member

SUMMARY

We see an issue in the container logs, and this test is meant to make it a failure. Something about the image build and the ubuntu runners, probably.

ISSUE TYPE
  • Bug, Docs Fix or other nominal change
COMPONENT NAME
  • API

@AlanCoding
Copy link
Member Author

I added a step that runs sudo dmesg as suggested in #14259 and this confirmed we get repeated messages:

[  317.988954] audit: type=1400 audit(1734577552.024:221): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/var/lib/awx/rsyslog/rsyslog.conf" pid=8074 comm="rsyslogd" requested_mask="r" denied_mask="r" fsuid=1001 ouid=1001
[  317.997785] audit: type=1400 audit(1734577552.033:222): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/var/lib/awx/rsyslog/rsyslog.conf" pid=8075 comm="rsyslogd" requested_mask="r" denied_mask="r" fsuid=1001 ouid=1001

So that suggests that their solution will also work.

@AlanCoding
Copy link
Member Author

Here we go!

  • Download size of docker-compose-logs.zip 187 kB
  • Download size before this patch 1,139 kB

Extracted sizes are 1.9 MB vs 39.5 MB, even more dramatic. Now the logs look like this:

[pid: 550|app: 0|req: 1/1] 172.18.0.1 () {36 vars in 378 bytes} [Thu Dec 19 03:25:23 2024] GET /api/v2/ping/ => generated 440 bytes in 26 msecs (HTTP/1.1 200) 13 headers in 399 bytes (1 switches on core 0)
�[36;1m2024-12-19 03:25:38,133 INFO     [-] awx.analytics.job_lifecycle projectupdate-1 controller node chosen {"type": "projectupdate", "task_id": 1, "state": "controller_node_chosen", "work_unit_id": null, "task_name": "project-with-requirements", "controller_node": "awx-1"}�[0m
�[36;1m2024-12-19 03:25:38,133 INFO     [-] awx.analytics.job_lifecycle projectupdate-1 execution node chosen {"type": "projectupdate", "task_id": 1, "state": "execution_node_chosen", "work_unit_id": null, "task_name": "project-with-requirements", "execution_node": "awx-1"}�[0m
�[36;1m2024-12-19 03:25:38,144 INFO     [-] awx.analytics.job_lifecycle projectupdate-1 waiting {"type": "projectupdate", "task_id": 1, "state": "waiting", "work_unit_id": null, "task_name": "project-with-requirements"}�[0m
�[36;1m2024-12-19 03:25:38,216 INFO     [-] awx.analytics.job_lifecycle projectupdate-1 pre run {"type": "projectupdate", "task_id": 1, "state": "pre_run", "work_unit_id": null, "task_name": "project-with-requirements"}�[0m
�[36;1m2024-12-19 03:25:38,230 INFO     [-] awx.analytics.job_lifecycle projectupdate-1 preparing playbook {"type": "projectupdate", "task_id": 1, "state": "preparing_playbook", "work_unit_id": null, "task_name": "project-with-requirements"}�[0m
�[36;1m2024-12-19 03:25:38,247 INFO     [-] awx.analytics.job_lifecycle projectupdate-1 running playbook {"type": "projectupdate", "task_id": 1, "state": "running_playbook", "work_unit_id": null, "task_name": "project-with-requirements"}�[0m
�[36;1m2024-12-19 03:25:38,254 INFO     [-] awx.analytics.job_lifecycle projectupdate-1 work unit id received {"type": "projectupdate", "task_id": 1, "state": "work_unit_id_received", "work_unit_id": "awx1KwLhRjC5", "task_name": "project-with-requirements"}�[0m
�[36;1m2024-12-19 03:25:38,264 INFO     [-] awx.analytics.job_lifecycle projectupdate-1 work unit id assigned {"type": "projectupdate", "task_id": 1, "state": "work_unit_id_assigned", "work_unit_id": "awx1KwLhRjC5", "task_name": "project-with-requirements"}�[0m
WARNING 2024/12/19 03:26:12 Could not read in control service: read unix /var/run/awx-receptor/receptor.sock->@: use of closed network connection
WARNING 2024/12/19 03:26:12 Could not close connection: close unix /var/run/awx-receptor/receptor.sock->@: use of closed network connection
�[1m2024-12-19 03:26:12,104 INFO     [-] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 1�[0m

This is so must more relevant, and is very task-oriented.

Next thing to do might be to remove ansi codes with the -t option, but seems to be fairly hard to get that in there.

@AlanCoding AlanCoding marked this pull request as ready for review December 19, 2024 03:59
@AlanCoding AlanCoding changed the title Add test to detect rsyslog config problems Fix rsyslog permission error in github ubuntu tests from apparmor Dec 19, 2024
Copy link

sonarqubecloud bot commented Jan 6, 2025

@AlanCoding
Copy link
Member Author

Updated logs:

2025-01-06 19:41:22,143 INFO reaped unknown pid 831 (exit status 0)
2025-01-06 19:41:22,143 INFO reaped unknown pid 836 (exit status 0)
2025-01-06 19:41:22,143 INFO reaped unknown pid 833 (exit status 0)
WARNING 2025/01/06 19:41:22 Could not read in control service: read unix /var/run/awx-receptor/receptor.sock->@: use of closed network connection
WARNING 2025/01/06 19:41:22 Could not close connection: close unix /var/run/awx-receptor/receptor.sock->@: use of closed network connection
2025-01-06 19:41:22,199 INFO     [-] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 1
2025-01-06 19:41:22,210 INFO     [-] awx.analytics.job_lifecycle projectupdate-1 post run {"type": "projectupdate", "task_id": 1, "state": "post_run", "work_unit_id": "awx1I2WzSla3", "task_name": "project-with-requirements"}
2025-01-06 19:41:22,271 INFO     [-] awx.analytics.job_lifecycle projectupdate-1 finalize run {"type": "projectupdate", "task_id": 1, "state": "finalize_run", "work_unit_id": "awx1I2WzSla3", "task_name": "project-with-requirements"}
2025-01-06 19:41:23,047 INFO     [-] awx.analytics.job_lifecycle projectupdate-1 stats wrapup finished {"type": "projectupdate", "task_id": 1, "state": "stats_wrapup_finished", "work_unit_id": "awx1I2WzSla3", "task_name": "project-with-requirements"}
2025-01-06 19:41:26,951 WARNING  [-] awx.main.tasks.system Heartbeat skew - interval=62.7201, expected=60
2025-01-06 19:41:29,967 INFO     [-] awx.analytics.job_lifecycle systemjob-2 controller node chosen {"type": "systemjob", "task_id": 2, "state": "controller_node_chosen", "work_unit_id": null, "task_name": "Cleanup Job Details", "controller_node": "awx-1"}
2025-01-06 19:41:29,967 INFO     [-] awx.analytics.job_lifecycle systemjob-2 execution node chosen {"type": "systemjob", "task_id": 2, "state": "execution_node_chosen", "work_unit_id": null, "task_name": "Cleanup Job Details", "execution_node": "awx-1"}
2025-01-06 19:41:29,972 INFO     [-] awx.analytics.job_lifecycle systemjob-2 waiting {"type": "systemjob", "task_id": 2, "state": "waiting", "work_unit_id": null, "task_name": "Cleanup Job Details"}
2025-01-06 19:41:30,028 INFO     [-] awx.analytics.job_lifecycle systemjob-2 pre run {"type": "systemjob", "task_id": 2, "state": "pre_run", "work_unit_id": null, "task_name": "Cleanup Job Details"}
2025-01-06 19:41:30,040 INFO     [-] awx.analytics.job_lifecycle systemjob-2 preparing playbook {"type": "systemjob", "task_id": 2, "state": "preparing_playbook", "work_unit_id": null, "task_name": "Cleanup Job Details"}
2025-01-06 19:41:30,046 INFO     [-] awx.analytics.job_lifecycle systemjob-2 running playbook {"type": "systemjob", "task_id": 2, "state": "running_playbook", "work_unit_id": null, "task_name": "Cleanup Job Details"}
2025-01-06 19:41:32,282 INFO     [-] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 2
2025-01-06 19:41:32,284 INFO     [-] awx.analytics.job_lifecycle systemjob-2 post run {"type": "systemjob", "task_id": 2, "state": "post_run", "work_unit_id": null, "task_name": "Cleanup Job Details"}
2025-01-06 19:41:32,311 INFO     [-] awx.analytics.job_lifecycle systemjob-2 stats wrapup finished {"type": "systemjob", "task_id": 2, "state": "stats_wrapup_finished", "work_unit_id": null, "task_name": "Cleanup Job Details"}
2025-01-06 19:41:32,328 IN

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant