Sy/slurm integration #18893

steveny91 · 2024-10-22T22:23:52Z

What does this PR do?

Just the base implementation. All assets are missing. Will be added in separate PRs.

Python implementation
Config Template
Unit tests

For docs:

Spec.yaml
conf.yaml.example
Readme
Manifest description

github-actions · 2024-10-22T22:25:03Z

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

codecov · 2024-10-22T22:26:31Z

Codecov Report

Attention: Patch coverage is 87.39726% with 46 lines in your changes missing coverage. Please review.

Project coverage is 88.70%. Comparing base (575fc4f) to head (fbc43c9).
Report is 7 commits behind head on master.

Additional details and impacted files

Flag	Coverage Δ
activemq	`?`
cassandra	`?`
hive	`?`
hivemq	`?`
ignite	`?`
jboss_wildfly	`?`
kafka	`?`
presto	`?`
slurm	`87.39% <87.39%> (?)`
solr	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

github-actions · 2024-10-23T13:33:33Z

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

github-actions · 2024-10-25T06:03:39Z

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

github-actions · 2024-10-29T03:38:08Z

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

steveny91 · 2024-10-29T03:38:41Z

@hestonhoffman

I added the readme, and the metadata.csv. Would it be possible to review those 2 assets and also the spec.yaml/conf.data.example template?

The other stuff I plan to add in another PR such as Dashboard etc.

github-actions · 2024-10-29T03:49:52Z

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

github-actions · 2024-10-29T14:31:59Z

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

github-actions · 2024-10-30T13:20:32Z

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

hestonhoffman · 2024-10-30T20:02:19Z

@hestonhoffman

I added the readme, and the metadata.csv. Would it be possible to review those 2 assets and also the spec.yaml/conf.data.example template?

The other stuff I plan to add in another PR such as Dashboard etc.

Sure, I added @DataDog/documentation back, cos I don't have time to take a look today. Someone on the docs team will pick it up

drichards-87 · 2024-10-30T20:51:04Z

Created Jira card for Docs Team editorial review.

github-actions · 2024-10-31T14:08:40Z

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

github-actions · 2024-10-31T14:23:23Z

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

urseberry

Nicely written! I have some feedback, mostly style notes.

urseberry · 2024-11-01T20:56:46Z

slurm/README.md

[Applies throughout] The Slurm documentation (https://slurm.schedmd.com/documentation.html) formats the product name as "Slurm", not "SLURM" as this PR does. Which is correct?

urseberry · 2024-11-01T21:00:18Z

slurm/README.md

+
+SLURM (Simple Linux Utility for Resource Management) is an open-source workload manager used to schedule and manage jobs on large-scale compute clusters. It allocates resources, monitors job queues, and ensures efficient execution of parallel and batch jobs in high-performance computing environments.
+
+The check gathers metrics from slurmctld by executing and parsing the output of several command-line binaries, including [sinfo][8], [squeue][9], [sacct][10], [sdiag][11], and [sshare][12]. These commands provide detailed information on resource availability, job queues, accounting, diagnostics, and share usage in a SLURM-managed cluster.


Suggested change

The check gathers metrics from slurmctld by executing and parsing the output of several command-line binaries, including [sinfo][8], [squeue][9], [sacct][10], [sdiag][11], and [sshare][12]. These commands provide detailed information on resource availability, job queues, accounting, diagnostics, and share usage in a SLURM-managed cluster.

The check gathers metrics from `slurmctld` by executing and parsing the output of several command-line binaries, including [`sinfo`][8], [`squeue`][9], [`sacct`][10], [`sdiag`][11], and [`sshare`][12]. These commands provide detailed information on resource availability, job queues, accounting, diagnostics, and share usage in a SLURM-managed cluster.

urseberry · 2024-11-01T21:00:39Z

slurm/README.md

+
+## Setup
+
+Follow the instructions below to install and configure this check for an Agent running on a host. Since the Agent requires direct access to the various SLURM binaries, monitoring SLURM in containerized environments is not yet recommended.


Suggested change

Follow the instructions below to install and configure this check for an Agent running on a host. Since the Agent requires direct access to the various SLURM binaries, monitoring SLURM in containerized environments is not yet recommended.

Follow the instructions below to install and configure this check for an Agent running on a host. Since the Agent requires direct access to the various SLURM binaries, monitoring SLURM in containerized environments is not recommended.

urseberry · 2024-11-01T21:01:39Z

slurm/README.md

+
+1. Ensure that the dd-agent user has execute permissions on the relevant command binaries and the necessary permissions to access the directories where these binaries are located.
+
+2. Edit the `slurm.d/conf.yaml` file, in the `conf.d/` folder at the root of your Agent's configuration directory to start collecting your slurm data. See the [sample slurm.d/conf.yaml][3] for all available configuration options.


Suggested change

2. Edit the `slurm.d/conf.yaml` file, in the `conf.d/` folder at the root of your Agent's configuration directory to start collecting your slurm data. See the [sample slurm.d/conf.yaml][3] for all available configuration options.

2. Edit the `slurm.d/conf.yaml` file, in the `conf.d/` folder at the root of your Agent's configuration directory to start collecting your Slurm data. See the [sample slurm.d/conf.yaml][3] for all available configuration options.

urseberry · 2024-11-01T21:01:59Z

slurm/README.md

+```yaml
+init_config:
+
+    ## Feel free to customize this part incase the binaries are not located in the /usr/bin/ directory


Suggested change

## Feel free to customize this part incase the binaries are not located in the /usr/bin/ directory

## Customize this part if the binaries are not located in the /usr/bin/ directory

urseberry · 2024-11-01T21:09:03Z

slurm/assets/configuration/spec.yaml

+        Whether or not to enable debug logging for the sacct command. This will log the output of the sacct command
+        to the agent log.


Suggested change

Whether or not to enable debug logging for the sacct command. This will log the output of the sacct command

to the agent log.

Whether or not to enable debug logging for the sacct command. This logs the output of the sacct command

to the Agent log.

urseberry · 2024-11-01T21:10:00Z

slurm/assets/configuration/spec.yaml

+          This changes the collection interval of the check. For more information, see:
+          https://docs.datadoghq.com/developers/write_agent_check/#collection-interval
+
+          Most Slurm metrics are collected from calling the different binaries. Depending on the size of the slurm cluster,


Suggested change

Most Slurm metrics are collected from calling the different binaries. Depending on the size of the slurm cluster,

Most Slurm metrics are collected from calling the different binaries. Depending on the size of the Slurm cluster,

urseberry · 2024-11-01T21:10:38Z

slurm/datadog_checks/slurm/data/conf.yaml.example

+    ## collects data from from individual nodes as well but will be more verbose and include data such as CPU and 
+    ## memory usage as reported from the OS as well as additional tags.


Suggested change

## collects data from from individual nodes as well but will be more verbose and include data such as CPU and

## memory usage as reported from the OS as well as additional tags.

## collects data from from individual nodes as well but is more verbose and includes data such as CPU and

## memory usage as reported from the OS, as well as additional tags.

urseberry · 2024-11-01T21:11:01Z

slurm/datadog_checks/slurm/data/conf.yaml.example

+    ## @param metric_patterns - mapping - optional
+    ## A mapping of metrics to include or exclude, with each entry being a regular expression.
+    ##
+    ## Metrics defined in `exclude` will take precedence in case of overlap.


Suggested change

## Metrics defined in `exclude` will take precedence in case of overlap.

## Metrics defined in `exclude` take precedence in case of overlap.

urseberry · 2024-11-01T21:11:24Z

slurm/images/IMAGES_README.md

Please remove this file.

steveny91 added 11 commits October 22, 2024 18:23

initial slurm commit

3905db3

WIP

1dd907e

test fixtures

5b5f8a7

wip edits

d3c4235

refactor

86ddec9

add timestamping for sacct

cb33eb0

more options

d85f363

slight changes

eb9ab87

first test

4bd1e7d

test

e3c656d

no assets

f4b2ff7

datadog-assets bot added agent/review-requested docs/review-requested ecosystems/review-requested product/review-requested labels Oct 22, 2024

datadog-agent-integrations-bot bot added documentation dev/testing dev/tooling release qa/skip-qa Automatically skip this PR for the next QA labels Oct 22, 2024

steveny91 added 3 commits October 22, 2024 22:19

remove breakpoints

4899d83

error handling

1512a95

lint

96f2278

steveny91 added 2 commits October 23, 2024 10:15

lint

b87b2ae

more test

d98c9a3

styling

94c4e06

docs and metadata

4203364

manifest

85f41bd

add statelong

f33ab0b

Merge branch 'master' into sy/slurm-integration

12350ac

steveny91 requested a review from hestonhoffman October 30, 2024 18:07

Kyle-Neale previously approved these changes Oct 30, 2024

View reviewed changes

datadog-assets bot added agent/approved and removed agent/review-requested labels Oct 30, 2024

hestonhoffman requested review from a team and removed request for hestonhoffman October 30, 2024 20:01

drichards-87 added the editorial review Waiting on a more in-depth review from a docs team editor label Oct 30, 2024

validations

fb7418b

steveny91 dismissed Kyle-Neale’s stale review via fb7418b October 31, 2024 14:07

datadog-assets bot added agent/review-requested and removed agent/approved labels Oct 31, 2024

steveny91 added 2 commits October 31, 2024 10:17

license headers

50329c6

license headers

55ee608

labeler

4d4946e

datadog-agent-integrations-bot bot added the integration/slurm label Oct 31, 2024

Merge branch 'master' into sy/slurm-integration

fbc43c9

urseberry reviewed Nov 1, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sy/slurm integration #18893

Sy/slurm integration #18893

steveny91 commented Oct 22, 2024 •

edited

Loading

github-actions bot commented Oct 22, 2024

codecov bot commented Oct 22, 2024 •

edited

Loading

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 25, 2024

github-actions bot commented Oct 29, 2024

steveny91 commented Oct 29, 2024

github-actions bot commented Oct 29, 2024

github-actions bot commented Oct 29, 2024

github-actions bot commented Oct 30, 2024

hestonhoffman commented Oct 30, 2024

drichards-87 commented Oct 30, 2024

github-actions bot commented Oct 31, 2024

github-actions bot commented Oct 31, 2024

urseberry left a comment

urseberry Nov 1, 2024

urseberry Nov 1, 2024

urseberry Nov 1, 2024

urseberry Nov 1, 2024

urseberry Nov 1, 2024

urseberry Nov 1, 2024

urseberry Nov 1, 2024

urseberry Nov 1, 2024

urseberry Nov 1, 2024

urseberry Nov 1, 2024


		SLURM (Simple Linux Utility for Resource Management) is an open-source workload manager used to schedule and manage jobs on large-scale compute clusters. It allocates resources, monitors job queues, and ensures efficient execution of parallel and batch jobs in high-performance computing environments.

		The check gathers metrics from slurmctld by executing and parsing the output of several command-line binaries, including [sinfo][8], [squeue][9], [sacct][10], [sdiag][11], and [sshare][12]. These commands provide detailed information on resource availability, job queues, accounting, diagnostics, and share usage in a SLURM-managed cluster.

	The check gathers metrics from slurmctld by executing and parsing the output of several command-line binaries, including [sinfo][8], [squeue][9], [sacct][10], [sdiag][11], and [sshare][12]. These commands provide detailed information on resource availability, job queues, accounting, diagnostics, and share usage in a SLURM-managed cluster.
	The check gathers metrics from `slurmctld` by executing and parsing the output of several command-line binaries, including [`sinfo`][8], [`squeue`][9], [`sacct`][10], [`sdiag`][11], and [`sshare`][12]. These commands provide detailed information on resource availability, job queues, accounting, diagnostics, and share usage in a SLURM-managed cluster.


		## Setup

		Follow the instructions below to install and configure this check for an Agent running on a host. Since the Agent requires direct access to the various SLURM binaries, monitoring SLURM in containerized environments is not yet recommended.


		1. Ensure that the dd-agent user has execute permissions on the relevant command binaries and the necessary permissions to access the directories where these binaries are located.

		2. Edit the `slurm.d/conf.yaml` file, in the `conf.d/` folder at the root of your Agent's configuration directory to start collecting your slurm data. See the [sample slurm.d/conf.yaml][3] for all available configuration options.

	## Feel free to customize this part incase the binaries are not located in the /usr/bin/ directory
	## Customize this part if the binaries are not located in the /usr/bin/ directory

		Whether or not to enable debug logging for the sacct command. This will log the output of the sacct command
		to the agent log.

	Most Slurm metrics are collected from calling the different binaries. Depending on the size of the slurm cluster,
	Most Slurm metrics are collected from calling the different binaries. Depending on the size of the Slurm cluster,

		## collects data from from individual nodes as well but will be more verbose and include data such as CPU and
		## memory usage as reported from the OS as well as additional tags.

	## Metrics defined in `exclude` will take precedence in case of overlap.
	## Metrics defined in `exclude` take precedence in case of overlap.

Sy/slurm integration #18893

Are you sure you want to change the base?

Sy/slurm integration #18893

Conversation

steveny91 commented Oct 22, 2024 • edited Loading

What does this PR do?

github-actions bot commented Oct 22, 2024

codecov bot commented Oct 22, 2024 • edited Loading

Codecov Report

github-actions bot commented Oct 23, 2024

github-actions bot commented Oct 25, 2024

github-actions bot commented Oct 29, 2024

steveny91 commented Oct 29, 2024

github-actions bot commented Oct 29, 2024

github-actions bot commented Oct 29, 2024

github-actions bot commented Oct 30, 2024

hestonhoffman commented Oct 30, 2024

drichards-87 commented Oct 30, 2024

github-actions bot commented Oct 31, 2024

github-actions bot commented Oct 31, 2024

urseberry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steveny91 commented Oct 22, 2024 •

edited

Loading

codecov bot commented Oct 22, 2024 •

edited

Loading