Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

profile (libbpf): tool enhacements #5181

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
+1,041 −53
Open

profile (libbpf): tool enhacements #5181

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9144bb9
trace_helpers.c: add PID field to struct DSO
glima Nov 6, 2024
7d1c457
profile: new option -R (refresh DSOs), to capture short lived processes
glima Nov 6, 2024
613c87f
profile: new mode -A (addresses only), where no name resolution is at…
glima Nov 6, 2024
7d2487b
profile: add -P (pressure stall information loop) mode for the tool
glima Nov 1, 2024
de372e9
profile: add -o (output-path) mode, an addition to the original stdou…
glima Nov 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 104 additions & 4 deletions libbpf-tools/profile.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,8 @@ static struct env {
pid_t tids[MAX_TID_NR];
bool user_stacks_only;
bool kernel_stacks_only;
bool addrs_only;
bool refresh_dsos;
int stack_storage_size;
int perf_max_stack_depth;
int duration;
Expand Down Expand Up @@ -105,6 +107,8 @@ const char argp_program_doc[] =
" profile -p 185 # only profile process with PID 185\n"
" profile -L 185 # only profile thread with TID 185\n"
" profile -U # only show user space stacks (no kernel)\n"
" profile -A # only output addresses\n"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this not be achieved using the BPF_F_USER_BUILD_ID flag of the bpf_get_stackid helper function?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, thanks @ekyooo, let me take a look at that before responding.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, finally time for this. Not sure I follow, though, I'm afraid. bpf_get_stackid will only honor the following flags, it seems:

  • BPF_F_SKIP_FIELD_MASK
  • BPF_F_USER_STACK
  • BPF_F_FAST_STACK_CMP
  • BPF_F_REUSE_STACKID.

What did I miss? Are you talking about employing a different set of bpf helpers at the BPF program level, should that mode be in place, that would get to the same effect? But the parent profile.c wants to walk symbol resolution, through syms__map_addr(), regardless, doesn't it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, it's not bpf_get_stackid but bpf_get_stack. Sorry for the confusion. Please refer to:
https://youtu.be/20SO5thkvhI?list=PLbzoR-pLrL6oj1rVTXLnV7cOuetvjKn9q&t=145

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see what you mean. Yeah, even for my uses, I intend to downstream-fork the symbol resolution phase to adapt to company-only flows of symbol resolution.

BUILD ID could indeed be one of the things serialized there. However, can that be an addition to this? This mode is already useful for some as-is, right? Not everybody will have BUILD-ID annotaded binaries for their systems, to begin with.

I will revisit and think about BUILD-ID addition when I refine it internally (making something more generic to add here). Is that sound? Or do you want me to focus on something I missed?

Thanks.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ekyooo happy to accomodate, though, if you work with me deeper in your idea :) If my take is beneficial, gentle ping.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is no BUILD-ID, you can perform offline symbol resolving using the module name and module base offset that are outputted by the -v option.

However, since this information can vary with each build version, I think that using the build-id method could be more practical and has lower maintenance costs.

This is my opinion as a contributor, not a reviewer.

Copy link
Author

@glima glima Jan 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure where you are going here now, TBH. -v will only give you extra path and an offset

e.g. 0x00007fae6af9068b (/usr/lib64/firefox/libxul.so+0x479068b)

But the full context of start, end, file_offset etc is needed to actualy find the symbols (see my print_dso_info(), which serializes that), for this use case. I just replicated the same c logic in python, for the offline resolver.

Build ID matching or not, more context than -v mode is needed.

" profile -R # refresh DSO list during profiling execution\n"
" profile -K # only show kernel space stacks (no user)\n";

static const struct argp_option opts[] = {
Expand All @@ -114,6 +118,11 @@ static const struct argp_option opts[] = {
"show stacks from user space only (no kernel space stacks)", 0 },
{ "kernel-stacks-only", 'K', NULL, 0,
"show stacks from kernel space only (no user space stacks)", 0 },
{ "addrs-only", 'A', NULL, 0,
"output addresses only on final charts (no symbol resolution), also appending extra DSO info section", 0 },
{ "refresh-dsos", 'R', NULL, 0,
"refresh DSO list during profiling execution, not just at its end "
"(to catch short-lived processes)", 0 },
{ "frequency", 'F', "FREQUENCY", 0, "sample frequency, Hertz", 0 },
{ "delimited", 'd', NULL, 0, "insert delimiter between kernel/user stacks", 0 },
{ "include-idle ", 'I', NULL, 0, "include CPU idle stacks", 0 },
Expand All @@ -132,6 +141,7 @@ struct ksyms *ksyms;
struct syms_cache *syms_cache;
struct syms *syms;
static char syminfo[SYM_INFO_LEN];
volatile bool keep_running = true;

static error_t parse_arg(int key, char *arg, struct argp_state *state)
{
Expand Down Expand Up @@ -177,6 +187,12 @@ static error_t parse_arg(int key, char *arg, struct argp_state *state)
case 'K':
env.kernel_stacks_only = true;
break;
case 'A':
env.addrs_only = true;
break;
case 'R':
env.refresh_dsos = true;
break;
case 'F':
errno = 0;
env.sample_freq = strtol(arg, NULL, 10);
Expand Down Expand Up @@ -288,6 +304,7 @@ static int libbpf_print_fn(enum libbpf_print_level level, const char *format, va

static void sig_handler(int sig)
{
keep_running = false;
}

static int cmp_counts(const void *a, const void *b)
Expand Down Expand Up @@ -326,6 +343,13 @@ static int read_counts_map(int fd, struct key_ext_t *items, __u32 *count)

static const char *ksymname(unsigned long addr)
{
static char addr_str[32];

if (env.addrs_only) {
snprintf(addr_str, sizeof(addr_str), "[0x%lx]", addr);
return addr_str;
}

const struct ksym *ksym = ksyms__map_addr(ksyms, addr);

if (!env.verbose)
Expand Down Expand Up @@ -365,6 +389,12 @@ static const char *usyminfo(unsigned long addr)
static const char *usymname(unsigned long addr)
{
const struct sym *sym;
static char addr_str[32];

if (env.addrs_only) {
snprintf(addr_str, sizeof(addr_str), "[0x%lx]", addr);
return addr_str;
}

if (!env.verbose) {
sym = syms__map_addr(syms, addr);
Expand Down Expand Up @@ -462,6 +492,58 @@ static int print_count(struct key_t *event, __u64 count, int stack_map, bool fol
return 0;
}

static void bump_dso(struct key_t *event, int stack_map, unsigned long *ip)
{
if (bpf_map_lookup_elem(stack_map, &event->user_stack_id, ip) != 0) {
return;
} else {
syms_cache__get_syms(syms_cache, event->pid);
}
}

static int refresh_dso(struct key_t *event, int stack_map)
{
unsigned long *ip;

ip = calloc(env.perf_max_stack_depth, sizeof(unsigned long));
if (!ip) {
fprintf(stderr, "failed to alloc ip\n");
return -ENOMEM;
}

bump_dso(event, stack_map, ip);

free(ip);

return 0;
}

static int refresh_dsos(int counts_map, int stack_map)
{
struct key_ext_t *counts;
__u32 nr_count = MAX_ENTRIES;
int i, ret = 0;

counts = calloc(MAX_ENTRIES, sizeof(struct key_ext_t));
if (!counts) {
fprintf(stderr, "Out of memory\n");
return -ENOMEM;
}

ret = read_counts_map(counts_map, counts, &nr_count);
if (ret)
goto cleanup;

for (i = 0; i < nr_count; i++) {
refresh_dso(&counts[i].k, stack_map);
}

cleanup:
free(counts);

return ret;
}

static int print_counts(int counts_map, int stack_map)
{
struct key_ext_t *counts;
Expand Down Expand Up @@ -495,6 +577,15 @@ static int print_counts(int counts_map, int stack_map)
has_collision = CHECK_STACK_COLLISION(event->user_stack_id, event->kern_stack_id);
}


if (env.addrs_only) {
//separator into next section
printf("==========\n");
print_dsos_info(syms_cache);
//separator again (to accomodate kernel symbols)
printf("==========\n");
}

if (nr_missing_stacks > 0) {
fprintf(stderr, "WARNING: %zu stack traces could not be displayed.%s\n",
nr_missing_stacks, has_collision ?
Expand Down Expand Up @@ -659,15 +750,24 @@ int main(int argc, char **argv)
goto cleanup;

signal(SIGINT, sig_handler);

if (!env.folded)
print_headers();
signal(SIGALRM, sig_handler);

/*
* We'll get sleep interrupted when someone presses Ctrl-C.
* (which will be "handled" with noop by sig_handler)
*/
sleep(env.duration);
if (env.refresh_dsos) {
alarm(env.duration);
while (keep_running) {
refresh_dsos(bpf_map__fd(obj->maps.counts),
bpf_map__fd(obj->maps.stackmap));
usleep(100000);
}
} else
sleep(env.duration);

if (!env.folded)
print_headers();

print_counts(bpf_map__fd(obj->maps.counts),
bpf_map__fd(obj->maps.stackmap));
Expand Down
Loading