Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Expectations on CPU usage and number of processes? #40

Open
qacollective opened this issue Oct 4, 2020 · 1 comment
Open

Expectations on CPU usage and number of processes? #40

qacollective opened this issue Oct 4, 2020 · 1 comment

Comments

@qacollective
Copy link

Hello,

Just writing in to check my expectations on what the timescaledb-parallel-copy script should look like while running.

I am running it inside a docker container but that container doesn't have any restrictions on memory or CPU usage. I have matched the number of workers to the number of CPU cores. I am only however seeing minimal (10-20%) CPU usage, and cannot see more than one process running in the 'top' output.

Is this to be expected?

This is the command I'm running from within a Python script:

/bin/bash -c 'timescaledb-parallel-copy --connection "postgres://postgres:****@localhost:5432/trade" --schema XXX --table YYY --file file.csv --workers 8 --reporting-period 30s --skip-header'

This is typical top output:

top - 11:46:02 up 4 days, 12:10,  0 users,  load average: 4.13, 4.58, 4.86
Tasks:   5 total,   1 running,   4 sleeping,   0 stopped,   0 zombie
%Cpu(s): 22.8 us, 12.3 sy,  0.2 ni, 59.2 id,  4.5 wa,  0.0 hi,  1.0 si,  0.0 st
KiB Mem : 32824804 total,   946600 free, 12929264 used, 18948940 buff/cache
KiB Swap:  2097148 total,        0 free,  2097148 used. 10609320 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                     
      7 winer     20   0 1373344  27052   6212 S  10.6  0.1   0:04.85 timescaledb-par                                                                                                                             
      1 winer     20   0   96640  20192  12084 S   0.0  0.1   0:00.09 python3                                                                                                                                     
      6 winer     20   0    4624    784    716 S   0.0  0.0   0:00.00 sh                                                                                                                                          
     23 winer     20   0   18504   3344   2968 S   0.0  0.0   0:00.02 bash                                                                                                                                        
     37 winer     20   0   38724   3156   2664 R   0.0  0.0   0:00.00 top

This is some sample program output:

2020-10-04 11:52:25,278 - TickDataLoader - INFO - at 30s, row rate 79833.11/sec (period), row rate 79833.11/sec (overall), 2.395000E+06 total rows
2020-10-04 11:52:55,278 - TickDataLoader - INFO - at 1m0s, row rate 76833.29/sec (period), row rate 78333.20/sec (overall), 4.700000E+06 total rows
2020-10-04 11:53:25,278 - TickDataLoader - INFO - at 1m30s, row rate 85500.03/sec (period), row rate 80722.14/sec (overall), 7.265000E+06 total rows
2020-10-04 11:53:55,278 - TickDataLoader - INFO - at 2m0s, row rate 93999.98/sec (period), row rate 84041.60/sec (overall), 1.008500E+07 total rows
2020-10-04 11:54:25,278 - TickDataLoader - INFO - at 2m30s, row rate 94333.36/sec (period), row rate 86099.95/sec (overall), 1.291500E+07 total rows
2020-10-04 11:54:55,279 - TickDataLoader - INFO - at 3m0s, row rate 101999.91/sec (period), row rate 88749.94/sec (overall), 1.597500E+07 total rows
2020-10-04 11:55:25,286 - TickDataLoader - INFO - at 3m30s, row rate 104166.73/sec (period), row rate 90952.34/sec (overall), 1.910000E+07 total rows
2020-10-04 11:55:55,278 - TickDataLoader - INFO - at 4m0s, row rate 99333.38/sec (period), row rate 91999.97/sec (overall), 2.208000E+07 total rows
2020-10-04 11:56:25,278 - TickDataLoader - INFO - at 4m30s, row rate 99833.33/sec (period), row rate 92870.34/sec (overall), 2.507500E+07 total rows
2020-10-04 11:56:55,278 - TickDataLoader - INFO - at 5m0s, row rate 93333.32/sec (period), row rate 92916.64/sec (overall), 2.787500E+07 total rows
2020-10-04 11:57:25,278 - TickDataLoader - INFO - at 5m30s, row rate 102166.62/sec (period), row rate 93757.55/sec (overall), 3.094000E+07 total rows
2020-10-04 11:57:55,278 - TickDataLoader - INFO - at 6m0s, row rate 102666.66/sec (period), row rate 94499.97/sec (overall), 3.402000E+07 total rows
2020-10-04 11:58:25,278 - TickDataLoader - INFO - at 6m30s, row rate 106166.71/sec (period), row rate 95397.41/sec (overall), 3.720500E+07 total rows
2020-10-04 11:58:55,278 - TickDataLoader - INFO - at 7m0s, row rate 107833.29/sec (period), row rate 96285.69/sec (overall), 4.044000E+07 total rows
2020-10-04 11:59:25,278 - TickDataLoader - INFO - at 7m30s, row rate 105666.75/sec (period), row rate 96911.09/sec (overall), 4.361000E+07 total rows
2020-10-04 11:59:55,278 - TickDataLoader - INFO - at 8m0s, row rate 104499.99/sec (period), row rate 97385.40/sec (overall), 4.674500E+07 total rows
2020-10-04 12:00:25,278 - TickDataLoader - INFO - at 8m30s, row rate 105499.99/sec (period), row rate 97862.73/sec (overall), 4.991000E+07 total rows
2020-10-04 12:00:55,278 - TickDataLoader - INFO - at 9m0s, row rate 103500.00/sec (period), row rate 98175.91/sec (overall), 5.301500E+07 total rows
2020-10-04 12:01:25,278 - TickDataLoader - INFO - at 9m30s, row rate 102666.50/sec (period), row rate 98412.26/sec (overall), 5.609500E+07 total rows
2020-10-04 12:01:55,278 - TickDataLoader - INFO - at 10m0s, row rate 107500.01/sec (period), row rate 98866.65/sec (overall), 5.932000E+07 total rows
2020-10-04 12:02:25,278 - TickDataLoader - INFO - at 10m30s, row rate 106500.19/sec (period), row rate 99230.15/sec (overall), 6.251500E+07 total rows
2020-10-04 12:02:55,278 - TickDataLoader - INFO - at 11m0s, row rate 108333.32/sec (period), row rate 99643.93/sec (overall), 6.576500E+07 total rows
2020-10-04 12:03:25,278 - TickDataLoader - INFO - at 11m30s, row rate 106666.69/sec (period), row rate 99949.26/sec (overall), 6.896500E+07 total rows
2020-10-04 12:03:55,278 - TickDataLoader - INFO - at 12m0s, row rate 108166.62/sec (period), row rate 100291.65/sec (overall), 7.221000E+07 total rows
2020-10-04 12:04:25,278 - TickDataLoader - INFO - at 12m30s, row rate 106666.62/sec (period), row rate 100546.65/sec (overall), 7.541000E+07 total rows
2020-10-04 12:04:55,278 - TickDataLoader - INFO - at 13m0s, row rate 108666.68/sec (period), row rate 100858.96/sec (overall), 7.867000E+07 total rows
2020-10-04 12:05:25,278 - TickDataLoader - INFO - at 13m30s, row rate 111000.01/sec (period), row rate 101234.56/sec (overall), 8.200000E+07 total rows
2020-10-04 12:05:55,279 - TickDataLoader - INFO - at 14m0s, row rate 106994.01/sec (period), row rate 101440.26/sec (overall), 8.521000E+07 total rows
2020-10-04 12:06:25,278 - TickDataLoader - INFO - at 14m30s, row rate 109339.51/sec (period), row rate 101712.63/sec (overall), 8.849000E+07 total rows
2020-10-04 12:06:55,278 - TickDataLoader - INFO - at 15m0s, row rate 114666.39/sec (period), row rate 102144.43/sec (overall), 9.193000E+07 total rows
2020-10-04 12:07:25,278 - TickDataLoader - INFO - at 15m30s, row rate 112500.16/sec (period), row rate 102478.48/sec (overall), 9.530500E+07 total rows
@yazeed
Copy link

yazeed commented Dec 12, 2020 •
edited
Loading

Looking for the same information here as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yazeed @qacollective