Clean up Procedure

In some, hopefully rare, cases, the DAQ system can have problems both during or at the end of a run. In this case one can stop the main server with the shutdown command (or with CTRL-C if the server is stuck) and then execute the following procedures to manually stop all DAQ processes and clean all DAQ control directories.

Level1/Merger node clean-up

This procedure must be executed on nodes l1padme3 (showed as l1padme1, once connected) and l1padme4 (showed as l1padme2, once connected). Here we show the example for l1padme4.

Logon l1padme4 as daq (password for daq is on the board in Control Room)

[daq@l1padme2 ~]$ cd DAQ
[daq@l1padme2 DAQ]$ tools/kill_lvl1.sh
kill 112873 112958 113045 113132 113217 113304 113389 113476 113563 113650 116332
[daq@l1padme2 DAQ]$ kill 112873 112958 113045 113132 113217 113304 113389 113476 113563 113650 116332
[daq@l1padme2 DAQ]$ tools/kill_nc.sh 
kill 120215 120300 120387 120472 120558 120645 120732 120819 120906 120993 121078 121163 121248 121333 121418 121503 121588 121675 121762 121849 121936 122023 122108 122193 122278 122365 122449 122537 122624 122711
[daq@l1padme2 DAQ]$ kill 120215 120300 120387 120472 120558 120645 120732 120819 120906 120993 121078 121163 121248 121333 121418 121503 121588 121675 121762 121849 121936 122023 122108 122193 122278 122365 122449 122537 122624 122711

Please note that, as a safety measure, the kill_lvl1 and kill_nc scripts only show the command needed to kill all Level1 and network processes. It is up to the user to copy these commands and execute them, as shown in the example.

At this point issuing the ps -fu daq command should not show any PadmeLevel1, PadmeMerger or nc processes. If this is not the case, repeat the procedure and, if this fails again, contact an expert.

DAQ nodes clean-up

This procedure must be executed on nodes l0padme4 and l0padme5. Here we show the example for l0padme4.

Logon l0padme4 as daq

[daq@l0padme4 ~]$ cd DAQ
[daq@l0padme4 DAQ]$ tools/kill_daq.sh 
kill -9 24723 24812 24900 24989 25076 25162 25249 25342 25428 25516 25603 25689 25778 25864 25951 26038 26125 26212 26299 26386 26472 26560 26647 26735 26829 26917 27002 27091
[daq@l0padme4 DAQ]$ kill -9 24723 24812 24900 24989 25076 25162 25249 25342 25428 25516 25603 25689 25778 25864 25951 26038 26125 26212 26299 26386 26472 26560 26647 26735 26829 26917 27002 27091
[daq@l0padme4 DAQ]$ tools/kill_nc.sh 
kill 23567 23656 23658 23747 23833 23922 24011 24102 24192 24281 24370 24457 24548 24635
[daq@l0padme4 DAQ]$ kill 23567 23656 23658 23747 23833 23922 24011 24102 24192 24281 24370 24457 24548 24635
[daq@l0padme4 DAQ]$ cd local/run
[daq@l0padme4 run]$ rm *

At this point issuing the ps -fu daq command should not show any PadmeDAQ or nc processes. If this is not the case, repeat the procedure and, if this fails again, contact an expert.

Trigger node clean-up

Logon l0padme1 as daq

[daq@l0padme1 ~]$ cd DAQ
[daq@l0padme1 DAQ]$ tools/kill_trig.sh
kill -9 24723
[daq@l0padme1 DAQ]$ kill -9 24723
[daq@l0padme1 DAQ]$ tools/kill_nc.sh 
kill 23567 23656 23658 23747
[daq@l0padme1 DAQ]$ kill 23567 23656 23658 23747
[daq@l0padme1 DAQ]$ cd local/run
[daq@l0padme1 run]$ rm *

Streams folders clean-up

Sometimes, it could happen that the merger file looks like this:

- Opening output stream  4 '/home/daq/DAQ/local/streams/run_0000000_20181030_173618/run_0000000_20181030_173618_lvl1_04'
Default pipe size: 65536
Pipe size: 65536
set pipe size failed.: Cannot allocate memory
set pipe size failed.: Cannot allocate memory
- Using a total of 5 output Level1 streams
File /home/daq/DAQ/local/streams/run_0000000_20181030_173618/run_0000000_20181030_173618_trigger opened with format version 3
ERROR - File does not start with the right tag - Expected 0x9 - Found 0x0
ERROR while reading header of file /home/daq/DAQ/local/streams/run_0000000_20181030_173618/run_0000000_20181030_173618_b00_zsup. Aborting

In this case, you need to remove all the files inside the DAQ/local/streams folder, in every machine (that means daq user for l1padme3, l1padme4, l0padme4, l0padme5).

Clean up check

To check if the clean up procedure was correctly executed and the DAQ system is ready to be restarted the shifter can use the scripts described in the Verifying DAQ and ADC boards status section of Running the DAQ.

Notes

When this procedure is completed you can restart the DAQ system as described in Running the DAQ.