Transfer Requests for Run2016 data #666

mdunser · 2016-07-26T10:02:16Z

Hi all,

I'm starting a new issue for data-transfers of Run2016. Many of the datasets, despite being subscribed originally to CERN are being deleted fairly rapidly.

After some interaction with CompOps it has become clear that there is no longer any guarantee that any dataset of any run-period in 2016 will remain at the CERN T2. Computing resources are hitting their absolute limit pretty much everywhere, so whatever workflow that used to work in 2015 is no longer guaranteed to work in 2016.

For users of heppy and heppy_batch that means that they should have a look into using one of two options:

useAAA which copies the root files locally to /tmp via xrd and runs on them from there. In my experience this works well, though it sometimes takes many resubmissions for it to finally run through. This requires setting the environment variable X509_USER_PROXY to an empty file that exists.
crab3 works well, with some adaptions to the run-config and the crab/ directory. If you plan on running loads of data now or in the future, this might be worthwhile checking out. In my experience this work well.

It is, of course, still possible to transfer datasets to our local buffer, but be advised that this may take up to a week or more from your request on this thread to the dataset finally being present at CERN, and in addition the buffer has a limited size, so there is no way we can store all data in it just for the sake of it.

Best,
-m

gpetruc · 2016-07-26T10:12:35Z

On Tue, Jul 26, 2016 at 12:02 PM, mdunser [email protected] wrote:

Hi all,

I'm starting a new issue for data-transfers of Run2016. Many of the
datasets, despite being subscribed originally to CERN are being deleted
fairly rapidly.

After some interaction with CompOps it has become clear that there is no
longer any guarantee that any dataset of any run-period in 2016 will
remain at the CERN T2. Computing resources are hitting their absolute limit
pretty much everywhere, so whatever workflow that used to work in 2015 is
no longer guaranteed to work in 2016.

For users of heppy and heppy_batch that means that they should have a look
into using one of two options:

useAAA which copies the root files locally to /tmp via xrd and runs on
them from there. In my experience this works well, though it sometimes
takes many resubmissions for it to finally run through. This requires
setting the environment variable X509_USER_PROXY to an empty file that
exists.

crab3 works well, with some adaptions to the run-config and the crab/
directory. If you plan on running loads of data now or in the future, this
might be worthwhile checking out. In my experience this work well.

It is, of course, still possible to transfer datasets to our local buffer,
but be advised that this may take up to a week or more from your request on
this thread to the dataset finally being present at CERN, and in addition
the buffer has a limited size, so there is no way we can store all data in
it just for the sake of it.

We should discuss in a CMG group meeting - I believe the group should be
able to find the resources for hosting the data at CERN.
MC is less of an issue since individual job failures do not affect the
latency (one doesn't need 100.0% of complete jobs to be able to use a MC)

Giovanni

Best,
-m

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#666, or mute the thread
https://github.com/notifications/unsubscribe-auth/AEbbR2lmmafnsL0cT57J7gDIY_SnOB0xks5qZdspgaJpZM4JU-5I
.

steggema · 2016-08-24T10:56:14Z

Hi Marc,

Can you transfer

/SingleMuon/Run2016G-PromptReco-v1/MINIAOD ?

Thanks,
Jan

mdunser · 2016-08-24T11:16:25Z

Voila: https://cmsweb.cern.ch/phedex/prod/Request::View?request=765363
-m

On 24 Aug 2016, at 12:56, Jan Steggemann <[email protected]mailto:[email protected]> wrote:

Hi Marc,

Can you transfer

/SingleMuon/Run2016G-PromptReco-v1/MINIAOD ?

Thanks,
Jan

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/666#issuecomment-242026014, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEdevChErlIW1wy5AeIxxaJKjzTq9Hnfks5qjCNPgaJpZM4JU-5I.

cbotta · 2016-10-07T16:39:13Z

Hi Marc,

could you please transfer

/MET/Run2016F-PromptReco-v1/MINIAOD
/DoubleMuon/Run2016G-PromptReco-v1/MINIAOD

Many thanks,
cristina

cbotta · 2016-10-07T16:41:37Z

sorry, I meant:
/MET/Run2016F-PromptReco-v1/MINIAOD
/DoubleMuon/Run2016F-PromptReco-v1/MINIAOD

mdunser · 2016-10-07T16:42:43Z

hi cristina,

RunG i had requested some time ago, so that one should be there.
I just requested all RunF datasets to be replicated at CERN, but that
transfer will take a while I suppose.

-m

On 07 Oct 2016, at 18:39, cbotta <[email protected]mailto:[email protected]> wrote:

Hi Marc,

could you please transfer

/MET/Run2016F-PromptReco-v1/MINIAOD
/DoubleMuon/Run2016G-PromptReco-v1/MINIAOD

Many thanks,
cristina

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/666#issuecomment-252300878, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEdevLiIGwqj1qPBZxmF50MmweKmyKoGks5qxnWygaJpZM4JU-5I.

mdunser · 2016-10-08T10:02:58Z

just for the record, here the request: https://cmsweb.cern.ch/phedex/prod/Request::View?request=802579
-m

On 07 Oct 2016, at 18:42, Marc Dunser <[email protected]mailto:[email protected]> wrote:

hi cristina,

RunG i had requested some time ago, so that one should be there.
I just requested all RunF datasets to be replicated at CERN, but that
transfer will take a while I suppose.

-m

On 07 Oct 2016, at 18:39, cbotta <[email protected]mailto:[email protected]> wrote:

Hi Marc,

could you please transfer

/MET/Run2016F-PromptReco-v1/MINIAOD
/DoubleMuon/Run2016G-PromptReco-v1/MINIAOD

Many thanks,
cristina

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/666#issuecomment-252300878, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEdevLiIGwqj1qPBZxmF50MmweKmyKoGks5qxnWygaJpZM4JU-5I.

rmanzoni · 2016-10-31T19:57:21Z

Hi Marc,

I need to look into these data somewhat urgently

/SingleMuon/Run2016H-PromptReco-v1/MINIAOD
/SingleMuon/Run2016H-PromptReco-v2/MINIAOD
/SingleMuon/Run2016H-PromptReco-v3/MINIAOD

Given your initial message, would a transfer request still make sense or I'd be better off using crab / AAA?

Thanks,
Riccardo

mdunser · 2016-10-31T20:18:30Z

Hi Riccardo,

so the first dataset (v1) is already at CERN so you should be able to access it
without crab or AAA.

The other two don’t exist.

Best,
-m

On 31 Oct 2016, at 20:57, Riccardo Manzoni <[email protected]mailto:[email protected]> wrote:

Hi Marc,

I need to look into these data somewhat urgently

/SingleMuon/Run2016G-PromptReco-v1/MINIAOD
/SingleMuon/Run2016G-PromptReco-v2/MINIAOD
/SingleMuon/Run2016G-PromptReco-v3/MINIAOD

Given your initial message, would a transfer request still make sense or I'd be better off using crab / AAA?

Thanks,
Riccardo

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/666#issuecomment-257402686, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEdevLTtWW7ZCT42zFhpELvwptJIUSipks5q5kgigaJpZM4JU-5I.

rmanzoni · 2016-10-31T20:19:44Z

Sorry Marc, I corrected the first message, I meant 2016H v1 to v3.
Thanks,
Riccardo

mdunser · 2016-10-31T20:27:50Z

Hi,

I made the request for all RunH (v1,v2,v3) samples just now:

https://cmsweb.cern.ch/phedex/prod/Request::View?request=817882

It will probably be approved tomorrow, and only then will the transfer start, so
you can expect the samples earliest tomorrow ~evening.

Depending on your interpretation of “urgently” you might want to consider
using AAA / crab.

Best,
-m

On 31 Oct 2016, at 21:19, Riccardo Manzoni <[email protected]mailto:[email protected]> wrote:

Sorry Marc, I corrected the first message, I meant 2016H v1 to v3.
Thanks,
Riccardo

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/666#issuecomment-257408766, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEdevJRogB-2ElcP7Bk8QN7f0BeLX7erks5q5k1jgaJpZM4JU-5I.

cbotta · 2016-11-11T10:48:17Z

Hello Marc,

could you please add to
https://cmsweb.cern.ch/phedex/prod/Request::View?request=822372
also
/DoubleMuon/Run2016_-23Sep2016-v_/MINIAOD
?

Many thanks
cheers,
Cristina

mdunser · 2016-11-11T10:57:47Z

Hi,

this was already part of this request:
https://cmsweb.cern.ch/phedex/prod/Request::View?request=820680
And the files should already be here...

Or am I missing something?

-m

On 11 Nov 2016, at 11:48, cbotta <[email protected]mailto:[email protected]> wrote:

Hello Marc,

could you please add to
https://cmsweb.cern.ch/phedex/prod/Request::View?request=822372
also
/DoubleMuon/Run2016-23Sep2016-v/MINIAOD
?

Many thanks
cheers,
Cristina

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/666#issuecomment-259931784, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEdevBx_HE895lc1ngMcYPIASbzHvPRuks5q9EfygaJpZM4JU-5I.

mariadalfonso · 2016-11-11T11:02:46Z

Ciao Marc.

Would be great if you can add the

/SinglePhoton/Run2016B-23Sep2016-v3/MINIAOD
/SinglePhoton/Run2016C-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016D-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016E-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016F-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016G-23Sep2016-v1/MINIAOD

do not see in the https://cmsweb.cern.ch/phedex/prod/Request::View?request=820680

Thanks

Maria

mdunser · 2016-11-11T11:14:29Z

Hi Maria, all,

here’s the list of the datasets in the local space:

https://cmsweb.cern.ch/phedex/prod/Data::Subscriptions#state=create_since%3D1344868584%3Bgroup%3Dlocal%3Bnode%3D1561https://cmsweb.cern.ch/phedex/prod/Data::Subscriptions#state=create_since=1344868584;group=local;node=1561

The request which has the SinglePhotons in it:

https://cmsweb.cern.ch/phedex/prod/Request::View?request=820681

For all I can tell right now, all these datasets are here.

-m

On 11 Nov 2016, at 12:02, mariadalfonso <[email protected]mailto:[email protected]> wrote:

Ciao Marc.

Would be great if you can add the

/SinglePhoton/Run2016B-23Sep2016-v3/MINIAOD
/SinglePhoton/Run2016C-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016D-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016E-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016F-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016G-23Sep2016-v1/MINIAOD

do not see in the https://cmsweb.cern.ch/phedex/prod/Request::View?request=820680

Thanks

Maria

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/666#issuecomment-259934215, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEdevCkChh2zc1jHELRaaLZKnEmMIt5Oks5q9EtWgaJpZM4JU-5I.

mdunser · 2017-01-11T13:30:08Z

Hi all,

new year, new cleaning-up campaign. Hooray!

The situation is that the extended local space which is now 300 TB is overfull. We are a few TB above our quota, so there is no way around cleaning up a certain amount of the data we have subscribed to the local space.

So, unless there are some solid reasons for keeping any of the following datasets in the local space, I will request their deletion. This will free up roughly 50 TB, which should buy us some time before we (I) have to think about something new/smarter.

These are all PromptReco of 2016 data PDs.

Before you request keeping them, please have a good look and some solid reasoning. Keeping datasets just for funsies won't cut it anymore, unfortunately.

Also: datasets can clearly be retrieved back at a later point if that were necessary.

Best,
-m

/BTagCSV/Run2016*-PromptReco-v*/MINIAOD
/BTagMu/Run2016*-PromptReco-v*/MINIAOD
/Charmonium/Run2016*-PromptReco-v*/MINIAOD
/Commissioning/Run2016*-PromptReco-v*/MINIAOD
/DisplacedJet/Run2016*-PromptReco-v*/MINIAOD
/EmptyBX/Run2016*-PromptReco-v*/MINIAOD
/FSQJets/Run2016*-PromptReco-v*/MINIAOD
/HINCaloJets/Run2016*-PromptReco-v*/MINIAOD
/HINPFJets/Run2016*-PromptReco-v*/MINIAOD
/HINPhoton/Run2016*-PromptReco-v*/MINIAOD
/HLTPhysics/Run2016*-PromptReco-v*/MINIAOD
/HLTPhysicsBunchTrains/Run2016*-PromptReco-v*/MINIAOD
/HLTPhysicsIsolatedBunch/Run2016*-PromptReco-v*/MINIAOD
/HcalHPDNoise/Run2016*-PromptReco-v*/MINIAOD
/HcalNZS/Run2016*-PromptReco-v*/MINIAOD
/HighMultiplicityEOF/Run2016*-PromptReco-v*/MINIAOD
/L1MinimumBia*/Run2016*-PromptReco-v*/MINIAOD
/MinimumBias/Run2016*-PromptReco-v*/MINIAOD
/MuOnia/Run2016*-PromptReco-v*/MINIAOD
/NoBPTX/Run2016*-PromptReco-v*/MINIAOD
/ParkingScoutingMonitor/Run2016*-PromptReco-v*/MINIAOD
/ZeroBia*/Run2016*-PromptReco-v*/MINIAOD

steggema · 2017-03-21T09:31:13Z

Hi Marc,

Can you transfer the following datasets to CERN:

/Tau/Run2016B-03Feb2017_ver2-v2/MINIAOD
/Tau/Run2016E-03Feb2017-v1/MINIAOD
/Tau/Run2016G-03Feb2017-v1/MINIAOD

Thanks!
Jan

mdunser · 2017-03-21T16:16:32Z

Hi Jan, request is made: https://cmsweb.cern.ch/phedex/prod/Request::View?request=967091 Best, -m On 21 Mar 2017, at 10:31, Jan Steggemann <[email protected]<mailto:[email protected]>> wrote: Hi Marc, Can you transfer the following datasets to CERN: /Tau/Run2016B-03Feb2017_ver2-v2/MINIAOD /Tau/Run2016E-03Feb2017-v1/MINIAOD /Tau/Run2016G-03Feb2017-v1/MINIAOD Thanks! Jan — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#666 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEdevKmD46V1W99CvdHtXT_KDV-aVMatks5rn5jhgaJpZM4JU-5I>.

mariadalfonso · 2017-09-21T14:21:01Z

@mdunser
Can you please transfer this dataset /DoubleMuon/Run2016E-03Feb2017-v1/MINIAOD
Maria

mdunser · 2017-09-25T08:19:22Z

sorry for being a bit late on this, request is here: https://cmsweb.cern.ch/phedex/prod/Request::View?request=1113675 -m On 21 Sep 2017, at 16:21, mariadalfonso <[email protected]<mailto:[email protected]>> wrote: @mdunser<https://github.com/mdunser> Can you please transfer this dataset /DoubleMuon/Run2016E-03Feb2017-v1/MINIAOD Maria — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#666 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEdevJOEynsZ5x0nwt31dSQIHBq1mu-iks5sknDTgaJpZM4JU-5I>.

mariadalfonso · 2017-10-13T09:16:35Z

@mdunser

can you please transfer , with high priority,
/DoubleEG/Run2016D-03Feb2017-v1/MINIAOD
/DoubleMuon/Run2016H-03Feb2017_ver3-v1/MINIAOD
/SinglePhoton/Run2016H-03Feb2017_ver3-v1/MINIAOD

Thanks

mdunser · 2017-10-16T16:51:21Z

here’s the request: https://cmsweb.cern.ch/phedex/prod/Request::View?request=1132385 -m On 13 Oct 2017, at 11:16, mariadalfonso <[email protected]<mailto:[email protected]>> wrote: @mdunser<https://github.com/mdunser> can you please transfer , with high priority, /DoubleEG/Run2016D-03Feb2017-v1/MINIAOD /DoubleMuon/Run2016H-03Feb2017_ver3-v1/MINIAOD /SinglePhoton/Run2016H-03Feb2017_ver3-v1/MINIAOD Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#666 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEdevDSTDNlwRa0GvaeEpWkvpvYywu2iks5sryp1gaJpZM4JU-5I>.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transfer Requests for Run2016 data #666

Transfer Requests for Run2016 data #666

mdunser commented Jul 26, 2016

gpetruc commented Jul 26, 2016

steggema commented Aug 24, 2016

mdunser commented Aug 24, 2016

cbotta commented Oct 7, 2016

cbotta commented Oct 7, 2016

mdunser commented Oct 7, 2016

mdunser commented Oct 8, 2016

rmanzoni commented Oct 31, 2016 •

edited

Loading

mdunser commented Oct 31, 2016

rmanzoni commented Oct 31, 2016

mdunser commented Oct 31, 2016

cbotta commented Nov 11, 2016

mdunser commented Nov 11, 2016

mariadalfonso commented Nov 11, 2016

mdunser commented Nov 11, 2016

mdunser commented Jan 11, 2017

steggema commented Mar 21, 2017

mdunser commented Mar 21, 2017 via email

mariadalfonso commented Sep 21, 2017

mdunser commented Sep 25, 2017 via email

mariadalfonso commented Oct 13, 2017

mdunser commented Oct 16, 2017 via email

Transfer Requests for Run2016 data #666

Transfer Requests for Run2016 data #666

Comments

mdunser commented Jul 26, 2016

gpetruc commented Jul 26, 2016

steggema commented Aug 24, 2016

mdunser commented Aug 24, 2016

cbotta commented Oct 7, 2016

cbotta commented Oct 7, 2016

mdunser commented Oct 7, 2016

mdunser commented Oct 8, 2016

rmanzoni commented Oct 31, 2016 • edited Loading

mdunser commented Oct 31, 2016

rmanzoni commented Oct 31, 2016

mdunser commented Oct 31, 2016

cbotta commented Nov 11, 2016

mdunser commented Nov 11, 2016

mariadalfonso commented Nov 11, 2016

mdunser commented Nov 11, 2016

mdunser commented Jan 11, 2017

steggema commented Mar 21, 2017

mdunser commented Mar 21, 2017 via email

mariadalfonso commented Sep 21, 2017

mdunser commented Sep 25, 2017 via email

mariadalfonso commented Oct 13, 2017

mdunser commented Oct 16, 2017 via email

rmanzoni commented Oct 31, 2016 •

edited

Loading