inconsistency between GPU/CPU inference

[ntedgi]

its not an floating point issue between devices and then only the certainty changes
on GPU i get (4/18) on CPU (16/18)

in order to hard code GPU and then CPU i follow this commit
6286a47
and change this 2 lines

line 20:
valid_output = torch.zeros(batch_size,max_len,feat_dim,dtype=torch.float32,device='cuda' if torch.cuda.is_available() else 'cpu')
...
Line 38:
self.device = "cuda" if torch.cuda.is_available() else "cpu"

one time to 'cpu' and in the other to 'cuda'

tested on:
Ubuntu 18.04
NVIDIA-SMI 435.21
Driver Version: 435.21
CUDA Version: 10.1
GeForce RTX 2070

Python 3.6
pytorch-transformers==1.2.0
torch==1.2.0

example : 18 posts

 tests = [
        "only 3 African countries voted in favour (Botswana, Djibouti, Liberia) - only 6 Asian countries voted in favour (Georgia, Japan, Maldives, Singapore, NATO'\''s Turkey and, of course, Israel) Is this the international community I keep hearing about? LMAO",
        "08 Apr 2018 – 20:55 Palestinian Journalists take part in a protest against the killing of fellow journalist Yasser Murtaja, near the Israel-Gaza border, in Rafah in the southern Gaza Strip, on April 8, 2018. AFP / Said Khatib By Fares Akram I Associated Press GAZA CITY:  Hundreds attended the funeral of a well-known Palestinian …  Gaza, buries, journalist , killed , while , covering, mass, protests",
        "Syria shot two surface-to-air missiles at Israeli jets—and they shot at the Israeli jets not even over their own air space.These were not errant missiles or spillover shots which is military parlance for mistakes. These missiles were calculated and intentionally targeted shots from an S-200 Russian missile system aimed at Israeli jets that were, quite frankly, targeting Syrian rocket launchers.This is a major change in Syrian engagement. It could signal an escalation of tensions between Israel and Syria, especially because three mortars were shot from Syria into Israel later that day.The reason Syria feels it can engage Israel is very clear. One word—Russia. Syrian leadership believes that Russia will protect them from Israel.Until now Israel had a carte blanche to fly over Syrian airspace. They did as they wished, and there was no significant response. Israel had a relatively free hand. That ended on Tuesday when Syrian President Bashar Assad began implementing his decision to change the entire status quo and push Israel out of Syrian airspace and off the offensive.Shooting the S-200s was part of this plan. This weapon was delivered to Assad’s father by the Russians in the late 1960s and was almost never used. The S-200 has a range of almost 200 miles (300km). A range that puts almost all of Israel’s major population centers at risk as well as all commercial aircraft coming in and out of Tel Aviv’s Ben Gurion Airport.The Syrian press, which Israel closely monitors, declared that their High Command repelled Israeli aggression. According to the official statement released by the government to the press, “The high command issued an order at 1 a.m. to strike at the aggressive Israeli attack.”  The statement also claimed that Syria “downed an Israeli plane west of Kunetra.”The Syrian headline read “The Army High Command Said That the Syrian Air Defense Systems Activated Against Israel War Jets and Shot Them Down.” This is more than an exaggeration. It is propaganda—part of a two-pronged effort aimed at establishing Syria’s military force as powerful and foreboding. First, it is an attempt to make Syrians feel as if they are finally standing up and taking a military stand against Israel. Second it serves as a message to terror groups opposing Assad, telling them that his forces are now strong enough to confront Israel. And if they are strong enough to confront Israel, they are certainly powerful enough to wipe out ISIS and Al Qaeda.But using a weapon like the S-200 was either a plain old big mistake, or it was a huge message out of proportion to the events.  Maybe it was both. The Israeli fighters were flying low. Despite its relatively old age in military years, the S-200 is still a long range and high altitude missile of 40,000 ft.Until now Israel has very clear mission when acting inside Syria. Whenever dangerous weapons were moving or arriving that would change the balance of power, Israel would strike. If the weapons were from Iran and headed to Hezbollah, Israel would strike. Israel would also punish Syria when its rockets and missiles specifically targeted Israelis or important locations—not rockets gone astray.We know that Russia has given Syria the green light in many areas. I cannot imagine, however, that Russia signed off on this latest adventure. Russia does not want to test its technological military prowess against Israel over Syria.The potential embarrassment to Russia would be so humiliating to them and jeopardize their entire Middle East plan. Imagine if there engaged in a dog fight over Syria or the Mediterranean and Russia took more losses than Israel. The entire episode would take only a few seconds—blink and it’s over. If Israel were to emerge victorious, which is highly probable, not only would Russia lose face but the ripple effect of that encounter in the Arabic and Muslim world would turn Israel into a superpower even greater than the Arab world already believes her to be.Russia cannot afford that kind of risk. They are playing for the long run in the Middle East. This was an amateur move on Syria’s part. There will be repercussions. Swaggering Syria Shoots at Israeli Jets, Flying Toward War? - 16 September 2016 israel, syria, hezbollah, russia, damascus, moscow, israel forever",
        "‼️REMEMBER - TODAY 👇😡👉before they start claiming, #Israel returned fire. 4 Palestinians Killed, Dozens Wounded by Israeli Live Fire in #Gaza Protests. Israeli military deploys #IronDome batteries bracing for #rocket fire. #ICC4Israel #BDS #Palestine https://t.co/berzwiWqBi https://www.haaretz.com/israel-news/.premium-gaza-protests-thousands-of-palestinians-expected-to-march-on-israeli-border-1.6157628?utm_source=Push_Notification&utm_medium=web_push&utm_campaign=General",
        " Golan Heights Asirigasyon a great nation of Israel .",
        "#Israel is afraid of any confrontation because it could take place within its territory, says #Hezbollah chief Sayyed Hassan Nasrallah. Sputnik",
        "#Trump'\''s announcement on the #Golan comes with vague accusations by #Pompeo that the Golan will serve as a buffer zone to protect #Israel from #Iran/#Hezbollah threat, when in fact this will likely prompt Hezbollah to move its ops to the Golan  @ejmalrai https://t.co/BFRYaKsc2D https://ejmagnier.com/2019/03/23/the-occupied-golan-heights-is-offered-to-israel-and-yet-the-indignation-is-verbal-nothing-more/",
        "@POTUS clearly loves Israel as America does. Americans understand the need for security there and the Golan Heights is key to that. @HeyTammyBruce joined @VarneyCo to weigh in on Israel'\''s sovereignty over #GolanHeights. #IWpol #IWfp https://t.co/DUbruvNwAW",
        "A Brief History of the Golan Heights, Claimed by Israel and Syria by JACEY FORTIN via NYT https://t.co/GcWtc5Zmuc https://t.co/E6U0KoaV8V https://www.nytimes.com/2019/03/21/world/middleeast/golan-heights-israel.html?partner=IFTTT",
        "Acrimony breaks out in cabinet over whether military’s deadly crackdown on protesters in #Gaza has been harsh enough. RW government of #Israel competes with itself to be even more brutal & inhumane. This isn'\''t a civilised #democracy, it'\''s a terror state. https://t.co/3c5GGgutC8 https://www.middleeasteye.net/news/israeli-cabinet-turmoil-minister-says-deadly-crackdown-gaza-protests-too-soft-1243574072",
        "After 52 years it is time for the United States to fully recognize Israel'\''s Sovereignty over the Golan Heights, which is of critical strategic and security importance to the State of Israel and Regional Stability! Trump tweets today. HUGE",
        "Ahed Tamimi, Palestinian Teenager Who Slapped Israeli Soldier, Is Released From Jail by REUTERS and THE ASSOCIATED PRESS via NYT The New York Times https://t.co/2W81L31RZw https://www.nytimes.com/2018/07/29/world/middleeast/ahed-tamimi-israel-west-bank.html?partner=IFTT",
        "All-Out War Coming: Record Number Of Israeli Tanks Amassed On Gaza Border   Zero Hedge https://t.co/j2lfIlUcQ0 https://www.zerohedge.com/news/2018-10-19/gaza-brink-huge-conflict-israel-amasses-tanks-along-border",
        "As long as the civilians in Gaza do not get relief, there will be no real progress toward calm and the situation could escalate at any moment. https://t.co/RlUY6yCI8h https://t.co/iE9JOSEicp https://www.haaretz.com/israel-news/egypt-urges-hamas-to-show-restraint-ahead-of-anniversary-of-gaza-border-protests-1.7063849",
        "Beware of becoming the monster you were afraid of Israel is more NAZI than the NAZI'\''s were https://t.co/4s9xhwfFxT https://twitter.com/KeesvdPijl1/status/1124968758919991297",
        "But why? Why have there been ruthless and merciless waves of venom poured out by New Zealand media and politicians onto Israel Folau? Why the level? The constancy? Not even Donald Trump has faced such a withering barrage of Left wing bigotry. So why? I humbly submit that Israel represents what the Left hate the most. Elliot Ikilei https://www.whaleoil.co.nz/2019/04/why-does-the-left-have-such-venom-when-it-comes-to-folau/ New Conservative NZ",
        "Canada now has over 350,000 patients, Israel is in excess of 35,000 patients for a population of eight million. … Here in Australia it'\''s developing really rapidly. ABC News",
        "Christians and Muslims are suffering. It’s not justice in here. It’s not a democracy in here. Christians in #Jerusalem share their stories of living in the Jewish state of Israel. #قروب_فلسطيني https://t.co/YO0K0kfBqU"
    ]

index	GPU	CPU
0	[NerEntity(entity=singapore, type=LOCATION, offset=(0, 252), probability=1.0), NerEntity(entity=botswana, type=LOCATION, offset=(0, 252), probability=1.0), NerEntity(entity=djibouti, type=LOCATION, offset=(0, 252), probability=1.0), NerEntity(entity=maldives, type=LOCATION, offset=(0, 252), probability=1.0), NerEntity(entity=liberia, type=LOCATION, offset=(0, 252), probability=1.0), NerEntity(entity=georgia, type=LOCATION, offset=(0, 252), probability=1.0), NerEntity(entity=japan, type=LOCATION, offset=(0, 252), probability=1.0), NerEntity(entity=nato, type=ORGANIZATION, offset=(0, 252), probability=1.0), NerEntity(entity=lmao, type=ORGANIZATION, offset=(0, 252), probability=1.0)]	[NerEntity(entity=singapore, type=LOCATION, offset=(0, 252), probability=1.0), NerEntity(entity=botswana, type=LOCATION, offset=(0, 252), probability=1.0), NerEntity(entity=djibouti, type=LOCATION, offset=(0, 252), probability=1.0), NerEntity(entity=maldives, type=LOCATION, offset=(0, 252), probability=1.0), NerEntity(entity=liberia, type=LOCATION, offset=(0, 252), probability=1.0), NerEntity(entity=georgia, type=LOCATION, offset=(0, 252), probability=1.0), NerEntity(entity=israel, type=LOCATION, offset=(0, 252), probability=1.0), NerEntity(entity=japan, type=LOCATION, offset=(0, 252), probability=1.0), NerEntity(entity=nato, type=ORGANIZATION, offset=(0, 252), probability=1.0), NerEntity(entity=lmao, type=ORGANIZATION, offset=(0, 252), probability=1.0)]
1	[NerEntity(entity=yasser murtaja, type=PERSON, offset=(0, 401), probability=1.0), NerEntity(entity=said khatib, type=ORGANIZATION, offset=(0, 401), probability=1.0), NerEntity(entity=fares akram, type=PERSON, offset=(0, 401), probability=1.0), NerEntity(entity=gaza strip, type=LOCATION, offset=(0, 401), probability=1.0), NerEntity(entity=rafah, type=LOCATION, offset=(0, 401), probability=1.0), NerEntity(entity=afp, type=ORGANIZATION, offset=(0, 401), probability=1.0)]	[NerEntity(entity=yasser murtaja, type=PERSON, offset=(0, 401), probability=1.0), NerEntity(entity=said khatib, type=ORGANIZATION, offset=(0, 401), probability=1.0), NerEntity(entity=fares akram, type=PERSON, offset=(0, 401), probability=1.0), NerEntity(entity=gaza strip, type=LOCATION, offset=(0, 401), probability=1.0), NerEntity(entity=rafah, type=LOCATION, offset=(0, 401), probability=1.0), NerEntity(entity=afp, type=ORGANIZATION, offset=(0, 401), probability=1.0)]
2	[]	[NerEntity(entity=hezbollah, type=ORGANIZATION, offset=(0, 4276), probability=1.0), NerEntity(entity=israel, type=LOCATION, offset=(0, 4276), probability=1.0), NerEntity(entity=syria, type=LOCATION, offset=(0, 4276), probability=1.0), NerEntity(entity=iran, type=LOCATION, offset=(0, 4276), probability=1.0)]
3	[]	[]
4	[]	[NerEntity(entity=golan heights, type=LOCATION, offset=(0, 53), probability=1.0), NerEntity(entity=asirigasyon, type=ORGANIZATION, offset=(0, 53), probability=1.0), NerEntity(entity=israel, type=LOCATION, offset=(0, 53), probability=1.0)]
5	[]	[NerEntity(entity=sayyed hassan nasrallah, type=PERSON, offset=(0, 143), probability=1.0), NerEntity(entity=sputnik, type=ORGANIZATION, offset=(0, 143), probability=1.0)]
6	[]	[NerEntity(entity=hezbollah, type=ORGANIZATION, offset=(0, 409), probability=1.0), NerEntity(entity=golan, type=LOCATION, offset=(0, 409), probability=1.0)]
7	[]	[NerEntity(entity=golan heights, type=LOCATION, offset=(0, 258), probability=1.0), NerEntity(entity=america, type=LOCATION, offset=(0, 258), probability=1.0), NerEntity(entity=israel, type=LOCATION, offset=(0, 258), probability=1.0)]
8	[]	[NerEntity(entity=golan heights, type=LOCATION, offset=(0, 229), probability=1.0), NerEntity(entity=israel, type=LOCATION, offset=(0, 229), probability=1.0), NerEntity(entity=syria, type=LOCATION, offset=(0, 229), probability=1.0), NerEntity(entity=jacey, type=PERSON, offset=(0, 229), probability=1.0), NerEntity(entity=nyt, type=ORGANIZATION, offset=(0, 229), probability=1.0)]
9	[]	[]
10	[]	[NerEntity(entity=state of israel, type=LOCATION, offset=(0, 240), probability=1.0), NerEntity(entity=united states, type=LOCATION, offset=(0, 240), probability=1.0), NerEntity(entity=golan heights, type=LOCATION, offset=(0, 240), probability=1.0)]
11	[]	[NerEntity(entity=ahed tamimi, type=PERSON, offset=(0, 270), probability=1.0), NerEntity(entity=reuters, type=ORGANIZATION, offset=(0, 270), probability=1.0), NerEntity(entity=nyt, type=ORGANIZATION, offset=(0, 270), probability=1.0)]
12	[NerEntity(entity=gaza, type=LOCATION, offset=(0, 211), probability=1.0)]	[NerEntity(entity=gaza border, type=LOCATION, offset=(0, 211), probability=1.0)]
13	[NerEntity(entity=gaza, type=LOCATION, offset=(0, 317), probability=1.0)]	[NerEntity(entity=gaza, type=LOCATION, offset=(0, 317), probability=1.0)]
14	[]	[NerEntity(entity=israel, type=LOCATION, offset=(0, 175), probability=1.0)]
15	[]	[NerEntity(entity=new conservative nz, type=ORGANIZATION, offset=(0, 445), probability=1.0), NerEntity(entity=elliot ikilei, type=PERSON, offset=(0, 445), probability=1.0), NerEntity(entity=israel folau, type=PERSON, offset=(0, 445), probability=1.0), NerEntity(entity=donald trump, type=PERSON, offset=(0, 445), probability=1.0), NerEntity(entity=new zealand, type=LOCATION, offset=(0, 445), probability=1.0)]
16	[]	[NerEntity(entity=australia, type=LOCATION, offset=(0, 174), probability=1.0), NerEntity(entity=abc news, type=ORGANIZATION, offset=(0, 174), probability=1.0), NerEntity(entity=canada, type=LOCATION, offset=(0, 174), probability=1.0), NerEntity(entity=israel, type=LOCATION, offset=(0, 174), probability=1.0)]
17	[]	[NerEntity(entity=state of israel, type=LOCATION, offset=(0, 217), probability=1.0)]
18	[]	[]
19	[]	[]

[ntedgi]

torch.no_grad() impacts the autograd engine and deactivate it. It will reduce memory usage and speed up computations but you won’t be able to backprop (which you don’t want in an eval script).

[ntedgi]

#71
torch.no_grad() crush on tokens larger the 512
this pr will close it