Rework support for XEP-0384: OMEMO Encryption

    Changes:

    Rework integration tests
    New structure of base integration test classes
    bump dependency on signal-protocol-java from 2.4.0 to 2.6.2
    Introduced CachingOmemoStore implementations
    Use CachingOmemoStore classes in integration tests
    Removed OmemoSession classes (replaced with more logical OmemoRatchet classes)
    Consequently also removed load/storeOmemoSession methods from OmemoStore
    Removed some clutter from KeyUtil classes
    Moved trust decision related code from OmemoStore to TrustCallback
    Require authenticated connection for many functions
    Add async initialization function in OmemoStore
    Refactor omemo test package (/java/org/jivesoftware/smack/omemo -> /java/org/jivesoftware/smackx)
    Remove OmemoStore method isFreshInstallation() as well as defaultDeviceId related stuff
    FileBasedOmemoStore: Add cleaner methods to store/load base data types (Using tryWithResource, only for future releases, once Android API gets bumped)
    Attempt to make OmemoManager thread safe
    new logic for getInstanceFor() deviceId determination
    OmemoManagers encrypt methods now don't throw exceptions when encryption for some devices fails. Instead message gets encrypted when possible and more information about failures gets returned alongside the message itself
    Added OmemoMessage class for that purpose
    Reworked entire OmemoService class
    Use safer logic for creating trust-ignoring messages (like ratchet-update messages)
    Restructure elements/provider in order to prepare for OMEMO namespace bumps
    Remove OmemoManager.regenerate() methods in favor of getInstanceFor(connection, randomDeviceId)
    Removed some unnecessary configuration options
    Prepare for support of more AES message key types
    Simplify session creation
    Where possible, avoid side effects in methods
    Add UntrustedOmemoIdentityException
    Add TrustState enum
    More improved tests

build.gradle

@@ -548,10 +548,12 @@ task clirrRootReport(type: org.kordamp.gradle.clirr.ClirrReportTask) {
 }
 
 task integrationTest {
+	description 'Verify correct functionality of Smack by running some integration tests.'
 	dependsOn project(':smack-integration-test').tasks.run
 }
 
 task omemoSignalIntTest {
+	description 'Run integration tests of the smack-omemo module in combination with smack-omemo-signal.'
 	dependsOn project(':smack-omemo-signal-integration-test').tasks.run
 }
 

documentation/extensions/omemo_migration_4.2.0_head.md

@@ -0,0 +1,43 @@
+Migrating smack-omemo from 4.2.1 to 4.x.x
+=========================================
+
+The implementation of smack-omemo and smack-omemo-signal was originally started as an
+academic project under pressure of time.
+For that reason, the API was not perfect when OMEMO support was first introduced in
+Smack in version 4.2.1.
+Many issues of smack-omemo have been resolved over the course of the last year in
+a major effort, which is why smack-omemo and smack-omemo-signalwere excluded from
+the 4.2.2 release.
+
+During this time major parts of the implementation were redone and the API changed
+as a consequence of that. This guide will go through all notable changes in order
+to make the process of upgrading as easy and straight forward as possible.
+
+## Trust
+One major change is, that the OmemoStore implementations no longer store trust decisions.
+Methods related to trust have been removed from OmemoStore implementations.
+Instead the client is now responsible to store those.
+Upon startup, the client now must pass an `OmemoTrustCallback` to the `OmemoManager`
+which is used to access and change trust decisions.
+
+It is recommended for the client to store trust decisions as tuples of (omemo device,
+fingerprint of identityKey, trust state).
+When querying a trust decision (aka. "Is this fingerprint trusted for that device?),
+the local fingerprint should be compared to the provided fingerprint.
+
+The method signatures for setting and querying trust from inside the OmemoManager are
+still the same. Internally they access the `OmemoTrustCallback` set by the client.
+
+## Encryption
+Message encryption in smack-omemo 4.2.1 was ugly. Encryption for multiple devices
+could fail because session negotiation could go wrong, which resulted in an
+exception, which contained all devices with working sessions.
+That exception could then be used in 
+`OmemoManager.encryptForExistingSessions(CannotEstablishOmemoSessionException exception, String message)`,
+to encrypt the message for all devices with a session.
+
+The new API is 
+
+
+
+

smack-integration-test/build.gradle

@@ -13,6 +13,7 @@ dependencies {
 	compile project(':smack-experimental')
 	compile project(':smack-omemo')
 	compile project(':smack-debug')
+	compile project(path: ":smack-omemo", configuration: "testRuntime")
 	compile 'org.reflections:reflections:0.9.9-RC1'
 	compile 'eu.geekplace.javapinning:java-pinning-java7:1.1.0-alpha1'
 	// Note that the junit-vintage-engine runtime dependency is not

smack-integration-test/src/main/java/org/jivesoftware/smackx/omemo/AbstractOmemoIntegrationTest.java

@@ -16,40 +16,21 @@
  */
 package org.jivesoftware.smackx.omemo;
 
-import java.io.File;
-import java.util.logging.Level;
-
 import org.jivesoftware.smack.SmackException;
 import org.jivesoftware.smack.XMPPException;
 
 import org.igniterealtime.smack.inttest.AbstractSmackIntegrationTest;
 import org.igniterealtime.smack.inttest.SmackIntegrationTestEnvironment;
 import org.igniterealtime.smack.inttest.TestNotPossibleException;
-import org.junit.AfterClass;
-import org.junit.BeforeClass;
 
 /**
  * Super class for OMEMO integration tests.
  */
 public abstract class AbstractOmemoIntegrationTest extends AbstractSmackIntegrationTest {
 
-    private static final File storePath;
-
-    static {
-        String userHome = System.getProperty("user.home");
-        if (userHome != null) {
-            File f = new File(userHome);
-            storePath = new File(f, ".config/smack-integration-test/store");
-        } else {
-            storePath = new File("int_test_omemo_store");
-        }
-    }
-
     public AbstractOmemoIntegrationTest(SmackIntegrationTestEnvironment environment) throws XMPPException.XMPPErrorException, SmackException.NotConnectedException, InterruptedException, SmackException.NoResponseException, TestNotPossibleException {
         super(environment);
-        if (OmemoConfiguration.getFileBasedOmemoStoreDefaultPath() == null) {
-            OmemoConfiguration.setFileBasedOmemoStoreDefaultPath(storePath);
-        }
+
         // Test for server support
         if (!OmemoManager.serverSupportsOmemo(connection, connection.getXMPPServiceDomain())) {
             throw new TestNotPossibleException("Server does not support OMEMO (PubSub)");
@@ -59,23 +40,8 @@ public AbstractOmemoIntegrationTest(SmackIntegrationTestEnvironment environment)
         if (!OmemoService.isServiceRegistered()) {
             throw new TestNotPossibleException("No OmemoService registered.");
         }
-    }
-
-    @BeforeClass
-    public void beforeTest() {
-        LOGGER.log(Level.INFO, "START EXECUTION");
-        OmemoIntegrationTestHelper.deletePath(storePath);
-        before();
-    }
 
-    @AfterClass
-    public void afterTest() {
-        after();
-        OmemoIntegrationTestHelper.deletePath(storePath);
-        LOGGER.log(Level.INFO, "END EXECUTION");
+        OmemoConfiguration.setCompleteSessionWithEmptyMessage(true);
+        OmemoConfiguration.setRepairBrokenSessionsWithPrekeyMessages(true);
     }
-
-    public abstract void before();
-
-    public abstract void after();
 }

smack-integration-test/src/main/java/org/jivesoftware/smackx/omemo/AbstractOmemoMessageListener.java

@@ -0,0 +1,149 @@
+/**
+ *
+ * Copyright 2018 Paul Schaub
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.jivesoftware.smackx.omemo;
+
+import org.jivesoftware.smack.packet.Message;
+import org.jivesoftware.smack.packet.Stanza;
+import org.jivesoftware.smackx.carbons.packet.CarbonExtension;
+import org.jivesoftware.smackx.omemo.listener.OmemoMessageListener;
+
+import org.igniterealtime.smack.inttest.util.ResultSyncPoint;
+import org.igniterealtime.smack.inttest.util.SimpleResultSyncPoint;
+
+/**
+ * Convenience class. This listener is used so that implementers of OmemoMessageListener don't have to implement
+ * both messages. Instead they can just overwrite the message they want to implement.
+ */
+public class AbstractOmemoMessageListener implements OmemoMessageListener {
+
+    @Override
+    public void onOmemoMessageReceived(Stanza stanza, OmemoMessage.Received decryptedMessage) {
+        // Override me
+    }
+
+    @Override
+    public void onOmemoCarbonCopyReceived(CarbonExtension.Direction direction, Message carbonCopy, Message wrappingMessage, OmemoMessage.Received decryptedCarbonCopy) {
+        // Override me
+    }
+
+    private static class SyncPointListener extends AbstractOmemoMessageListener {
+        protected final ResultSyncPoint<?,?> syncPoint;
+
+        SyncPointListener(ResultSyncPoint<?,?> syncPoint) {
+            this.syncPoint = syncPoint;
+        }
+
+        public ResultSyncPoint<?, ?> getSyncPoint() {
+            return syncPoint;
+        }
+    }
+
+    static class MessageListener extends SyncPointListener {
+
+        protected final String expectedMessage;
+
+        MessageListener(String expectedMessage, SimpleResultSyncPoint syncPoint) {
+            super(syncPoint);
+            this.expectedMessage = expectedMessage;
+        }
+
+        MessageListener(String expectedMessage) {
+            this(expectedMessage, new SimpleResultSyncPoint());
+        }
+
+        @Override
+        public void onOmemoMessageReceived(Stanza stanza, OmemoMessage.Received received) {
+            SimpleResultSyncPoint srp = (SimpleResultSyncPoint) syncPoint;
+            if (received.isKeyTransportMessage()) {
+                return;
+            }
+
+            if (received.getBody().equals(expectedMessage)) {
+                srp.signal();
+            } else {
+                srp.signalFailure("Received decrypted message was not equal to sent message.");
+            }
+        }
+    }
+
+    static class PreKeyMessageListener extends MessageListener {
+        PreKeyMessageListener(String expectedMessage, SimpleResultSyncPoint syncPoint) {
+            super(expectedMessage, syncPoint);
+        }
+
+        PreKeyMessageListener(String expectedMessage) {
+            this(expectedMessage, new SimpleResultSyncPoint());
+        }
+
+        @Override
+        public void onOmemoMessageReceived(Stanza stanza, OmemoMessage.Received received) {
+            SimpleResultSyncPoint srp = (SimpleResultSyncPoint) syncPoint;
+            if (received.isKeyTransportMessage()) {
+                return;
+            }
+
+            if (received.isPreKeyMessage()) {
+                if (received.getBody().equals(expectedMessage)) {
+                    srp.signal();
+                } else {
+                    srp.signalFailure("Received decrypted message was not equal to sent message.");
+                }
+            } else {
+                srp.signalFailure("Received message was not a PreKeyMessage.");
+            }
+        }
+    }
+
+    static class KeyTransportListener extends SyncPointListener {
+
+        KeyTransportListener(SimpleResultSyncPoint resultSyncPoint) {
+            super(resultSyncPoint);
+        }
+
+        KeyTransportListener() {
+            this(new SimpleResultSyncPoint());
+        }
+
+        @Override
+        public void onOmemoMessageReceived(Stanza stanza, OmemoMessage.Received received) {
+            SimpleResultSyncPoint s = (SimpleResultSyncPoint) syncPoint;
+            if (received.isKeyTransportMessage()) {
+                s.signal();
+            }
+        }
+    }
+
+    static class PreKeyKeyTransportListener extends KeyTransportListener {
+        PreKeyKeyTransportListener(SimpleResultSyncPoint resultSyncPoint) {
+            super(resultSyncPoint);
+        }
+
+        PreKeyKeyTransportListener() {
+            this(new SimpleResultSyncPoint());
+        }
+
+        @Override
+        public void onOmemoMessageReceived(Stanza stanza, OmemoMessage.Received received) {
+            SimpleResultSyncPoint s = (SimpleResultSyncPoint) syncPoint;
+            if (received.isPreKeyMessage()) {
+                if (received.isKeyTransportMessage()) {
+                    s.signal();
+                }
+            }
+        }
+    }
+}

smack-integration-test/src/main/java/org/jivesoftware/smackx/omemo/AbstractTwoUsersOmemoIntegrationTest.java

@@ -0,0 +1,75 @@
+/**
+ *
+ * Copyright 2017 Paul Schaub
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.jivesoftware.smackx.omemo;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+
+import java.util.logging.Level;
+
+import org.jivesoftware.smack.SmackException;
+import org.jivesoftware.smack.XMPPException;
+
+import org.igniterealtime.smack.inttest.SmackIntegrationTestEnvironment;
+import org.igniterealtime.smack.inttest.TestNotPossibleException;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+
+/**
+ * Abstract OMEMO integration test framing, which creates and initializes two OmemoManagers (for conOne and conTwo).
+ * Both users subscribe to one another and trust their identities.
+ * After the test traces in PubSub and in the users Rosters are removed.
+ */
+public abstract class AbstractTwoUsersOmemoIntegrationTest extends AbstractOmemoIntegrationTest {
+
+    protected OmemoManager alice, bob;
+
+    public AbstractTwoUsersOmemoIntegrationTest(SmackIntegrationTestEnvironment environment)
+            throws XMPPException.XMPPErrorException, SmackException.NotConnectedException, InterruptedException,
+            SmackException.NoResponseException, TestNotPossibleException {
+        super(environment);
+    }
+
+    @BeforeClass
+    public void setup() throws Exception {
+        alice = OmemoManagerSetupHelper.prepareOmemoManager(conOne);
+        bob = OmemoManagerSetupHelper.prepareOmemoManager(conTwo);
+
+        LOGGER.log(Level.FINE, "Alice: " + alice.getOwnDevice() + " Bob: " + bob.getOwnDevice());
+        assertFalse(alice.getDeviceId().equals(bob.getDeviceId()));
+
+        // Subscribe presences
+        OmemoManagerSetupHelper.syncSubscribePresence(alice.getConnection(), bob.getConnection(), "bob", null);
+        OmemoManagerSetupHelper.syncSubscribePresence(bob.getConnection(), alice.getConnection(), "alice", null);
+
+        OmemoManagerSetupHelper.trustAllIdentitiesWithTests(alice, bob);    // Alice trusts Bob's devices
+        OmemoManagerSetupHelper.trustAllIdentitiesWithTests(bob, alice);    // Bob trusts Alice' and Mallory's devices
+
+        assertEquals(bob.getOwnFingerprint(), alice.getActiveFingerprints(bob.getOwnJid()).get(bob.getOwnDevice()));
+        assertEquals(alice.getOwnFingerprint(), bob.getActiveFingerprints(alice.getOwnJid()).get(alice.getOwnDevice()));
+    }
+
+    @AfterClass
+    public void cleanUp() {
+        alice.stopStanzaAndPEPListeners();
+        bob.stopStanzaAndPEPListeners();
+        OmemoManagerSetupHelper.cleanUpPubSub(alice);
+        OmemoManagerSetupHelper.cleanUpRoster(alice);
+        OmemoManagerSetupHelper.cleanUpPubSub(bob);
+        OmemoManagerSetupHelper.cleanUpRoster(bob);
+    }
+}