Fix test_packages failures

[rileyjmurray]

Running pytest test/test_packages on the latest develop branch results in errors. Summary output:

 ==================================================================================== short test summary info ====================================================================================
FAILED test/test_packages/drivers/test_timedep.py::TimeDependentTestCase::test_time_dependent_datagen - AssertionError: False is not true
FAILED test/test_packages/objects/test_evaltree.py::LayoutTestCase::test_map_layout - AttributeError: 'MatrixForwardSimulator' object has no attribute 'calclib'
FAILED test/test_packages/drivers/test_drivers.py::TestDriversMethods::test_longSequenceGST_CPTP - ValueError: Vectors `a` and `b` don't have the same conjugate-pair structure,  and so they cannot be matched in a way the preserves this structure.

I've dug into each of these.

Failures in test/test_packages/drivers/test_timedep.py are related to the use of float32 to hold count data in dataset.py. After the 12/17 dev meeting I implemented a proper fix.

Failures in test/test_packages/objects/test_evaltree.py were very mild. I implemented fixes in this PR but removed them since fixes are already present in #517.

All that remains are failures in test/test_packages/drivers/test_drivers.py.

Click me for details on those failures.

====================================================================== FAILURES =======================================================================
____________________________________________________ TestDriversMethods.test_longSequenceGST_CPTP _____________________________________________________

self = <test.test_packages.drivers.test_drivers.TestDriversMethods testMethod=test_longSequenceGST_CPTP>

    def test_longSequenceGST_CPTP(self):
        ds = self.ds
    
        target_model = self.model
        target_model.set_all_parameterizations("CPTPLND")
    
        maxLens = self.maxLens
        result = self.runSilent(pygsti.run_long_sequence_gst,
                                ds, target_model, self.prep_fiducials, self.meas_fiducials,
                                self.germs, maxLens, disable_checkpointing=True,
                                advanced_options= {'max_iterations':3})
    
        #create a report...
>       pygsti.report.construct_standard_report(result, title="CPTP Gates report", verbosity=0).write_html(temp_files + "/full_report_CPTPGates")

test/test_packages/drivers/test_drivers.py:125: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pygsti/report/report.py:174: in write_html
    qtys = self._build(build_options)
pygsti/report/report.py:89: in _build
    qtys.update(section.render(self._workspace, **full_params))
pygsti/report/section/gauge.py:32: in render
    **super().render(
pygsti/report/section/__init__.py:78: in render
    return {
pygsti/report/section/__init__.py:79: in <dictcomp>
    k: v(workspace, brevity=brevity, **kwargs)
pygsti/report/section/gauge.py:108: in final_gauge_inv_metric_table
    return workspace.GaugeRobustMetricTable(
<string>:2: in factoryfn
    ???
pygsti/report/workspacetables.py:888: in __init__
    super(GaugeRobustMetricTable, self).__init__(ws, self._create, model, target_model,
pygsti/report/workspace.py:2056: in __init__
    self.ws.switched_compute(self.tablefn, *self.initargs)
pygsti/report/workspace.py:713: in switched_compute
    key, result = self.smartCache.cached_compute(fn, argVals)
pygsti/baseobjs/smartcache.py:312: in cached_compute
    self.cache[key] = fn(*arg_vals, **kwargs)
pygsti/report/workspacetables.py:932: in _create
    Ugauge = _tools.compute_best_case_gauge_transform(gate_mx, target_gate_mx)
pygsti/tools/optools.py:2228: in compute_best_case_gauge_transform
    pairs = _mt.minweight_match_realmxeigs(evals_gate, evals_tgt)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = array([ 1.        , -0.08031917, -0.00217066,  0.0025635 ])
b = array([1.00000000e+00+0.j, 2.64708821e-16+1.j, 2.64708821e-16-1.j,
       1.00000000e+00+0.j]), metricfn = None, pass_indices_to_metricfn = False
eps = 1e-09

    def minweight_match_realmxeigs(a, b, metricfn=None,
                                   pass_indices_to_metricfn=False, eps=1e-9):
        """
        Matches the elements of `a` and `b`, whose elements are assumed to either real or one-half of a conjugate pair.
    
        Matching is performed by minimizing the weight between elements,
        defined as the sum of `metricfn(x,y)` over all `(x,y)` pairs
        (`x` in `a` and `y` in `b`).  If straightforward matching fails
        to preserve eigenvalue conjugacy relations, then real and conjugate-
        pair eigenvalues are matched *separately* to ensure relations are
        preserved (but this can result in a sub-optimal matching).  A
        ValueError is raised when the elements of `a` and `b` have incompatible
        conjugacy structures (#'s of conjugate vs. real pairs).
    
        Parameters
        ----------
        a : numpy.ndarray
            First 1D array to match.
    
        b : numpy.ndarray
            Second 1D array to match.
    
        metricfn : function, optional
            A function of two float parameters, `x` and `y`,which defines the cost
            associated with matching `x` with `y`.  If None, `abs(x-y)` is used.
    
        pass_indices_to_metricfn : bool, optional
            If True, the metric function is passed two *indices* into the `a` and
            `b` arrays, respectively, instead of the values.
    
        eps : float, optional
            Tolerance when checking if eigenvalues are equal to each other.
    
        Returns
        -------
        pairs : list
            A list of 2-tuple pairs of indices `(ix,iy)` giving the indices into
            `a` and `b` respectively of each matched pair.
        """
    
        def check(pairs):
            for i, (p0, p1) in enumerate(pairs):
                for q0, q1 in pairs[i + 1:]:
                    a_conj = _np.isclose(a[p0], _np.conjugate(a[q0]))
                    b_conj = _np.isclose(b[p1], _np.conjugate(b[q1]))
                    if (abs(a[p0].imag) > 1e-6 and a_conj and not b_conj) or \
                       (abs(b[p1].imag) > 1e-6 and b_conj and not a_conj):
                        #print("DB: FALSE at: ",(p0,p1),(q0,q1),(a[p0],b[p1]),(a[q0],b[q1]),a_conj,b_conj)
                        return False
            return True
    
        #First attempt:
        # See if matching everything at once satisfies conjugacy relations
        # (if this works, this is the best, since it considers everything)
        _, pairs = minweight_match(a, b, metricfn, True,
                                   pass_indices_to_metricfn)
    
        if check(pairs):
            return pairs  # we're done! that was easy
    
        #Otherwise we fall back to considering real values and conj pairs separately
    
        #identify real values and conjugate pairs
        def split_real_conj(ar):
            real_inds = []; conj_inds = []
            for i, v in enumerate(ar):
                if abs(v.imag) < eps: real_inds.append(i)
                else:
                    for pair in conj_inds:
                        if i in pair: break  # ok, we've already found v's pair
                    else:
                        for j, v2 in enumerate(ar[i + 1:], start=i + 1):
                            if _np.isclose(_np.conj(v), v2) and all([(j not in cpair) for cpair in conj_inds]):
                                conj_inds.append((i, j)); break
                        else:
                            raise ValueError("No conjugate pair found for %s" % str(v))
    
            # choose 'a+ib' to be representative of pair
            conj_rep_inds = [p0 if (ar[p0].imag > ar[p1].imag) else p1
                             for (p0, p1) in conj_inds]
    
            return real_inds, conj_inds, conj_rep_inds
    
        def add_conjpair(ar, conj_inds, conj_rep_inds, real_inds):
            for ii, i in enumerate(real_inds):
                for jj, j in enumerate(real_inds[ii + 1:], start=ii + 1):
                    if _np.isclose(ar[i], ar[j]):
                        conj_inds.append((i, j))
                        conj_rep_inds.append(i)
                        del real_inds[jj]; del real_inds[ii]  # note: we know jj > ii
                        return True
            return False
    
        a_real, a_conj, a_reps = split_real_conj(a)  # hold indices to a & b arrays
        b_real, b_conj, b_reps = split_real_conj(b)  # hold indices to a & b arrays
    
        while len(a_conj) > len(b_conj):  # try to add real-pair(s) to b_conj
            if not add_conjpair(b, b_conj, b_reps, b_real):
                raise ValueError(("Vectors `a` and `b` don't have the same conjugate-pair structure, "
                                  " and so they cannot be matched in a way the preserves this structure."))
        while len(b_conj) > len(a_conj):  # try to add real-pair(s) to a_conj
            if not add_conjpair(a, a_conj, a_reps, a_real):
>               raise ValueError(("Vectors `a` and `b` don't have the same conjugate-pair structure, "
                                  " and so they cannot be matched in a way the preserves this structure."))
E               ValueError: Vectors `a` and `b` don't have the same conjugate-pair structure,  and so they cannot be matched in a way the preserves this structure.

pygsti/tools/matrixtools.py:1241: ValueError
---------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------
Running tests from /Users/rjmurr/Documents/pygsti-general/pyGSTi/test/test_packages
---------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------

WARNING: Treating result as *converged* after maximum iterations (3) were exceeded.

WARNING: Treating result as *converged* after maximum iterations (3) were exceeded.

WARNING: Treating result as *converged* after maximum iterations (3) were exceeded.
=============================================================== short test summary info ===============================================================
FAILED test/test_packages/drivers/test_drivers.py::TestDriversMethods::test_longSequenceGST_CPTP - ValueError: Vectors `a` and `b` don't have the same conjugate-pair structure,  and so they cannot be matched in a way the preserves this structure.
================================================================== 1 failed in 3.39s ==================================================================

[coreyostrove]

Looks great, thanks for the fixes!

[rileyjmurray]

@coreyostrove I actually haven't finished the third failure yet

[coreyostrove]

@coreyostrove I actually haven't finished the third failure yet

Whoops! I rescind the approval in that case, my bad.

[rileyjmurray]

I'd like #517 to be merged before finishing / merging in this PR.

[sserita]

#517 is in now :)

[rileyjmurray]

Excellent. I've launched the tests again. If they pass then I'll remove the WIP prefix and be okay with a merge.

[rileyjmurray]

Tests passed! Of course, we already discussed how these test failures didn't reliably manifest on other branches, but this PR includes formally correct fixes. It's reasonable to merge as-is.

[sserita]

Just to double-check, this was setting things that are numerically 0 to 0 so that line 1741 drops it if record_zero_counts is False? If so, can we add that as a comment? I don't work with the array mask syntax much and this took me a second.

[rileyjmurray]

Yup! Comment added :)

[sserita]

Thanks! Also, just a random sidenote: This looked much more natural to me in 515 where it was just array[array < 1e-10] = 0... I don't know why having the mask split out on its own line took me by surprise 😅

[sserita]

Perfect thanks!