Merge branch 'branch-23.08' into branch-23.08-resultsset

CHANGELOG.md

@@ -1,3 +1,75 @@
+# cuGraph 23.08.00 (9 Aug 2023)
+
+## 🚨 Breaking Changes
+
+- Change the renumber_sampled_edgelist function behavior. ([#3762](https://github.com/rapidsai/cugraph/pull/3762)) [@seunghwak](https://github.com/seunghwak)
+- PLC and Python Support for Sample-Side MFG Creation ([#3734](https://github.com/rapidsai/cugraph/pull/3734)) [@alexbarghi-nv](https://github.com/alexbarghi-nv)
+- Stop using setup.py in build.sh ([#3704](https://github.com/rapidsai/cugraph/pull/3704)) [@vyasr](https://github.com/vyasr)
+- Refactor edge betweenness centrality ([#3672](https://github.com/rapidsai/cugraph/pull/3672)) [@jnke2016](https://github.com/jnke2016)
+- [FIX] Fix the hang in cuGraph Python Uniform Neighbor Sample, Add Logging to Bulk Sampler ([#3669](https://github.com/rapidsai/cugraph/pull/3669)) [@alexbarghi-nv](https://github.com/alexbarghi-nv)
+
+## 🐛 Bug Fixes
+
+- Change the renumber_sampled_edgelist function behavior. ([#3762](https://github.com/rapidsai/cugraph/pull/3762)) [@seunghwak](https://github.com/seunghwak)
+- Fix bug discovered in Jaccard testing ([#3758](https://github.com/rapidsai/cugraph/pull/3758)) [@ChuckHastings](https://github.com/ChuckHastings)
+- fix inconsistent graph properties between the SG and the MG API ([#3757](https://github.com/rapidsai/cugraph/pull/3757)) [@jnke2016](https://github.com/jnke2016)
+- Fixes options for `--pydevelop` to remove unneeded CWD path ("."), restores use of `setup.py` temporarily for develop builds ([#3747](https://github.com/rapidsai/cugraph/pull/3747)) [@rlratzel](https://github.com/rlratzel)
+- Fix sampling call parameters if compiled with -DNO_CUGRAPH_OPS ([#3729](https://github.com/rapidsai/cugraph/pull/3729)) [@ChuckHastings](https://github.com/ChuckHastings)
+- Fix primitive bug discovered in MG edge betweenness centrality testing ([#3723](https://github.com/rapidsai/cugraph/pull/3723)) [@ChuckHastings](https://github.com/ChuckHastings)
+- Reorder dependencies.yaml channels ([#3721](https://github.com/rapidsai/cugraph/pull/3721)) [@raydouglass](https://github.com/raydouglass)
+- [BUG]   Fix namesapce to default_hash and hash_functions ([#3711](https://github.com/rapidsai/cugraph/pull/3711)) [@naimnv](https://github.com/naimnv)
+- [BUG] Fix Bulk Sampling Test Issue ([#3701](https://github.com/rapidsai/cugraph/pull/3701)) [@alexbarghi-nv](https://github.com/alexbarghi-nv)
+- Make `pylibcugraphops` optional imports in `cugraph-dgl` and `-pyg` ([#3693](https://github.com/rapidsai/cugraph/pull/3693)) [@tingyu66](https://github.com/tingyu66)
+- [FIX] Rename `cugraph-ops` symbols (refactoring) and update GHA workflows to call pytest via `python -m pytest` ([#3688](https://github.com/rapidsai/cugraph/pull/3688)) [@naimnv](https://github.com/naimnv)
+- [FIX] Fix the hang in cuGraph Python Uniform Neighbor Sample, Add Logging to Bulk Sampler ([#3669](https://github.com/rapidsai/cugraph/pull/3669)) [@alexbarghi-nv](https://github.com/alexbarghi-nv)
+- force atlas notebook changes to run in cugraph 23.08 container. ([#3656](https://github.com/rapidsai/cugraph/pull/3656)) [@acostadon](https://github.com/acostadon)
+
+## 📖 Documentation
+
+- this fixes github links in cugraph, cugraph-dgl and cugraph-pyg ([#3650](https://github.com/rapidsai/cugraph/pull/3650)) [@acostadon](https://github.com/acostadon)
+- Fix minor typo in README.md ([#3636](https://github.com/rapidsai/cugraph/pull/3636)) [@akasper](https://github.com/akasper)
+- Created landing spot for centrality and similarity algorithms ([#3620](https://github.com/rapidsai/cugraph/pull/3620)) [@acostadon](https://github.com/acostadon)
+
+## 🚀 New Features
+
+- Compute shortest distances between given sets of origins and destinations for large diameter graphs ([#3741](https://github.com/rapidsai/cugraph/pull/3741)) [@seunghwak](https://github.com/seunghwak)
+- Update primitive to compute weighted Jaccard, Sorensen and Overlap similarity ([#3728](https://github.com/rapidsai/cugraph/pull/3728)) [@naimnv](https://github.com/naimnv)
+- Add CUDA 12.0 conda environment. ([#3725](https://github.com/rapidsai/cugraph/pull/3725)) [@bdice](https://github.com/bdice)
+- Renumber utility function for sampling output ([#3707](https://github.com/rapidsai/cugraph/pull/3707)) [@seunghwak](https://github.com/seunghwak)
+- Integrate C++ Sampling Source Behavior Updates ([#3699](https://github.com/rapidsai/cugraph/pull/3699)) [@alexbarghi-nv](https://github.com/alexbarghi-nv)
+- Adds `fail_on_nonconvergence` option to `pagerank` to provide pagerank results even on non-convergence ([#3639](https://github.com/rapidsai/cugraph/pull/3639)) [@rlratzel](https://github.com/rlratzel)
+- Add Benchmark for Bulk Sampling ([#3628](https://github.com/rapidsai/cugraph/pull/3628)) [@alexbarghi-nv](https://github.com/alexbarghi-nv)
+- cugraph: Build CUDA 12 packages ([#3456](https://github.com/rapidsai/cugraph/pull/3456)) [@vyasr](https://github.com/vyasr)
+
+## 🛠️ Improvements
+
+- Pin `dask` and `distributed` for `23.08` release ([#3761](https://github.com/rapidsai/cugraph/pull/3761)) [@galipremsagar](https://github.com/galipremsagar)
+- Fix `build.yaml` workflow ([#3756](https://github.com/rapidsai/cugraph/pull/3756)) [@ajschmidt8](https://github.com/ajschmidt8)
+- Support MFG creation on sampling gpus for cugraph dgl ([#3742](https://github.com/rapidsai/cugraph/pull/3742)) [@VibhuJawa](https://github.com/VibhuJawa)
+- PLC and Python Support for Sample-Side MFG Creation ([#3734](https://github.com/rapidsai/cugraph/pull/3734)) [@alexbarghi-nv](https://github.com/alexbarghi-nv)
+- Switch to new wheel building pipeline ([#3731](https://github.com/rapidsai/cugraph/pull/3731)) [@vyasr](https://github.com/vyasr)
+- Remove RAFT specialization. ([#3727](https://github.com/rapidsai/cugraph/pull/3727)) [@bdice](https://github.com/bdice)
+- C API for renumbering the samples ([#3724](https://github.com/rapidsai/cugraph/pull/3724)) [@ChuckHastings](https://github.com/ChuckHastings)
+- Only run cugraph conda CI for CUDA 11. ([#3713](https://github.com/rapidsai/cugraph/pull/3713)) [@bdice](https://github.com/bdice)
+- Promote `Datasets` to stable and clean-up unit tests ([#3712](https://github.com/rapidsai/cugraph/pull/3712)) [@nv-rliu](https://github.com/nv-rliu)
+- [BUG] Unsupported graph for similiarity algos ([#3710](https://github.com/rapidsai/cugraph/pull/3710)) [@jnke2016](https://github.com/jnke2016)
+- Stop using setup.py in build.sh ([#3704](https://github.com/rapidsai/cugraph/pull/3704)) [@vyasr](https://github.com/vyasr)
+- [WIP] Make edge ids optional ([#3702](https://github.com/rapidsai/cugraph/pull/3702)) [@VibhuJawa](https://github.com/VibhuJawa)
+- Use rapids-cmake testing to run tests in parallel ([#3697](https://github.com/rapidsai/cugraph/pull/3697)) [@robertmaynard](https://github.com/robertmaynard)
+- Sampling modifications to support PyG and DGL options ([#3696](https://github.com/rapidsai/cugraph/pull/3696)) [@ChuckHastings](https://github.com/ChuckHastings)
+- Include cuCollection public header for hash functions ([#3694](https://github.com/rapidsai/cugraph/pull/3694)) [@seunghwak](https://github.com/seunghwak)
+- Refactor edge betweenness centrality ([#3672](https://github.com/rapidsai/cugraph/pull/3672)) [@jnke2016](https://github.com/jnke2016)
+- Refactor RMAT ([#3662](https://github.com/rapidsai/cugraph/pull/3662)) [@jnke2016](https://github.com/jnke2016)
+- [REVIEW] Optimize bulk sampling ([#3661](https://github.com/rapidsai/cugraph/pull/3661)) [@VibhuJawa](https://github.com/VibhuJawa)
+- Update to CMake 3.26.4 ([#3648](https://github.com/rapidsai/cugraph/pull/3648)) [@vyasr](https://github.com/vyasr)
+- Optimize cugraph-dgl MFG creation ([#3646](https://github.com/rapidsai/cugraph/pull/3646)) [@VibhuJawa](https://github.com/VibhuJawa)
+- use rapids-upload-docs script ([#3640](https://github.com/rapidsai/cugraph/pull/3640)) [@AyodeAwe](https://github.com/AyodeAwe)
+- Fix dependency versions for `23.08` ([#3638](https://github.com/rapidsai/cugraph/pull/3638)) [@ajschmidt8](https://github.com/ajschmidt8)
+- Unpin `dask` and `distributed` for development ([#3634](https://github.com/rapidsai/cugraph/pull/3634)) [@galipremsagar](https://github.com/galipremsagar)
+- Remove documentation build scripts for Jenkins ([#3627](https://github.com/rapidsai/cugraph/pull/3627)) [@ajschmidt8](https://github.com/ajschmidt8)
+- Unpin scikit-build upper bound ([#3609](https://github.com/rapidsai/cugraph/pull/3609)) [@vyasr](https://github.com/vyasr)
+- Implement C++ Edge Betweenness Centrality ([#3602](https://github.com/rapidsai/cugraph/pull/3602)) [@ChuckHastings](https://github.com/ChuckHastings)
+
 # cuGraph 23.06.00 (7 Jun 2023)
 
 ## 🚨 Breaking Changes

python/cugraph/cugraph/structure/graph_classes.py

@@ -68,11 +68,14 @@ def __init__(self, m_graph=None, directed=False):
             if isinstance(m_graph, MultiGraph):
                 elist = m_graph.view_edge_list()
                 if m_graph.is_weighted():
-                    weights = "weights"
+                    weights = m_graph.weight_column
                 else:
                     weights = None
                 self.from_cudf_edgelist(
-                    elist, source="src", destination="dst", edge_attr=weights
+                    elist,
+                    source=m_graph.source_columns,
+                    destination=m_graph.destination_columns,
+                    edge_attr=weights,
                 )
             else:
                 raise TypeError(

python/cugraph/cugraph/structure/graph_implementation/simpleDistributedGraph.py

@@ -79,6 +79,8 @@ def __init__(self, properties):
         self.properties = simpleDistributedGraphImpl.Properties(properties)
         self.source_columns = None
         self.destination_columns = None
+        self.weight_column = None
+        self.vertex_columns = None
 
     def _make_plc_graph(
         sID,
@@ -175,6 +177,7 @@ def __from_edgelist(
                 "and destination parameters"
             )
         ddf_columns = s_col + d_col
+        self.vertex_columns = ddf_columns.copy()
         _client = default_client()
         workers = _client.scheduler_info()["workers"]
         # Repartition to 2 partitions per GPU for memory efficient process
@@ -214,10 +217,11 @@ def __from_edgelist(
             # The symmetrize step may add additional edges with unknown
             # ids and types for an undirected graph.  Therefore, only
             # directed graphs may be used with ids and types.
+            # FIXME: Drop the check in symmetrize.py as it is redundant
             if len(edge_attr) == 3:
                 if not self.properties.directed:
                     raise ValueError(
-                        "User-provided edge ids and edge "
+                        "User-provided edge ids and/or edge "
                         "types are not permitted for an "
                         "undirected graph."
                     )
@@ -285,6 +289,7 @@ def __from_edgelist(
         self.properties.renumber = renumber
         self.source_columns = source
         self.destination_columns = destination
+        self.weight_column = weight
 
         # If renumbering is not enabled, this function will only create
         # the edgelist_df and not do any renumbering.
@@ -316,7 +321,6 @@ def __from_edgelist(
         ddf = ddf.map_partitions(lambda df: df.copy())
         ddf = persist_dask_df_equal_parts_per_worker(ddf, _client)
         num_edges = len(ddf)
-        self._number_of_edges = num_edges
         ddf = get_persisted_df_worker_map(ddf, _client)
         delayed_tasks_d = {
             w: delayed(simpleDistributedGraphImpl._make_plc_graph)(
@@ -356,6 +360,8 @@ def renumbered(self):
 
     def view_edge_list(self):
         """
+        FIXME: Should this also return the edge ids and types?
+
         Display the edge list. Compute it if needed.
         NOTE: If the graph is of type Graph() then the displayed undirected
         edges are the same as displayed by networkx Graph(), but the direction
@@ -386,7 +392,59 @@ def view_edge_list(self):
         """
         if self.edgelist is None:
             raise RuntimeError("Graph has no Edgelist.")
-        return self.edgelist.edgelist_df
+
+        edgelist_df = self.input_df
+        is_string_dtype = False
+        is_multi_column = False
+        wgtCol = simpleDistributedGraphImpl.edgeWeightCol
+        if not self.properties.directed:
+            srcCol = self.source_columns
+            dstCol = self.destination_columns
+            if self.renumber_map.unrenumbered_id_type == "object":
+                # FIXME: Use the renumbered vertices instead and then un-renumber.
+                # This operation can be expensive.
+                is_string_dtype = True
+                edgelist_df = self.edgelist.edgelist_df
+                srcCol = self.renumber_map.renumbered_src_col_name
+                dstCol = self.renumber_map.renumbered_dst_col_name
+
+            if isinstance(srcCol, list):
+                srcCol = self.renumber_map.renumbered_src_col_name
+                dstCol = self.renumber_map.renumbered_dst_col_name
+                edgelist_df = self.edgelist.edgelist_df
+                # unrenumber before extracting the upper triangular part
+                if len(self.source_columns) == 1:
+                    edgelist_df = self.renumber_map.unrenumber(edgelist_df, srcCol)
+                    edgelist_df = self.renumber_map.unrenumber(edgelist_df, dstCol)
+                else:
+                    is_multi_column = True
+
+            edgelist_df[srcCol], edgelist_df[dstCol] = edgelist_df[
+                [srcCol, dstCol]
+            ].min(axis=1), edgelist_df[[srcCol, dstCol]].max(axis=1)
+
+            edgelist_df = edgelist_df.groupby(by=[srcCol, dstCol]).sum().reset_index()
+            if wgtCol in edgelist_df.columns:
+                # FIXME: This breaks if there are are multi edges as those will
+                # be dropped during the symmetrization step and the original 'weight'
+                # will be halved.
+                edgelist_df[wgtCol] /= 2
+
+        if is_string_dtype or is_multi_column:
+            # unrenumber the vertices
+            edgelist_df = self.renumber_map.unrenumber(edgelist_df, srcCol)
+            edgelist_df = self.renumber_map.unrenumber(edgelist_df, dstCol)
+
+        if self.properties.renumbered:
+            edgelist_df = edgelist_df.rename(
+                columns=self.renumber_map.internal_to_external_col_names
+            )
+
+        # If there is no 'wgt' column, nothing will happen
+        edgelist_df = edgelist_df.rename(columns={wgtCol: self.weight_column})
+
+        self.properties.edge_count = len(edgelist_df)
+        return edgelist_df
 
     def delete_edge_list(self):
         """
@@ -405,23 +463,7 @@ def number_of_vertices(self):
         Get the number of nodes in the graph.
         """
         if self.properties.node_count is None:
-            if self.edgelist is not None:
-                if self.renumbered is True:
-                    src_col_name = self.renumber_map.renumbered_src_col_name
-                    dst_col_name = self.renumber_map.renumbered_dst_col_name
-                # FIXME: from_dask_cudf_edgelist() currently requires
-                # renumber=True for MG, so this else block will not be
-                # used. Should this else block be removed and added back when
-                # the restriction is removed?
-                else:
-                    src_col_name = "src"
-                    dst_col_name = "dst"
-
-                ddf = self.edgelist.edgelist_df[[src_col_name, dst_col_name]]
-                # ddf = self.edgelist.edgelist_df[["src", "dst"]]
-                self.properties.node_count = ddf.max().max().compute() + 1
-            else:
-                raise RuntimeError("Graph is Empty")
+            self.properties.node_count = len(self.nodes())
         return self.properties.node_count
 
     def number_of_nodes(self):
@@ -434,10 +476,16 @@ def number_of_edges(self, directed_edges=False):
         """
         Get the number of edges in the graph.
         """
-        if self.edgelist is not None:
-            return self._number_of_edges
-        else:
-            raise RuntimeError("Graph is Empty")
+
+        if directed_edges and self.edgelist is not None:
+            return len(self.edgelist.edgelist_df)
+
+        if self.properties.edge_count is None:
+            if self.edgelist is not None:
+                self.view_edge_list()
+            else:
+                raise RuntimeError("Graph is Empty")
+        return self.properties.edge_count
 
     def in_degree(self, vertex_subset=None):
         """
@@ -1021,19 +1069,8 @@ def edges(self):
         sources and destinations. It does not return the edge weights.
         For viewing edges with weights use view_edge_list()
         """
-        if self.renumbered is True:
-            src_col_name = self.renumber_map.renumbered_src_col_name
-            dst_col_name = self.renumber_map.renumbered_dst_col_name
-            # FIXME: from_dask_cudf_edgelist() currently requires
-            # renumber=True for MG, so this else block will not be
-            # used. Should this else block be removed and added back when
-            # the restriction is removed?
-        else:
-            src_col_name = "src"
-            dst_col_name = "dst"
 
-        # return self.view_edge_list()[["src", "dst"]]
-        return self.view_edge_list()[[src_col_name, dst_col_name]]
+        return self.view_edge_list()[self.vertex_columns]
 
     def nodes(self):
         """
@@ -1045,23 +1082,26 @@ def nodes(self):
         a dataframe and do 'renumber_map.unrenumber' or 'G.unrenumber'
         """
 
-        if self.renumbered:
-            # FIXME: This relies on current implementation
-            #        of NumberMap, should not really expose
-            #        this, perhaps add a method to NumberMap
+        if self.edgelist is not None:
+            if self.renumbered:
+                # FIXME: This relies on current implementation
+                #        of NumberMap, should not really expose
+                #        this, perhaps add a method to NumberMap
 
-            df = self.renumber_map.implementation.ddf.drop(columns="global_id")
+                df = self.renumber_map.implementation.ddf.drop(columns="global_id")
 
-            if len(df.columns) > 1:
-                return df
-            else:
-                return df[df.columns[0]]
+                if len(df.columns) > 1:
+                    return df
+                else:
+                    return df[df.columns[0]]
 
+            else:
+                df = self.input_df
+                return dask_cudf.concat(
+                    [df[self.source_columns], df[self.destination_columns]]
+                ).drop_duplicates()
         else:
-            df = self.input_df
-            return dask_cudf.concat(
-                [df[self.source_columns], df[self.destination_columns]]
-            ).drop_duplicates()
+            raise RuntimeError("Graph is Empty")
 
     def neighbors(self, n):
         if self.edgelist is None: