Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error in snakemamke Plot_changes? #14

Open
c2b2pss opened this issue May 15, 2024 · 8 comments
Open

Error in snakemamke Plot_changes? #14

c2b2pss opened this issue May 15, 2024 · 8 comments

Comments

@c2b2pss
Copy link

c2b2pss commented May 15, 2024 •
edited
Loading

Hi,

The snakemake pipeline said 99.9% complete but failed on this:

Activating conda environment: .snakemake/conda/b64f8caf
[Tue May 14 17:56:01 2024]
Error in rule lineplot_changes:
    jobid: 36
    input: /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/TFBS/bindetect_results.txt
    output: /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/overview/bindetect_results_subset.txt, /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/overview/most_changed_TFs.txt, /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/overview/TF_changes.pdf
    log: /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/logs/TF_changes.log (check log file(s) for error details)
    conda-env: /home/premsubramaniam/TOBIAS_snakemake/.snakemake/conda/b64f8caf
    shell:
        filter_important_factors.py -in /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/TFBS/bindetect_results.txt -filter 5 -o /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/overview/bindetect_results_subset.txt > /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/overview/most_changed_TFs.txt;TOBIAS PlotChanges --bindetect /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/TFBS/bindetect_results.txt --TFS /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/overview/most_changed_TFs.txt --output /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/overview/TF_changes.pdf > /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/logs/TF_changes.log
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Removing output files of failed job lineplot_changes since they might be corrupted:
/home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/overview/bindetect_results_subset.txt, /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/overview/most_changed_TFs.txt, /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/overview/TF_changes.pdf
Select jobs to execute...

[Tue May 14 17:56:02 2024]
Job 2259: Plotting split between bound/unbound around TFBS for TF "PROP1_HUMAN.H11MO.0.D_PROP1_HUMAN.H11MO.0.D" in condition "PR"
Reason: Missing output files: /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/TFBS/PROP1_HUMAN.H11MO.0.D_PROP1_HUMAN.H11MO.0.D/plots/PROP1_HUMAN.H11MO.0.D_PROP1_HUMAN.H11MO.0.D_PR_aggregate.pdf

Activating conda environment: .snakemake/conda/b64f8caf
[Tue May 14 17:56:07 2024]
Finished job 4782.
1 of 2399 steps (0.0%) done
Removing temporary output /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/TFBS/WT1_HUMAN.H11MO.0.C_WT1_HUMAN.H11MO.0.C/plots/Intervene_venn.pdf.

Then I tried to run TOBIAS Plot_changes manually, so I ran this command first:

filter_important_factors.py -in /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/TFBS/bindetect_results.txt -filter 5 -o /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/overview/bindetect_results_subset.txt > /home/premsubramaniam/TOBIAS_snakemake/LNCAP_ALL/overview/most_changed_TFs.txt

That worked, but the next TOBIAS PlotChanges failed with the following:


TOBIAS PlotChanges --bindetect /home/user/TOBIAS_snakemake/LNCAP_ALL/TFBS/bindetect_results.txt --TFS /home/user/TOBIAS_snakemake/LNCAP_ALL/overview/most_changed_TFs.txt --output /home/user/TOBIAS_snakemake/LNCAP_ALL/overview/TF_changes.pdf > /home/user/TOBIAS_snakemake/LNCAP_ALL/logs/TF_changes.log
Traceback (most recent call last):
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1490, in array_func
    result = self.grouper._cython_operation(
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 959, in _cython_operation
    return cy_op.cython_operation(
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 657, in cython_operation
    return self._cython_op_ndim_compat(
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 497, in _cython_op_ndim_compat
    return self._call_cython_op(
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 541, in _call_cython_op
    func = self._get_cython_function(self.kind, self.how, values.dtype, is_numeric)
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 173, in _get_cython_function
    raise NotImplementedError(
NotImplementedError: function is not implemented for this dtype: [how->mean,dtype->object]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/nanops.py", line 1692, in _ensure_numeric
    x = float(x)
ValueError: could not convert string to float: 'ANDR_HUMAN.H11MO.1.A_ANDR_HUMAN.H11MO.1.APRGR_HUMAN.H11MO.0.A_PRGR_HUMAN.H11MO.0.AGCR_HUMAN.H11MO.0.A_GCR_HUMAN.H11MO.0.A'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/nanops.py", line 1696, in _ensure_numeric
    x = complex(x)
ValueError: could not convert string to complex: 'ANDR_HUMAN.H11MO.1.A_ANDR_HUMAN.H11MO.1.APRGR_HUMAN.H11MO.0.A_PRGR_HUMAN.H11MO.0.AGCR_HUMAN.H11MO.0.A_GCR_HUMAN.H11MO.0.A'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/bin/TOBIAS", line 8, in <module>
    sys.exit(main())
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/tobias/TOBIAS.py", line 154, in main
    args.func(args)		
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/tobias/tools/plot_changes.py", line 94, in run_plotchanges
    table = bindetect.loc[chosen_TFS,].groupby("cluster").mean() #mean of each column
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1855, in mean
    result = self._cython_agg_general(
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1507, in _cython_agg_general
    new_mgr = data.grouped_reduce(array_func)
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 1503, in grouped_reduce
    applied = sb.apply(func)
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/internals/blocks.py", line 329, in apply
    result = func(self.values, **kwargs)
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1503, in array_func
    result = self._agg_py_fallback(values, ndim=data.ndim, alt=alt)
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1457, in _agg_py_fallback
    res_values = self.grouper.agg_series(ser, alt, preserve_dtype=True)
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 994, in agg_series
    result = self._aggregate_series_pure_python(obj, func)
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/groupby/ops.py", line 1015, in _aggregate_series_pure_python
    res = func(group)
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/groupby/groupby.py", line 1857, in <lambda>
    alt=lambda x: Series(x).mean(numeric_only=numeric_only),
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/generic.py", line 11556, in mean
    return NDFrame.mean(self, axis, skipna, numeric_only, **kwargs)
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/generic.py", line 11201, in mean
    return self._stat_function(
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/generic.py", line 11158, in _stat_function
    return self._reduce(
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/series.py", line 4666, in _reduce
    return op(delegate, skipna=skipna, **kwds)
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/nanops.py", line 96, in _f
    return f(*args, **kwargs)
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/nanops.py", line 158, in f
    result = alt(values, axis=axis, skipna=skipna, **kwds)
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/nanops.py", line 421, in new_func
    result = func(values, axis=axis, skipna=skipna, mask=mask, **kwargs)
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/nanops.py", line 727, in nanmean
    the_sum = _ensure_numeric(values.sum(axis, dtype=dtype_sum))
  File "/home/user/TOBIAS_snakemake/.snakemake/conda/b64f8caf/lib/python3.10/site-packages/pandas/core/nanops.py", line 1699, in _ensure_numeric
    raise TypeError(f"Could not convert {x} to numeric") from err
TypeError: Could not convert ANDR_HUMAN.H11MO.1.A_ANDR_HUMAN.H11MO.1.APRGR_HUMAN.H11MO.0.A_PRGR_HUMAN.H11MO.0.AGCR_HUMAN.H11MO.0.A_GCR_HUMAN.H11MO.0.A to numeric

Your help is greatly appreciated!
@mohobein
Copy link
Collaborator

Hey @c2b2pss,

thank you for providing the error messages, that is quite helpful.
My first guess is that there might be a problem with formatting of the TF names in the input files, though this does not explain why the pipeline did not correctly manage that by itself for this last step only.

Could you please provide me with your most_changed_TFs.txt file as well as your bindetect_results.txt (or at least a few lines of each, should that is not possible). I'd like to try to replicate the problem to be able to investigate further.

Best regards,
Moritz

@c2b2pss
Copy link
Author

c2b2pss commented May 15, 2024 •
edited
Loading

Hi Moritz,

Thanks for looking into this. Yes, it errors only on these files.

Here are the files:
bindetect_results.txt
most_changed_TFs.txt
TF_changes.pdf

PS: A follow up question is how does one reformat the volcano plots to ones needs? So which file is the volcano plot derived from?

@mohobein
Copy link
Collaborator

Hm, that is weird. When I run PlotChanges with your files (TOBIAS PlotChanges --bindetect bindetect_results.txt --TFS most_changed_TFs.txt), it works perfectly fine.
Here is the output file: bindetect_changes.pdf.

Can you please tell me which version of TOBIAS you are using (TOBIAS --version) and perhaps show the overview over the package versions in your TOBIAS environment used by the pipeline/by you manually (conda list -n <ENV_NAME>).

Regarding your follow up question:
All information needed to create the volcano plot is in the bindetect_results.txt file. For each pairwise comparison between conditions, there is a *_change and a *_pvalue column containing the values necessary for the plot. If you want to take a look at how TOBIAS creates the plot, these sections of code might be of interest to you: data acquisition, plot preprocessing and plot generation. You may want to start from there and make changes to suit your own style.

@c2b2pss
Copy link
Author

c2b2pss commented May 16, 2024

Thanks for the reply and the results file!

  1. TOBIAS ver: 0.13.3 -- I did not upgrade or anything since it was working and didn't want to jinx it. :-)

  2. I would like to get more than just 15 TFs in the bindetect_changes results -- is this set by P-value or is is a fixed number of TFS?

  3. Environment:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_py-xgboost-mutex         2.0                       cpu_0    conda-forge
adjusttext                0.7.3.1                    py_1    conda-forge
aioeasywebdav             2.4.0           py37h89c1867_1001    conda-forge
aiohttp                   3.7.4.post0      py37h5e8e339_1    conda-forge
alsa-lib                  1.2.8                h166bdaf_0    conda-forge
amply                     0.1.5              pyhd8ed1ab_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
async-timeout             3.0.1                   py_1000    conda-forge
atk-1.0                   2.38.0               hd4edc92_1    conda-forge
attmap                    0.13.2             pyhd8ed1ab_0    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
attrs                     22.2.0             pyh71513ae_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
bcrypt                    3.2.2            py37h540881e_0    conda-forge
bedtools                  2.30.0               h468198e_3    bioconda
biopython                 1.79             py37h540881e_2    conda-forge
boto3                     1.26.119           pyhd8ed1ab_0    conda-forge
botocore                  1.29.119           pyhd8ed1ab_0    conda-forge
bottleneck                1.3.5            py37hda87dfa_0    conda-forge
brotli                    1.0.9                h166bdaf_8    conda-forge
brotli-bin                1.0.9                h166bdaf_8    conda-forge
brotlipy                  0.7.0           py37h540881e_1004    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.12.7            ha878542_0    conda-forge
cachetools                5.3.0              pyhd8ed1ab_0    conda-forge
cairo                     1.16.0            ha61ee94_1014    conda-forge
certifi                   2022.12.7          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1           py37h43b0acd_1    conda-forge
chardet                   4.0.0            py37h89c1867_3    conda-forge
charset-normalizer        3.1.0              pyhd8ed1ab_0    conda-forge
coin-or-cbc               2.10.10              h9002f0b_0    conda-forge
coin-or-cgl               0.60.7               h516709c_0    conda-forge
coin-or-clp               1.17.8               h1ee7a9c_0    conda-forge
coin-or-osi               0.108.8              ha2443b9_0    conda-forge
coin-or-utils             2.11.9               hee58242_0    conda-forge
coincbc                   2.10.10           0_metapackage    conda-forge
conda                     22.9.0           py37h89c1867_1    conda-forge
conda-package-handling    2.0.2              pyh38be061_0    conda-forge
conda-package-streaming   0.7.0              pyhd8ed1ab_1    conda-forge
configargparse            1.5.3              pyhd8ed1ab_0    conda-forge
configs                   3.0.3                      py_0    conda-forge
connection_pool           0.0.3              pyhd3deb0d_0    conda-forge
cryptography              38.0.2           py37h38fbfac_1    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
datrie                    0.8.2            py37h540881e_4    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
docutils                  0.19             py37h89c1867_0    conda-forge
dpath                     2.0.6            py37h89c1867_1    conda-forge
dropbox                   11.36.0            pyhd8ed1ab_0    conda-forge
exceptiongroup            1.1.1              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
fftw                      3.3.10          nompi_hc118613_107    conda-forge
filechunkio               1.8                        py_2    conda-forge
filelock                  3.12.0             pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.38.0           py37h540881e_0    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
fribidi                   1.0.10               h36c2ea0_0    conda-forge
ftputil                   5.0.4              pyhd8ed1ab_0    conda-forge
gdk-pixbuf                2.42.8               hff1cb4f_1    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
ghostscript               9.54.0               h27087fc_2    conda-forge
giflib                    5.2.1                h0b41bf4_3    conda-forge
gitdb                     4.0.10             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.31             pyhd8ed1ab_0    conda-forge
glib                      2.76.2               hfc55251_0    conda-forge
glib-tools                2.76.2               hfc55251_0    conda-forge
google-api-core           2.11.0             pyhd8ed1ab_0    conda-forge
google-api-python-client  2.86.0             pyhd8ed1ab_0    conda-forge
google-auth               2.17.3             pyh1a96a4e_0    conda-forge
google-auth-httplib2      0.1.0              pyhd8ed1ab_1    conda-forge
google-cloud-core         2.3.2              pyhd8ed1ab_0    conda-forge
google-cloud-storage      2.8.0              pyh1a96a4e_0    conda-forge
google-crc32c             1.1.2            py37h5d4fa31_3    conda-forge
google-resumable-media    2.5.0              pyhd8ed1ab_0    conda-forge
googleapis-common-protos  1.57.1             pyhd8ed1ab_0    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
graphviz                  6.0.2                h99bc08f_0    conda-forge
grpc-cpp                  1.48.1               h05bd8bd_1    conda-forge
grpcio                    1.48.1           py37h3fae8ff_1    conda-forge
gst-plugins-base          1.21.3               h4243ec0_1    conda-forge
gstreamer                 1.21.3               h25f0c4b_1    conda-forge
gstreamer-orc             0.4.33               h166bdaf_0    conda-forge
gtk2                      2.24.33              h90689f9_2    conda-forge
gts                       0.7.6                h64030ff_2    conda-forge
harfbuzz                  6.0.0                h8e241bc_0    conda-forge
htslib                    1.17                 h6bc39ce_0    bioconda
httplib2                  0.22.0             pyhd8ed1ab_0    conda-forge
humanfriendly             10.0             py37h89c1867_2    conda-forge
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
igvtools                  2.14.1               hdfd78af_0    bioconda
imagemagick               7.1.0_55        pl5321h0d24a18_0    conda-forge
importlib-metadata        4.11.4           py37h89c1867_0    conda-forge
importlib_metadata        4.11.4               hd8ed1ab_0    conda-forge
importlib_resources       5.12.0             pyhd8ed1ab_0    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
jack                      1.9.22               h11f4161_0    conda-forge
jbig                      2.1               h7f98852_2003    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h0b41bf4_3    conda-forge
jsonschema                4.17.3             pyhd8ed1ab_0    conda-forge
jupyter_core              4.11.1           py37h89c1867_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4            py37h7cecad7_0    conda-forge
kneed                     0.8.2              pyhd8ed1ab_0    conda-forge
krb5                      1.20.1               hf9c8cef_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.14                 h6ed2654_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20220623.0      cxx17_h05df665_6    conda-forge
libarchive                3.5.2                hb890918_3    conda-forge
libblas                   3.9.0           16_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_8    conda-forge
libbrotlidec              1.0.9                h166bdaf_8    conda-forge
libbrotlienc              1.0.9                h166bdaf_8    conda-forge
libcap                    2.66                 ha37c62d_0    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libclang                  15.0.7          default_had23c3d_1    conda-forge
libclang13                15.0.7          default_h3e3d535_1    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcups                   2.3.3                h36d4200_3    conda-forge
libcurl                   7.87.0               h6312ad2_0    conda-forge
libdb                     6.2.32               h9c3ff4c_0    conda-forge
libdeflate                1.13                 h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.4.2                h27087fc_0    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgcrypt                 1.10.1               h166bdaf_0    conda-forge
libgd                     2.3.3                h18fbbfe_3    conda-forge
libgfortran-ng            12.2.0              h69a702a_19    conda-forge
libgfortran5              12.2.0              h337968e_19    conda-forge
libglib                   2.76.2               hebfc3b9_0    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libgpg-error              1.46                 h620e276_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
liblapacke                3.9.0           16_linux64_openblas    conda-forge
libllvm15                 15.0.7               hadd5161_1    conda-forge
libmamba                  0.27.0               h0dd8ff0_0    conda-forge
libmambapy                0.27.0           py37h1ee4b26_0    conda-forge
libnghttp2                1.51.0               hdcd2b5c_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     15.1                 h2baec63_3    conda-forge
libprotobuf               3.21.8               h6239696_0    conda-forge
librsvg                   2.54.4               h7abd40a_0    conda-forge
libsndfile                1.2.0                hb75c966_0    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsolv                   0.7.23               h3eb15da_0    conda-forge
libsqlite                 3.40.0               h753d276_1    conda-forge
libssh2                   1.10.0               haa6b8db_3    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libsystemd0               252                  h2a991cd_0    conda-forge
libtiff                   4.4.0                h0e0dad5_3    conda-forge
libtool                   2.4.7                h27087fc_0    conda-forge
libudev1                  253                  h0b41bf4_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp                   1.2.4                h522a892_0    conda-forge
libwebp-base              1.2.4                h166bdaf_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxgboost                1.6.2            cpu_ha3b9936_1    conda-forge
libxkbcommon              1.5.0                h79f4944_1    conda-forge
libxml2                   2.10.3               hca2bb57_4    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
logmuse                   0.2.6              pyh8c360ce_0    conda-forge
logomaker                 0.8                pyh864c0ab_1    bioconda
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mamba                     0.27.0           py37h6dacc13_0    conda-forge
markdown-it-py            2.2.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.1            py37h540881e_1    conda-forge
matplotlib                3.5.3            py37h89c1867_2    conda-forge
matplotlib-base           3.5.3            py37hf395dca_2    conda-forge
mdurl                     0.1.0              pyhd8ed1ab_0    conda-forge
moods                     1.9.4.1          py37h675a0cb_2    bioconda
mpg123                    1.31.3               hcb278e6_0    conda-forge
multidict                 6.0.2            py37h540881e_1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.32               h14678bc_0    conda-forge
mysql-libs                8.0.32               h54cf53e_0    conda-forge
nbformat                  5.8.0              pyhd8ed1ab_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
nomkl                     1.0                  h5ca1d4c_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.89                 he45b914_0    conda-forge
numexpr                   2.8.3           py37h85a3170_100    conda-forge
numpy                     1.21.6           py37h976b520_0    conda-forge
oauth2client              4.1.3                      py_0    conda-forge
openjdk                   17.0.3               h58dac75_5    conda-forge
openjpeg                  2.5.0                h7d73246_1    conda-forge
openssl                   1.1.1t               h0b41bf4_0    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pandas                    1.3.5            py37h8c16a72_0  
pango                     1.50.14              hd33c08f_0    conda-forge
paramiko                  3.1.0              pyhd8ed1ab_0    conda-forge
patsy                     0.5.3              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
peppy                     0.35.5             pyhd8ed1ab_0    conda-forge
perl                      5.32.1          2_h7f98852_perl5    conda-forge
pillow                    9.2.0            py37h850a105_2    conda-forge
pip                       23.1.1             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
pkg-config                0.29.2            h36c2ea0_1008    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
plac                      1.3.5              pyhd8ed1ab_0    conda-forge
pluggy                    1.0.0            py37h89c1867_3    conda-forge
ply                       3.11                       py_1    conda-forge
prettytable               3.7.0              pyhd8ed1ab_0    conda-forge
protobuf                  4.21.8           py37hd23a5d3_0    conda-forge
psutil                    5.9.3            py37h540881e_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pulp                      2.6.0            py37h89c1867_1    conda-forge
pulseaudio                16.1                 h4ab2085_1    conda-forge
py-xgboost                1.6.2           cpu_py37h08536eb_1    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pybedtools                0.9.0            py37h595c7a6_1    bioconda
pybigwig                  0.3.18           py37hdc12a6d_2    bioconda
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.4            py37h540881e_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.15.1             pyhd8ed1ab_0    conda-forge
pynacl                    1.5.0            py37h540881e_1    conda-forge
pyopenssl                 23.1.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pypdf2                    2.11.1             pyhd8ed1ab_0    conda-forge
pyqt                      5.15.7           py37hf30b843_1    conda-forge
pyqt5-sip                 12.11.0          py37hd23a5d3_1    conda-forge
pyrsistent                0.18.1           py37h540881e_1    conda-forge
pysam                     0.21.0           py37hee149a5_0    bioconda
pysftp                    0.2.9                      py_1    conda-forge
pysocks                   1.7.1            py37h89c1867_5    conda-forge
pytest                    7.2.0            py37h89c1867_0    conda-forge
python                    3.7.12          hb7a2778_100_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.16.3             pyhd8ed1ab_0    conda-forge
python-irodsclient        1.1.6              pyhd8ed1ab_0    conda-forge
python_abi                3.7                     3_cp37m    conda-forge
pytz                      2023.3             pyhd8ed1ab_0    conda-forge
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
pyyaml                    6.0              py37h540881e_4    conda-forge
qt-main                   5.15.6               h18908ee_6    conda-forge
re2                       2022.06.01           h27087fc_1    conda-forge
readline                  8.2                  h8228510_1    conda-forge
reportlab                 3.5.68           py37h69800bb_1    conda-forge
reproc                    14.2.4               h0b41bf4_0    conda-forge
reproc-cpp                14.2.4               hcb278e6_0    conda-forge
requests                  2.28.2             pyhd8ed1ab_1    conda-forge
reretry                   0.11.8             pyhd8ed1ab_0    conda-forge
rich                      13.3.4             pyhd8ed1ab_0    conda-forge
rsa                       4.9                pyhd8ed1ab_0    conda-forge
ruamel_yaml               0.15.80         py37h540881e_1007    conda-forge
s3transfer                0.6.0              pyhd8ed1ab_0    conda-forge
samtools                  1.17                 h00cdaf9_0    bioconda
scikit-learn              1.0.2            py37hf9e9bfc_0    conda-forge
scipy                     1.7.3            py37hf2a6cf1_0    conda-forge
seaborn                   0.12.2               hd8ed1ab_0    conda-forge
seaborn-base              0.12.2             pyhd8ed1ab_0    conda-forge
setuptools                67.7.2             pyhd8ed1ab_0    conda-forge
sip                       6.7.2            py37hd23a5d3_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
slacker                   0.14.0                     py_0    conda-forge
smart_open                6.3.0              pyhd8ed1ab_1    conda-forge
smmap                     3.0.5              pyh44b312d_0    conda-forge
snakemake                 7.24.0               hdfd78af_1    bioconda
snakemake-minimal         7.24.0             pyhdfd78af_0    bioconda
sqlite                    3.40.0               h4ff8645_1    conda-forge
statsmodels               0.13.2           py37hda87dfa_0    conda-forge
stone                     3.3.1              pyhd8ed1ab_0    conda-forge
stopit                    1.1.2                      py_0    conda-forge
svist4get                 1.3.1              pyhdfd78af_0    bioconda
swig                      4.1.1                h5d7ef5f_0    conda-forge
tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
throttler                 1.2.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tobias                    0.13.3           py37h37892f8_0    bioconda
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
toposort                  1.7                pyhd8ed1ab_0    conda-forge
tornado                   6.2              py37h540881e_0    conda-forge
traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.5.0                hd8ed1ab_0    conda-forge
typing_extensions         4.5.0              pyha770c72_0    conda-forge
ubiquerg                  0.6.2              pyhd8ed1ab_0    conda-forge
unicodedata2              14.0.0           py37h540881e_1    conda-forge
uritemplate               4.1.1              pyhd8ed1ab_0    conda-forge
urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
veracitools               0.1.3                      py_0    conda-forge
wand                      0.6.10             pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
wrapt                     1.14.1           py37h540881e_0    conda-forge
xcb-util                  0.4.0                h516909a_0    conda-forge
xcb-util-image            0.4.0                h166bdaf_0    conda-forge
xcb-util-keysyms          0.4.0                h516909a_0    conda-forge
xcb-util-renderutil       0.3.9                h166bdaf_0    conda-forge
xcb-util-wm               0.4.1                h516909a_0    conda-forge
xgboost                   1.6.2           cpu_py37h08536eb_1    conda-forge
xkeyboard-config          2.38                 h0b41bf4_0    conda-forge
xlsxwriter                3.0.9              pyhd8ed1ab_0    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-inputproto           2.3.2             h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.8.4                h0b41bf4_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-libxi                1.7.10               h7f98852_0    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-libxt                1.2.1                h7f98852_2    conda-forge
xorg-libxtst              1.2.3             h7f98852_1002    conda-forge
xorg-recordproto          1.14.2            h7f98852_1002    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yaml-cpp                  0.7.0                h27087fc_2    conda-forge
yarl                      1.7.2            py37h540881e_2    conda-forge
yte                       1.5.1            py37h89c1867_0    conda-forge
zipp                      3.15.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h166bdaf_4    conda-forge
zstandard                 0.18.0           py37h540881e_0    conda-forge
zstd                          .5.2                h3eb15da_6    conda-forge

@c2b2pss
Copy link
Author

c2b2pss commented May 21, 2024

Ugh.....no reply yet :-(

@mohobein
Copy link
Collaborator

Well, we had a public holiday and it was the weekend, and I do not work on these days.

  1. You can try creating an new environment with the lastest TOBIAS version and run the PlotChanges command there. If the error still persists, I suspect the problem might be in the way the environments are resolved during setup.
  2. filter_important_factors.py has and argument -filter. You can put in a number and that amount of TFs with the most extreme changes in each direction will be returned. So filter_important_factors.py -in bindetect_results.txt -filter 5 > most_changed_TFs.txt writes a file with 10 TFs total. Alternatively, you can just write a file yourself with one TF name per line that were chosen based on your own standards. The number of TFs displayed in the PlotChanges plot is purely dictated by the file you provide via --TFS.
  3. There are quite a lot of differences in our packages version, so I cannot really pinpoint the problem. See (1) and if that does not solve it, please send me the package list of that new environment again.

@c2b2pss
Copy link
Author

c2b2pss commented May 21, 2024 •
edited
Loading

Sorry, my comment was supposed to be funny and taken lightly! Seriously, though you guys are the fastest and the most proactive I have encountered on GitHub to responding to issues and even answering simple questions. So my apologies, and thank you.

  1. Can I update the present snake make with a "git pull"? I have the results mostly, so I can give updating a try. Otherwise I'll make a new env.
  2. OK. I'll play with the number of TFs.
  3. Re using WIlson. I am a little lost. Can it plot figures as ouputs besides heatmaps? Is there a Wilson for dummies?
  4. Some formatting questions:
    a. It would be great to modify TFBS signal heatmaps. It is difficult to see the differences using the current plots. What are your suggestions to modify the display of the heatmap? What code lines do I modify?
  • I am not a computer guy but a biologist, so I am still learning what outputs are generated and which can be modified for visualization.

b. I would like to do it in TOBIAS, since I don't see a results file to use in another heatmapping program? Is this correct?

Thanks!

@mohobein
Copy link
Collaborator

  1. No, that will only update the snakemake repository. You have to update your environment or install the newest version of TOBIAS, as it is only part of the environment snakemake uses and not included in this repository itself.
  3. Wilson can also generate other plots, such as box plots, bar plots, line plots, violin plots, scatter plots and dimension reductions. A guide on how to use it can be found in its wiki.
  4. I am assuming you are talking about the output of PlotHeatmap.

TOBIAS generates the heatmap like this. As I am not quite sure what modifications you have in mind exactly, it is hard to recommend specific lines of code to change. The heatmaps themselves are generated here. If you have a specific vision for your heatmap in mind that is not extremely similar to the one TOBIAS generates, it might be easier to write a script or jupyter notebook yourself from the ground up where you just import the input files, extract the relevant data and plot it the way you like. The PlotHeatmap output does not have a lot of customization options without modifying the code.

TOBIAS generates files using standard formats, so you can easily use them with different tools. For the heatmap, TFBS locations are taken from a .bed file and the signals at these locations are extracted from a .bw (bigwig) file (see the wiki link for PlotHeatmap above for more info which files exactly I mean). Using python, pybedtool and pyBigWig can be used to read these file types, respectively. .bed files can also be opened as a dataframe (as they are just tab-separated files), if that is easier for you. In R, you can use rtracklayer instead.
If you prefer not writing something specific yourself, you might want to take a look at deeptools. Using computeMatrix for preprocessing and plotHeatmap afterwards, you can use the .bed and .bw files I mentioned earlier to generate heatmaps. Perhaps they suit your purposes. However, are quite similar to the ones TOBIAS generates itself, but there are some customization options.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@c2b2pss @mohobein