how to quick start on linux platform ？

[fanqiangwei]

how to quick start on linux platform ？ I only find windows quick start！

[None1024Outlook]

how to quick start on linux platform ？ I only find windows quick start！

You can use this notebook locally (set useColab to False)
https://colab.research.google.com/drive/1trBvrdgyI-Ntd45ZnlT5lhGsI_HnKjC1?usp=sharing

You can take a look at this, too
https://nonewiki.top/2024/12/02/Fish-Speech-env-setup-and-train-tutorial/

[cod3r0k]

Can you help me in test it with one data from A-Z?

i have an error because i never has data in my colab.

The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.
0it [00:00, ?it/s]
2025-01-24 19:49:10.535 | INFO     | __main__:main:189 | RANK: 0 / 1 - Starting worker
Traceback (most recent call last):
  File "/content/drive/MyDrive/fish-speech/tools/vqgan/extract_vq.py", line 232, in <module>
    main()
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/content/drive/MyDrive/fish-speech/tools/vqgan/extract_vq.py", line 193, in main
    files = list_files(folder, AUDIO_EXTENSIONS, recursive=True, sort=False)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/fish_speech/utils/file.py", line 79, in list_files
    raise FileNotFoundError(f"Directory {path} does not exist.")
FileNotFoundError: Directory data does not exist.
Traceback (most recent call last):
  File "/content/drive/MyDrive/fish-speech/tools/llama/build_dataset.py", line 169, in <module>
    main()
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/content/drive/MyDrive/fish-speech/tools/llama/build_dataset.py", line 131, in main
    assert f.exists(), f"{f} not found"
           ^^^^^^^^^^
AssertionError: data not found

what do you suggest to edit code?

# Datasets

# Batch extraction of semantic tokens
!python tools/vqgan/extract_vq.py data --num-workers 1 --batch-size 16 --config-name {vqgan_config_name} --checkpoint-path {vqgan_model}

# Pack the dataset into protobuf
input_dir = "data" #@param {type: "string"}
output_dir = "data/protos" #@param {type: "string"}
!python tools/llama/build_dataset.py --input {input_dir} --output {output_dir} --text-extension .lab --num-workers 16

For example, using LJ Speech or any dataset, I test all the steps thoroughly with correct code. One person tested it before, and it is working. Then, I verify it to ensure no mistakes are made.

[None1024Outlook]

Update the python library and make sure the model is set correctly

…

________________________________ 发件人: cod3r0k ***@***.***> 发送时间: 2025年1月25日 3:52 收件人: fishaudio/fish-speech ***@***.***> 抄送: None1145 ***@***.***>; Comment ***@***.***> 主题: Re: [fishaudio/fish-speech] how to quick start on linux platform ？ (Discussion #811) Can you help me in test it with one data from A-Z? i have an error because i never has data in my colab. The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`. 0it [00:00, ?it/s] 2025-01-24 19:49:10.535 | INFO | __main__:main:189 | RANK: 0 / 1 - Starting worker Traceback (most recent call last): File "/content/drive/MyDrive/fish-speech/tools/vqgan/extract_vq.py", line 232, in <module> main() File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1161, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1082, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1443, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 788, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/content/drive/MyDrive/fish-speech/tools/vqgan/extract_vq.py", line 193, in main files = list_files(folder, AUDIO_EXTENSIONS, recursive=True, sort=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/fish_speech/utils/file.py", line 79, in list_files raise FileNotFoundError(f"Directory {path} does not exist.") FileNotFoundError: Directory data does not exist. Traceback (most recent call last): File "/content/drive/MyDrive/fish-speech/tools/llama/build_dataset.py", line 169, in <module> main() File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1161, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1082, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 1443, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/click/core.py", line 788, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/content/drive/MyDrive/fish-speech/tools/llama/build_dataset.py", line 131, in main assert f.exists(), f"{f} not found" ^^^^^^^^^^ AssertionError: data not found what do you suggest to edit code? # Datasets # Batch extraction of semantic tokens !python tools/vqgan/extract_vq.py data --num-workers 1 --batch-size 16 --config-name {vqgan_config_name} --checkpoint-path {vqgan_model} # Pack the dataset into protobuf input_dir = "data" ***@***.*** {type: "string"} output_dir = "data/protos" ***@***.*** {type: "string"} !python tools/llama/build_dataset.py --input {input_dir} --output {output_dir} --text-extension .lab --num-workers 16 For example, using LJ Speech or any dataset, I test all the steps thoroughly with correct code. One person tested it before, and it is working. Then, I verify it to ensure no mistakes are made. ― Reply to this email directly, view it on GitHub<#811 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AZS5HRTDA2Y27WJRJTTUXE32MKKZJAVCNFSM6AAAAABU3G5TWKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCOJUG4YDONY>. You are receiving this because you commented.Message ID: ***@***.***>

[cod3r0k]

No no, I just need download dataset as below:

import os
import tarfile
import requests

# Step 1: Download LJSpeech Dataset
def download_and_prepare_ljspeech():
    dataset_url = "https://data.keithito.com/data/speech/LJSpeech-1.1.tar.bz2"
    output_tar = "LJSpeech-1.1.tar.bz2"
    extract_dir = "data"

    # Download the dataset
    print("Downloading LJSpeech dataset...")
    response = requests.get(dataset_url, stream=True)
    with open(output_tar, "wb") as file:
        for chunk in response.iter_content(chunk_size=8192):
            file.write(chunk)
    print("Download complete.")

    # Extract the dataset
    print("Extracting dataset...")
    with tarfile.open(output_tar, 'r:bz2') as tar_ref:
        tar_ref.extractall(extract_dir)
    print("Extraction complete.")

    # Prepare the dataset in the required structure
    print("Preparing dataset...")
    source_dir = os.path.join(extract_dir, "LJSpeech-1.1")
    target_dir = os.path.join(extract_dir, "ljspeech")
    os.makedirs(target_dir, exist_ok=True)

    with open(os.path.join(source_dir, "metadata.csv"), "r", encoding="utf-8") as metadata:
        for line in metadata:
            parts = line.strip().split("|")
            wav_file = parts[0] + ".wav"
            transcription = parts[2]

            speaker_dir = os.path.join(target_dir, "speaker_1")
            os.makedirs(speaker_dir, exist_ok=True)

            # Move WAV file
            src_wav = os.path.join(source_dir, "wavs", wav_file)
            dst_wav = os.path.join(speaker_dir, wav_file)
            os.rename(src_wav, dst_wav)

            # Write transcription
            lab_file = wav_file.replace(".wav", ".lab")
            dst_lab = os.path.join(speaker_dir, lab_file)
            with open(dst_lab, "w", encoding="utf-8") as lab:
                lab.write(transcription)

    print("Dataset preparation complete.")

# Run the preparation function
download_and_prepare_ljspeech()

but I cant go further ... its training take long time .... it is just do epoch 0 and then finished

[None1024Outlook]

This is the normal situation (since your data set is somewhat large) and you can adjust the training parameters to train

…

________________________________ 发件人: cod3r0k ***@***.***> 发送时间: 2025年1月25日 21:06 收件人: fishaudio/fish-speech ***@***.***> 抄送: None1145 ***@***.***>; Comment ***@***.***> 主题: Re: [fishaudio/fish-speech] how to quick start on linux platform ？ (Discussion #811) No no, I just need download dataset as below: import os import tarfile import requests # Step 1: Download LJSpeech Dataset def download_and_prepare_ljspeech(): dataset_url = "https://data.keithito.com/data/speech/LJSpeech-1.1.tar.bz2" output_tar = "LJSpeech-1.1.tar.bz2" extract_dir = "data" # Download the dataset print("Downloading LJSpeech dataset...") response = requests.get(dataset_url, stream=True) with open(output_tar, "wb") as file: for chunk in response.iter_content(chunk_size=8192): file.write(chunk) print("Download complete.") # Extract the dataset print("Extracting dataset...") with tarfile.open(output_tar, 'r:bz2') as tar_ref: tar_ref.extractall(extract_dir) print("Extraction complete.") # Prepare the dataset in the required structure print("Preparing dataset...") source_dir = os.path.join(extract_dir, "LJSpeech-1.1") target_dir = os.path.join(extract_dir, "ljspeech") os.makedirs(target_dir, exist_ok=True) with open(os.path.join(source_dir, "metadata.csv"), "r", encoding="utf-8") as metadata: for line in metadata: parts = line.strip().split("|") wav_file = parts[0] + ".wav" transcription = parts[2] speaker_dir = os.path.join(target_dir, "speaker_1") os.makedirs(speaker_dir, exist_ok=True) # Move WAV file src_wav = os.path.join(source_dir, "wavs", wav_file) dst_wav = os.path.join(speaker_dir, wav_file) os.rename(src_wav, dst_wav) # Write transcription lab_file = wav_file.replace(".wav", ".lab") dst_lab = os.path.join(speaker_dir, lab_file) with open(dst_lab, "w", encoding="utf-8") as lab: lab.write(transcription) print("Dataset preparation complete.") # Run the preparation function download_and_prepare_ljspeech() but I cant go further ... its training take long time .... it is just do epoch 0 and then finished ― Reply to this email directly, view it on GitHub<#811 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AZS5HRWEBH7SWRJZIOTXHE32MOD3RAVCNFSM6AAAAABU3G5TWKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCOJVGE3DCNI>. You are receiving this because you commented.Message ID: ***@***.***>