bash train_embedding.sh，为什么显示要配置wandb秘钥？

[lxlx2084]

Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information.
wandb: ERROR api_key not configured (no-tty). call wandb.login(key=[your_api_key])
Traceback (most recent call last):
File "train_embedding.py", line 190, in
main()
File "train_embedding.py", line 85, in main
accelerator.init_trackers('embedding', config=vars(args))
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/accelerator.py", line 734, in _inner
return PartialState().on_main_process(function)(*args, **kwargs)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/accelerator.py", line 2701, in init_trackers
self.trackers.append(tracker_init(project_name, **init_kwargs.get(str(tracker), {})))
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/tracking.py", line 81, in execute_on_main_process
return function(self, *args, **kwargs)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/tracking.py", line 298, in init
self.run = wandb.init(project=self.run_name, **kwargs)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 1270, in init
wandb._sentry.reraise(e)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/analytics/sentry.py", line 161, in reraise
raise exc.with_traceback(sys.exc_info()[2])
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 1255, in init
wi.setup(kwargs)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 305, in setup
wandb_login._login(
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_login.py", line 347, in _login
wlogin.prompt_api_key()
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_login.py", line 281, in prompt_api_key
raise UsageError("api_key not configured (no-tty). call " + directive)
wandb.errors.errors.UsageError: api_key not configured (no-tty). call wandb.login(key=[your_api_key])
[rank0]: Traceback (most recent call last):
[rank0]: File "train_embedding.py", line 190, in
[rank0]: main()
[rank0]: File "train_embedding.py", line 85, in main
[rank0]: accelerator.init_trackers('embedding', config=vars(args))
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/accelerator.py", line 734, in _inner
[rank0]: return PartialState().on_main_process(function)(*args, **kwargs)
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/accelerator.py", line 2701, in init_trackers
[rank0]: self.trackers.append(tracker_init(project_name, **init_kwargs.get(str(tracker), {})))
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/tracking.py", line 81, in execute_on_main_process
[rank0]: return function(self, *args, **kwargs)
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/tracking.py", line 298, in init
[rank0]: self.run = wandb.init(project=self.run_name, **kwargs)
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 1270, in init
[rank0]: wandb._sentry.reraise(e)
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/analytics/sentry.py", line 161, in reraise
[rank0]: raise exc.with_traceback(sys.exc_info()[2])
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 1255, in init
[rank0]: wi.setup(kwargs)
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 305, in setup
[rank0]: wandb_login._login(
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_login.py", line 347, in _login
[rank0]: wlogin.prompt_api_key()
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_login.py", line 281, in prompt_api_key
[rank0]: raise UsageError("api_key not configured (no-tty). call " + directive)
[rank0]: wandb.errors.errors.UsageError: api_key not configured (no-tty). call wandb.login(key=[your_api_key])
E1103 23:47:34.393793 139994886894208 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 3625675) of binary: /data/liuxiang/anaconda3/envs/rag-retrieval/bin/python
Traceback (most recent call last):
File "/data/liuxiang/anaconda3/envs/rag-retrieval/bin/accelerate", line 8, in
sys.exit(main())
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/commands/launch.py", line 1155, in launch_command
multi_gpu_launcher(args)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/commands/launch.py", line 793, in multi_gpu_launcher
distrib_run.run(args)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/torch/distributed/run.py", line 892, in run
elastic_launch(
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 133, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

train_embedding.py FAILED

Failures:
<NO_OTHER_FAILURES>

Root Cause (first observed failure):
[0]:
time : 2024-11-03_23:47:34
host : tang3
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 3625675)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

readme文档中没有提到过要配置api，我不知道在哪里取消它。我也不需要什么可视化

[NLPJCL]

抱歉给你带来困扰，按照习惯默认用wandb来记录了，如果wandb比较麻烦话，您可以有以下两种解决办法：

1. log_with设置为tensorboard。（安装也比较简单）
2. 或者您可以修改源代码，删除这一行即可。https://github.com/NLPJCL/RAG-Retrieval/blob/d00419af6bd23a7131a0f207f30b38ebe640ed9b/rag_retrieval/train/embedding/train_embedding.py#L80C1-L81C1

accelerator的api文档在这里：
https://huggingface.co/docs/accelerate/en/package_reference/accelerator#accelerate.Accelerator.log_with

如果帮助到您，欢迎给RAG-Retrieval一个个star。

[lxlx2084]

感谢你的答复，我按照你说的把那一行删除了，并按照微调embedding里所说修改了代码和制作数据集，.sh内容如下：
CUDA_VISIBLE_DEVICES="0" nohup accelerate launch --config_file ../../../config/default_fsdp.yaml train_embedding.py
--model_name_or_path "/data/liuxiang/58/10_llm/6_比赛/bge-large-zh-v1.5"
--dataset "../../../example_data/train_data.jsonl"
--output_dir "./output/train_data_example"
--batch_size 4
--lr 5e-5
--epochs 4
--save_on_epoch_end 1
--gradient_accumulation_steps 24
#--log_with 'wandb'
--warmup_proportion 0.1
--neg_nums 15
--temperature 0.02
--query_max_len 256
--passage_max_len 3072 \

./logs/t2ranking_100_example.log &

但是执行后在nohub.out中可以看到报错：
(rag-retrieval) liuxiang@Tang3:embedding$cat nohup.out
Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information.
wandb: ERROR api_key not configured (no-tty). call wandb.login(key=[your_api_key])
Traceback (most recent call last):
File "train_embedding.py", line 190, in
main()
File "train_embedding.py", line 85, in main
accelerator.init_trackers('embedding', config=vars(args))
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/accelerator.py", line 734, in _inner
return PartialState().on_main_process(function)(*args, **kwargs)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/accelerator.py", line 2701, in init_trackers
self.trackers.append(tracker_init(project_name, **init_kwargs.get(str(tracker), {})))
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/tracking.py", line 81, in execute_on_main_process
return function(self, *args, **kwargs)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/tracking.py", line 298, in init
self.run = wandb.init(project=self.run_name, **kwargs)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 1270, in init
wandb._sentry.reraise(e)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/analytics/sentry.py", line 161, in reraise
raise exc.with_traceback(sys.exc_info()[2])
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 1255, in init
wi.setup(kwargs)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 305, in setup
wandb_login._login(
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_login.py", line 347, in _login
wlogin.prompt_api_key()
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_login.py", line 281, in prompt_api_key
raise UsageError("api_key not configured (no-tty). call " + directive)
wandb.errors.errors.UsageError: api_key not configured (no-tty). call wandb.login(key=[your_api_key])
[rank0]: Traceback (most recent call last):
[rank0]: File "train_embedding.py", line 190, in
[rank0]: main()
[rank0]: File "train_embedding.py", line 85, in main
[rank0]: accelerator.init_trackers('embedding', config=vars(args))
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/accelerator.py", line 734, in _inner
[rank0]: return PartialState().on_main_process(function)(*args, **kwargs)
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/accelerator.py", line 2701, in init_trackers
[rank0]: self.trackers.append(tracker_init(project_name, **init_kwargs.get(str(tracker), {})))
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/tracking.py", line 81, in execute_on_main_process
[rank0]: return function(self, *args, **kwargs)
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/tracking.py", line 298, in init
[rank0]: self.run = wandb.init(project=self.run_name, **kwargs)
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 1270, in init
[rank0]: wandb._sentry.reraise(e)
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/analytics/sentry.py", line 161, in reraise
[rank0]: raise exc.with_traceback(sys.exc_info()[2])
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 1255, in init
[rank0]: wi.setup(kwargs)
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_init.py", line 305, in setup
[rank0]: wandb_login._login(
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_login.py", line 347, in _login
[rank0]: wlogin.prompt_api_key()
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/wandb/sdk/wandb_login.py", line 281, in prompt_api_key
[rank0]: raise UsageError("api_key not configured (no-tty). call " + directive)
[rank0]: wandb.errors.errors.UsageError: api_key not configured (no-tty). call wandb.login(key=[your_api_key])
E1103 23:59:16.264089 140444912444032 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 3631194) of binary: /data/liuxiang/anaconda3/envs/rag-retrieval/bin/python
Traceback (most recent call last):
File "/data/liuxiang/anaconda3/envs/rag-retrieval/bin/accelerate", line 8, in
sys.exit(main())
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/commands/launch.py", line 1155, in launch_command
multi_gpu_launcher(args)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/commands/launch.py", line 793, in multi_gpu_launcher
distrib_run.run(args)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/torch/distributed/run.py", line 892, in run
elastic_launch(
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 133, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

train_embedding.py FAILED

Failures:
<NO_OTHER_FAILURES>

Root Cause (first observed failure):
[0]:
time : 2024-11-03_23:59:16
host : tang3
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 3631194)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
Batch size: 4
Start with seed: 666
Output dir: ./output/train_data_example
Model_name_or_path: /data/liuxiang/58/10_llm/6_比赛/bge-large-zh-v1.5
Dataset: ../../../example_data/train_data.jsonl
mixed_precision: fp16
gradient_accumulation_steps: 24
temperature: 0.02
log_with: wandb
neg_nums: 15
query_max_len: 128
passage_max_len: 512
use_mrl: False
mrl_dims: []
1912it [00:00, 50789.82it/s]
train_dataloader total is : 1434
train_dataloader data_type is : pair
Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
Start training for 4 epochs
Epoch 1/4 - loss: 0.063600 - lr: 0.000045: 100%|██████████| 1434/1434 [06:16<00:00, 3.81it/s]
./output/train_data_example/runs/checkpoints/checkpoint_0
/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:773: UserWarning: When using NO_SHARD for ShardingStrategy, full_state_dict willbe returned.
warnings.warn(
/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/torch/distributed/fsdp/_state_dict_utils.py:716: UserWarning: When using NO_SHARD for ShardingStrategy, full_state_dict willbe returned.
warnings.warn(
Epoch 2/4 - loss: 0.025800 - lr: 0.000027: 100%|██████████| 1434/1434 [06:04<00:00, 3.93it/s]
./output/train_data_example/runs/checkpoints/checkpoint_1
Epoch 3/4 - loss: 0.015700 - lr: 0.000008: 100%|██████████| 1434/1434 [06:00<00:00, 3.97it/s]
./output/train_data_example/runs/checkpoints/checkpoint_2
Epoch 4/4 - loss: 0.007800 - lr: 0.000000: 100%|██████████| 1434/1434 [06:13<00:00, 3.84it/s]
./output/train_data_example/runs/checkpoints/checkpoint_3
[rank0]: Traceback (most recent call last):
[rank0]: File "train_embedding.py", line 189, in
[rank0]: main()
[rank0]: File "train_embedding.py", line 177, in main
[rank0]: accelerator.wait_for_everyone()
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/accelerator.py", line 2657, in wait_for_everyone
[rank0]: wait_for_everyone()
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/utils/other.py", line 138, in wait_for_everyone
[rank0]: PartialState().wait_for_everyone()
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/state.py", line 375, in wait_for_everyone
[rank0]: torch.distributed.barrier()
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 79, in wrapper
[rank0]: return func(*args, **kwargs)
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3925, in barrier
[rank0]: opts.device = _get_pg_default_device(group)
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 669, in _get_pg_default_device
[rank0]: group = group or _get_default_group()
[rank0]: File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1025, in _get_default_group
[rank0]: raise ValueError(
[rank0]: ValueError: Default process group has not been initialized, please make sure to call init_process_group.
E1104 14:16:01.678559 140416843174528 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: 1) local_rank: 0 (pid: 3902171) of binary: /data/liuxiang/anaconda3/envs/rag-retrieval/bin/python
Traceback (most recent call last):
File "/data/liuxiang/anaconda3/envs/rag-retrieval/bin/accelerate", line 8, in
sys.exit(main())
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/commands/launch.py", line 1155, in launch_command
multi_gpu_launcher(args)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/accelerate/commands/launch.py", line 793, in multi_gpu_launcher
distrib_run.run(args)
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/torch/distributed/run.py", line 892, in run
elastic_launch(
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 133, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/data/liuxiang/anaconda3/envs/rag-retrieval/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

train_embedding.py FAILED

Failures:
<NO_OTHER_FAILURES>

Root Cause (first observed failure):
[0]:
time : 2024-11-04_14:16:01
host : tang3
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 3902171)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

[NLPJCL]

貌似是最终保存模型的时候，accelerate版本内部的bug（不过模型已经在epoch结束后保存了，可以看下），为了避免这个bug，可以尝试降低accelerate版本 0.34.0。

[NLPJCL]

抱歉回复晚了。
1.从效果层面来看，可能是你的场景在bge的覆盖范围内，而你又没有难负例，只有简单的batch内随机负例，query与正例的余弦相似度本来就很高，所以初始的时候loss为0，但是目前的loss打印的是当前batch的loss，可以手动shuff下你的数据，再来看看是否初始loss为0。
2.另外，如果你的场景在bge的覆盖范围内，可以思考下有必要在优化嘛？或者需要挖掘难负例来优化，简单的负例已经不够了。

[NLPJCL]

抱歉回复晚了。
1.从效果层面来看，可能是你的场景在bge的覆盖范围内，而你又没有难负例，只有简单的batch内随机负例，query与正例的余弦相似度本来就很高，所以初始的时候loss为0，但是目前的loss打印的是当前batch的loss，可以手动shuff下你的数据，再来看看是否初始loss为0。
2.另外，如果你的场景在bge的覆盖范围内，可以思考下有必要在优化嘛？或者需要挖掘难负例来优化，简单的负例已经不够了。

[lxlx2084]

1.什么是难负例？是和query非常接近但是不应该被选中的doc（比如正例是0.8，难负例是0.75），还是相似度最低的那几个doc？
2.可以手动选取难负例吗？比如我就选和当前query相似度最低的那几个doc作为难负例。如果一条query对应3个正例，难负例选几个合适？

再次感谢您的答疑，非常感谢

[NLPJCL]

难负例是你说的这个：和query非常接近但是不应该被选中的doc（比如正例是0.8，难负例是0.75）
挖掘难负例是一个重要的学术方向，可以看看论文，bge之类的。

[lxlx2084]

好的，非常感谢。我也会把这个仓库推荐给同学