need to update command on how to run coverage and fix directory path where riscof looks for elf

[dansmathers]

moving issue from riscv-software-src/riscv-config#135

https://riscof.readthedocs.io/en/stable/coverage.html

The current directions say to run coverage:
riscof --verbose debug coverage --suite /path/to/suite --env /path/to/suite

this will generate an error. -c or --cgf_file is a required option.

However, even if -cgf_file is used, there is another error. riscof looks for the elf file in the wrong directory:

FileNotFoundError: [Errno 2] No such file or directory: '/riscof/riscof_work/rv32i_m/C/src/caddi-01.S/ref.elf'

but it looks like it should instead be looking in:
./rv32i_m/I/src/addi-01.S/ref/ref.elf

Thanks

I don't think the paths are wrong. Both would probably point to the same thing. All paths are resolved to be absolute paths before the actual execution of riscof starts. The former simply looks like the absolute path of the latter. I think the compilation was not successful. Can you share the entire log?

@dansmathers its better to move/track this issue in the riscof repo since it doesn't have anything to do with riscv-config.

[dansmathers]

riscof coverage --config=config.ini --suite=riscv-arch-test/riscv-test-suite/ --env=riscv-arch-test/riscv-test-suite/env --cgf-file /riscof/riscv-arch-test/coverage/rvi_c.cgf --cgf-file /riscof/riscv-arch-test/coverage/rvi.cgf --cgf-file /riscof/riscv-arch-test/coverage/rvi_fencei.cgf --cgf-file /riscof/riscv-arch-test/coverage/rvi_m.cgf --cgf-file /riscof/riscv-arch-test/coverage/rvi_priv.cgf | tee cov.log
INFO | ****** RISCOF: RISC-V Architectural Test Framework 1.25.3 *******
INFO | using riscv_isac version : 0.17.0
INFO | using riscv_config version : 3.9.1
INFO | Reading configuration from: /riscof/config.ini
INFO | Preparing Models
INFO | Input-ISA file
INFO | Loading input file: /riscof/spike/spike_isa.yaml
INFO | Load Schema /.local/lib/python3.8/site-packages/riscv_config/schemas/schema_isa.yaml
INFO | Processing Hart: hart0
INFO | Initiating Validation
INFO | No errors for Hart: 0 :)
INFO | Updating fields node for each CSR
INFO | Dumping out Normalized Checked YAML: /riscof/riscof_work/spike_isa_checked.yaml
INFO | Input-Platform file
INFO | Loading input file: /riscof/spike/spike_platform.yaml
INFO | Load Schema /.local/lib/python3.8/site-packages/riscv_config/schemas/schema_platform.yaml
INFO | Initiating Validation
INFO | No Syntax errors in Input Platform Yaml. :)
INFO | Dumping out Normalized Checked YAML: /riscof/riscof_work/spike_platform_checked.yaml
INFO | Generating database for suite: /riscof/riscv-arch-test-20230726/riscv-test-suite
INFO | Database File Generated: /riscof/riscof_work/database.yaml
INFO | Env path set to/riscof/riscv-arch-test-20230726/riscv-test-suite/env
INFO | Will collect Coverage using RISCV-ISAC
INFO | CGF file(s) being used : ('/riscof/riscv-arch-test-20230726/coverage/rvi_c.cgf', '/riscof/riscv-arch-test-20230726/coverage/rvi.cgf', '/riscof/riscv-arch-test-20230726/coverage/rvi_fencei.cgf', '/riscof/riscv-arch-test-20230726/coverage/rvi_m.cgf', '/riscof/riscv-arch-test-20230726/coverage/rvi_priv.cgf')
INFO | Selecting Tests.
INFO | Running Tests on Reference.
ERROR | INFO | ****** RISC-V ISA Coverage 0.17.0 *******
INFO | Copyright (c) 2020, InCore Semiconductors Pvt. Ltd.
INFO | All Rights Reserved.

Traceback (most recent call last):
File "/.local/bin/riscv_isac", line 8, in
sys.exit(cli())
File "/.local/lib/python3.8/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/.local/lib/python3.8/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/.local/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/.local/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/.local/lib/python3.8/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/.local/lib/python3.8/site-packages/riscv_isac/main.py", line 136, in coverage
isac(output_file,elf,trace_file, window_size, expand_cgf(cgf_file,int(xlen),int(flen),log_redundant), parser_name, decoder_name, parser_path, decoder_path, detailed, test_label,
File "/.local/lib/python3.8/site-packages/riscv_isac/cgf_normalize.py", line 565, in expand_cgf
cgf = utils.load_cgf(cgf_files)
File "/.local/lib/python3.8/site-packages/riscv_isac/utils.py", line 163, in load_cgf
safe_yaml.dump(dict(safe_yaml.load(fp)),string_stream)
File "/.local/lib/python3.8/site-packages/ruamel/yaml/main.py", line 426, in load
return constructor.get_single_data()
File "/.local/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 111, in get_single_data
node = self.composer.get_single_node()
File "_ruamel_yaml.pyx", line 706, in _ruamel_yaml.CParser.get_single_node
File "_ruamel_yaml.pyx", line 724, in _ruamel_yaml.CParser._compose_document
File "_ruamel_yaml.pyx", line 775, in _ruamel_yaml.CParser._compose_node
File "_ruamel_yaml.pyx", line 889, in _ruamel_yaml.CParser._compose_mapping_node
File "_ruamel_yaml.pyx", line 775, in _ruamel_yaml.CParser._compose_node
File "_ruamel_yaml.pyx", line 889, in _ruamel_yaml.CParser._compose_mapping_node
File "_ruamel_yaml.pyx", line 775, in _ruamel_yaml.CParser._compose_node
File "_ruamel_yaml.pyx", line 889, in _ruamel_yaml.CParser._compose_mapping_node
File "_ruamel_yaml.pyx", line 743, in _ruamel_yaml.CParser._compose_node
ruamel.yaml.composer.ComposerError: found undefined alias
in "", line 15, column 11
make: *** [/riscof/riscof_work/Makefile.Reference-sail_c_simulator:5: TARGET0] Error 1
INFO | Merging Coverage reports
Traceback (most recent call last):
File "/.local/bin/riscof", line 8, in
sys.exit(cli())
File "/.local/lib/python3.8/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/.local/lib/python3.8/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/.local/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/.local/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/.local/lib/python3.8/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/.local/lib/python3.8/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "/.local/lib/python3.8/site-packages/riscof/cli.py", line 399, in coverage
report, for_html, test_stats, coverpoints = framework.run_coverage(base, isa_file, platform_file,
File "/.local/lib/python3.8/site-packages/riscof/framework/main.py", line 126, in run_coverage
'test_size': [str(entry) for entry in find_elf_size(elf)],
File "/.local/lib/python3.8/site-packages/riscof/framework/main.py", line 59, in find_elf_size
with open(elf, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/riscof/riscof_work/rv32i_m/C/src/cadd-01.S/ref/ref.elf'

[dansmathers]

@pawks Can you share your command you use to run coverage and I will try that? I was guessing at the correct command because I didn't see a quickstart command that I could just run correctly?

Thanks,
Dan

[pawks]

The error you are seeing is because you didnt include the dataset.cgf in the command line. It should be the first file in the list of cgf file arguments. The correct command should look something like the following:

riscof coverage --config=config.ini --suite=riscv-arch-test/riscv-test-suite/ --env=riscv-arch-test/riscv-test-suite/env --cgf-file /riscof/riscv-arch-test/dataset.cgf --cgf-file /riscof/riscv-arch-test/coverage/rvi_c.cgf --cgf-file /riscof/riscv-arch-test/coverage/rvi.cgf --cgf-file /riscof/riscv-arch-test/coverage/rvi_fencei.cgf --cgf-file /riscof/riscv-arch-test/coverage/rvi_m.cgf --cgf-file /riscof/riscv-arch-test/coverage/rvi_priv.cgf

[neelgala]

@dansmathers as we noted in the last SIG call.. the docs for "test contributors" is missing and we will be fixing that asap in the coming weeks. It would be great if you could list out the missing docs you encountered in your struggle so we can accurately address those and others together.

Thanks for the inputs.

[dansmathers]

Thanks. that command helped. I still see errors going by for some tests but the script ran. I was trying to build everything from scratch again to check but it looks like the current github sail build is currently broken. It does seem like it might be hard to do large unified pulls because there are 4 or 5 githubs that need to be kept in sync?