Allow matching of files to DATs via non-CRC32 checksums #818
Assignees
Labels
enhancement
New feature or request
Comments
|
I think what you're really looking for here is the ability to use something other than CRC32+size when matching scanned files to a DAT, rather than during testing. For For the file matching, it's going to be best to wait until I finally finish #740 which necessitates calculating the MD5 and SHA1 of scanned files. RE: caching, you've touched on every important part that I've thought of. And if we consider CHDs to be a deterministic archive of sorts, they would apply here as well. Thankfully Unfortunately, I don't see any active Node.js TorrentZip implementations, so the creation of them is unlikely to happen any time soon. But in general, I think local caching can be handled separately from the issue of alternative checksum matching. |
Yes, I maybe wasn't clear but this would be to match roms to DATs. So for |
|
File matching on something other than CRC32+size has been on my personal to-do list forever: igir/src/modules/fileIndexer.ts Line 40 in 7c6bfbb My free time is going to be limited most likely through the end of the year, but I think this is a good suggestion. It should be easier to land after #819 has merged. |
emmercm
changed the title
|
Note: this will be required for processing MAME "disk" files, the MAME DATs only provide a SHA1 for those files and no filesize. The code added in #835 will need to account for the zero filesize of these. |
This was referenced Feb 20, 2024
|
Question for you @TheBrainScrambler while I wrap this feature up - what DAT group includes SHA256 in their DATs? It's very non-standard, and I haven't come across it yet myself. |
|
While the No-Intro PC/XML set doesn't have any SHA256, the Standard DAT set has some. From |
|
Interesting. I have absolutely no idea why their P/C DATs differ so greatly from their "standard" ones. I see no reason why P/C couldn't include SHA256 as well. |
|
Thanks ! And any chance to see SHA256 support in igir as well ? |
|
@TheBrainScrambler absolutely, this will add it soon: #1032 |
🔒 Inactive issue lockThis issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. Comment generated by the GitHub Lock Issues workflow. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Is your feature request related to a problem?
No response
Describe the solution you'd like
From what I understand, igir checks if roms match a certain DAT by either comparing it to the CRC32 checksum stored in the header of a zip file, if the rom is zipped and the header exists, or by computing their CRC32 checksum.
I would like an option to force to verify roms by comparing them using the checksum algorithm of my choice, for me specifically SHA-256.
I was thinking of extending
igir reportby having a flag that allows me to specify that, maybe--checksum-algo sha256.If there are no sha256 checksums in the DAT, then you try weaker checksums like SHA1, MD5 or CRC32, but you report that you couldn't verify the rom using the requested checksum algorithm. Or perhaps you just add a field saying with which checksum algorithm you did the verification. In any case, igir should first try the algorithm you asked for.
I'm wondering though: in https://igir.io/alternatives/ , you say that "ROMs: scan/checksum caching" is not supported by design. Does this concern my feature request ?
I was also thinking about the efficiency of this since it would require to unzip archives of roms to verify them. This is where I think having some kind of caching would be good, but then I know you don't want caching by design. Perhaps igir could make some kind of DAT file which for each rom could also contain a checksum of the zipped rom ? If you're using a deterministic zipping algorithm like TorrentZip then this would work.
Additional context
No response
The text was updated successfully, but these errors were encountered: