diff --git a/README.md b/README.md index b3f0a8190..8e9d75191 100644 --- a/README.md +++ b/README.md @@ -29,14 +29,14 @@ A video of an example use case: With `igir` you can manage a ROM collection of any size: - 🔍 Scan for DATs, ROMs, and ROM patches - including those in archives (see [scanning](https://igir.io/input/file-scanning) & [archive docs](https://igir.io/input/reading-archives)) -- 📂 Organize ROM files by console (see [DAT docs](https://igir.io/input/dats)) -- 🪄 Name ROM files consistently, including the right extension (see [DAT docs](https://igir.io/input/dats)) +- 📂 Organize ROM files by console (see [DAT docs](https://igir.io/dats/overview)) +- 🪄 Name ROM files consistently, including the right extension (see [DAT docs](https://igir.io/dats/overview)) - ✂️ Filter out duplicate ROMs, or ROMs in languages you don't understand (see [filtering docs](https://igir.io/roms/filtering-preferences)) - 🗜️ Extract or archive ROMs in mass (see [archive docs](https://igir.io/output/writing-archives)) - 🩹 Patch ROMs automatically in mass (see [scanning](https://igir.io/input/file-scanning) & [patching docs](https://igir.io/roms/patching)) - 🎩 Parse ROMs with headers, and optionally remove them (see [header docs](https://igir.io/roms/headers)) - ↔️ Build & re-build (un-merge, split, or merge) MAME ROM sets (see [arcade docs](https://igir.io/usage/arcade)) -- 🔮 Report on what ROMs are present or missing for each console, and create fixdats for missing ROMs (see [reporting](https://igir.io/output/reporting) & [DAT docs](https://igir.io/input/dats)) +- 🔮 Report on what ROMs are present or missing for each console, and create fixdats for missing ROMs (see [reporting](https://igir.io/output/reporting) & [DAT docs](https://igir.io/dats/overview)) ## How do I run `igir`? diff --git a/docs/alternatives.md b/docs/alternatives.md index df4579462..0e26b766d 100644 --- a/docs/alternatives.md +++ b/docs/alternatives.md @@ -9,12 +9,12 @@ There are a few different popular ROM managers that have similar features: | App: OS compatibility | ✅ anything [Node.js supports](https://nodejs.org/en/download) | ⚠️ Windows, macOS & Linux via [Wine](https://www.winehq.org/) | ⚠️ Windows, Linux via [Mono](https://www.mono-project.com/) | ❌ Windows only | | App: UI or CLI | CLI only by design | UI only | Separate UI & CLI versions | UI only | | App: required setup steps | ✅ no setup required | ❌ requires "profile" setup per DAT | ⚠️ if specifying DAT & ROM dirs | ❌ requires per-DAT DB setup | -| DATs: supported formats | Logiqx XML, MAME ListXML, CMPro, HTGD SMDB ([DATs docs](input/dats.md)) | Logiqx XML, MAME ListXML, CMPro | Logiqx XML, MAME ListXML, CMPro, RomCenter, HTGD SMDB | Logiqx XML, CMPro, RomCenter | +| DATs: supported formats | Logiqx XML, MAME ListXML, CMPro, HTGD SMDB ([DATs docs](dats/overview.md)) | Logiqx XML, MAME ListXML, CMPro | Logiqx XML, MAME ListXML, CMPro, RomCenter, HTGD SMDB | Logiqx XML, CMPro, RomCenter | | DATs: process multiple at once | ✅ | ⚠️ via the batcher | ✅ | ❌ | | DATs: infer parent/clone info | ✅ | ❌ | ❌ | ❌ | | DATs: built-in download manager | ❌ | ❌ | ⚠️ via [DatVault](https://www.datvault.com/) | ❌ | | DATs: supports DAT URLs | ✅ | ❌ | ❌ | ❌ | -| DATs: create from files (dir2dat) | ❌ | ✅ | ✅ | ❌ | +| DATs: create from files (dir2dat) | ✅ [dir2dat docs](dats/dir2dat.md) | ✅ | ❓ | ❌ | | DATs: combine multiple | ❌ | ❌ | ✅ | ❌ | | Archives: extraction formats | ✅ many formats ([reading archives docs](input/reading-archives.md)) | ✅ `.zip`, `.7z`, `.rar` | ⚠️ `.zip`, `.7z` | ⚠️ `.zip`, `.7z` | | Archives: creation formats | ❌ `.zip` only by design ([writing archives docs](output/writing-archives.md)) | ✅ `.zip`, `.7z`, `.rar` | ⚠️ `.zip`, `.7z` | ⚠️ `.zip`, `.7z` | @@ -31,7 +31,7 @@ There are a few different popular ROM managers that have similar features: | Output: separate input & output dirs | ✅ | ❌ | ⚠️ yes but files are always moved | ❌ | | Output: subdirectory customization | ✅ | ❌ | ⚠️ depends on DAT organization | ❌ | | Output: create single archive for DAT | ✅ | ❌ | ✅ | ❌ | -| Output: fixdat creation | ✅ [DATs docs](input/dats.md) | ✅ | ✅ | ❌ | +| Output: fixdat creation | ✅ [Fixdat docs](dats/fixdats.md) | ✅ | ✅ | ❌ | !!! note diff --git a/docs/commands.md b/docs/commands.md index f82a9f0e1..d2d699b7f 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -18,9 +18,9 @@ Files in the input directories will be left alone, they will _not_ be modified o ### `move` -Move ROMs from an input directory to the output directory. The same directory can be specified for both input & output, resulting in ROMs being renamed as their names change in [DATs](input/dats.md). +Move ROMs from an input directory to the output directory. The same directory can be specified for both input & output, resulting in ROMs being renamed as their names change in [DATs](dats/overview.md). -ROMs will be deleted from their input directory after _all_ ROMs for _every_ [DAT](input/dats.md) have been written. +ROMs will be deleted from their input directory after _all_ ROMs for _every_ [DAT](dats/overview.md) have been written. ### `symlink` @@ -63,7 +63,7 @@ After performing one of the ROM writing commands, verify that the file was writt ### `clean` -Files in the output directory that do not match any ROM in any [DAT](input/dats.md) will be deleted. +Files in the output directory that do not match any ROM in any [DAT](dats/overview.md) will be deleted. See the [output cleaning page](output/cleaning.md) for more information. @@ -71,6 +71,6 @@ See the [output cleaning page](output/cleaning.md) for more information. ### `report` -A report will be generated of what input files were matched by what DAT, and what games in what [DATs](input/dats.md) have missing ROMs. +A report will be generated of what input files were matched by what DAT, and what games in what [DATs](dats/overview.md) have missing ROMs. See the [reporting page](output/reporting.md) for more information. diff --git a/docs/dats/dir2dat.md b/docs/dats/dir2dat.md new file mode 100644 index 000000000..926d343c8 --- /dev/null +++ b/docs/dats/dir2dat.md @@ -0,0 +1,84 @@ +# dir2dat + +"dir2dat" refers to DATs that have been automatically created based on files in an input directory. DATs generated this way are not typically useful as-is, they usually require some hand editing after creation. + +`igir` has the ability to create these DATs with the `igir dir2dat` command. Example: + +```shell +igir dir2dat --input [--input ..] +``` + +## dir2dat rules + +`igir` uses the following rules when creating dir2dat DAT files: + +- **A DAT file will be created for every input path.** + + If multiple input paths overlap, such as: + + === ":simple-windowsxp: Windows" + + ```batch + igir dir2dat ^ + --input "C:\ROMs" ^ + --input "C:\ROMs\NES" + ``` + + === ":simple-apple: macOS" + + ```shell + igir dir2dat \ + --input ~/ROMs/ \ + --input ~/ROMs/NES + ``` + + === ":simple-linux: Linux" + + ```shell + igir dir2dat \ + --input ~/ROMs/ \ + --input ~/ROMs/NES + ``` + + then ROMs can appear in multiple resulting dir2dat files. + +- **Each input path's [basename](https://linux.die.net/man/1/basename) will be used for the DAT's name.** + + Here are some examples: + + | Input path | DAT name | + |----------------------------|----------| + | `--input "ROMs"` | `ROMs` | + | `--input "ROMs/NES"` | `NES` | + | `--input "ROMs/SNES/*"` | `SNES` | + | `--input "ROMs/SNES/**/*"` | `SNES` | + +- **Archive files will be treated as a single game, with every archive entry being a separate ROM.** + + This is consistent with how the [`igir zip` command](../output/writing-archives.md) works, and with what [MAME expects](../usage/arcade.md). + +- **The input file's [basename](https://linux.die.net/man/1/basename) (without extension) will be used for the game name.** + + !!! warning + + This will cause input files with the same basename to be grouped together! + +## Combining with other options + +Once DATs have been generated from input files, they are processed the same as any other DAT file. That means: + +- **Parent/clone information may be [inferred](overview.md#parentclone-inference) from game names.** + + If your input files are in some kind of standard naming convention (e.g. [No-Intro](https://wiki.no-intro.org/index.php?title=Naming_Convention), [Redump](https://datomatic.no-intro.org/stuff/The%20Official%20No-Intro%20Convention%20(20071030).pdf), or [TOSEC](https://www.tosecdev.org/tosec-naming-convention)), then parent/clone information can be inferred for [1G1R preferences](../roms/filtering-preferences.md). + + Parent/clone information also allows for [merging & splitting](../usage/arcade.md) of ROM sets. + +- **[ROM filter options](../roms/filtering-preferences.md) can be applied.** + + If your input files are in some kind of standard naming convention (e.g. [No-Intro](https://wiki.no-intro.org/index.php?title=Naming_Convention), [Redump](https://datomatic.no-intro.org/stuff/The%20Official%20No-Intro%20Convention%20(20071030).pdf), or [TOSEC](https://www.tosecdev.org/tosec-naming-convention)) that contains region, language, or other tags, then [ROM filter options](../roms/filtering-preferences.md) can be applied. + +## Alternative tools + +It is unlikely that any ROM tool, including `igir`, will ever meet every person's exact DAT creation needs. + +[SabreTools](https://github.com/SabreTools/SabreTools) is a great tool for DAT management that offers many complex options for DAT creation, filtering, merging, and splitting. diff --git a/docs/dats/fixdats.md b/docs/dats/fixdats.md new file mode 100644 index 000000000..a22ed55a6 --- /dev/null +++ b/docs/dats/fixdats.md @@ -0,0 +1,52 @@ +# Fixdats + +"Fixdats" are DATs that contain only ROMs that are missing from your collection. Fixdats are derived from some other DAT (see the [DATs overview docs](overview.md) for how to obtain DATs), containing only a subset of the ROMs. Fixdats are specific to the state of each person's ROM collection, so they aren't necessarily meaningful to other people. + +Fixdats help you find files missing from your collection, and they can be used to generate a collection of those files once you've found them. This sub-collection of files can then be merged back into your main collection. + +The `fixdat` command creates a [Logiqx XML](http://www.logiqx.com/DatFAQs/) DAT for every input DAT (the `--dat ` option) that is missing ROMs. When writing (`copy`, `move`, and `symlink` [commands](../commands.md)), the fixdat will be written to the output directory, otherwise it will be written to the working directory. + +For example: + +=== ":simple-windowsxp: Windows" + + ```batch + igir copy zip fixdat ^ + --dat "Nintendo - Game Boy.dat" ^ + --dat "Nintendo - Game Boy Advance.dat" ^ + --dat "Nintendo - Game Boy Color.dat" ^ + --input ROMs\ ^ + --output ROMs-Sorted\ ^ + --fixdat + ``` + +=== ":simple-apple: macOS" + + ```shell + igir copy zip fixdat \ + --dat "Nintendo - Game Boy.dat" \ + --dat "Nintendo - Game Boy Advance.dat" \ + --dat "Nintendo - Game Boy Color.dat" \ + --input ROMs/ \ + --output ROMs-Sorted/ + ``` + +=== ":simple-linux: Linux" + + ```shell + igir copy zip fixdat \ + --dat "Nintendo - Game Boy.dat" \ + --dat "Nintendo - Game Boy Advance.dat" \ + --dat "Nintendo - Game Boy Color.dat" \ + --input ROMs/ \ + --output ROMs-Sorted/ + ``` + +may produce some fixdats in the `ROMs-Sorted/` directory, if any of the input DATs have ROMs that weren't found in the `ROMs/` input directory: + +```text +ROMs-Sorted/ +├── Nintendo - Game Boy (20230414-173400) fixdat.dat +├── Nintendo - Game Boy Advance (20230414-173400) fixdat.dat +└── Nintendo - Game Boy Color (20230414-173400) fixdat.dat +``` diff --git a/docs/input/dats.md b/docs/dats/overview.md similarity index 83% rename from docs/input/dats.md rename to docs/dats/overview.md index 43078cfc4..920f97d6f 100644 --- a/docs/input/dats.md +++ b/docs/dats/overview.md @@ -1,4 +1,4 @@ -# DATs +# DAT Overview ## Overview @@ -80,7 +80,7 @@ There have been a few DAT-like formats developed over the years. `igir` supports ## DAT input options -The `--dat ` supports files, archives, directories, and globs like any of the other file options. See the [file scanning page](file-scanning.md) for more information. +The `--dat ` supports files, archives, directories, and globs like any of the other file options. See the [file scanning page](../input/file-scanning.md) for more information. `igir` also supports URLs to DAT files and archives. This is helpful to make sure you're always using the most up-to-date version of a DAT hosted on sites such as GitHub. For example: @@ -162,56 +162,3 @@ The rule-of-thumb with DATs and arcade emulation is: your emulator probably has If you are using a desktop frontend such as [RetroArch](../usage/desktop/retroarch.md), it may come with multiple versions of the same emulator, and it is unlikely that any of them is the most recent version. Follow the frontend's documentation to location or download the correct DAT to use with each emulator. See the [arcade page](../usage/arcade.md) for more information on building & re-building arcade ROM sets. - -## Fixdats - -"Fixdats" are DATs that contain only ROMs that are missing from your collection. Fixdats are derived from some other DAT (see above for obtaining DATs), containing only a subset of the ROMs. Fixdats are specific to the state of each person's ROM collection, so they aren't necessarily meaningful to other people. - -Fixdats help you find files missing from your collection, and they can be used to generate a collection of those files once you've found them. This sub-collection of files can then be merged back into your main collection. - -The `fixdat` command creates a [Logiqx XML](http://www.logiqx.com/DatFAQs/) DAT for every input DAT (the `--dat ` option) that is missing ROMs. When writing (`copy`, `move`, and `symlink` commands), the fixdat will be written to the output directory, otherwise it will be written to the working directory. - -For example: - -=== ":simple-windowsxp: Windows" - - ```batch - igir copy zip fixdat ^ - --dat "Nintendo - Game Boy.dat" ^ - --dat "Nintendo - Game Boy Advance.dat" ^ - --dat "Nintendo - Game Boy Color.dat" ^ - --input ROMs\ ^ - --output ROMs-Sorted\ ^ - --fixdat - ``` - -=== ":simple-apple: macOS" - - ```shell - igir copy zip fixdat \ - --dat "Nintendo - Game Boy.dat" \ - --dat "Nintendo - Game Boy Advance.dat" \ - --dat "Nintendo - Game Boy Color.dat" \ - --input ROMs/ \ - --output ROMs-Sorted/ - ``` - -=== ":simple-linux: Linux" - - ```shell - igir copy zip fixdat \ - --dat "Nintendo - Game Boy.dat" \ - --dat "Nintendo - Game Boy Advance.dat" \ - --dat "Nintendo - Game Boy Color.dat" \ - --input ROMs/ \ - --output ROMs-Sorted/ - ``` - -may produce some fixdats in the `ROMs-Sorted/` directory, if any of the input DATs have ROMs that weren't found in the `ROMs/` input directory: - -```text -ROMs-Sorted/ -├── Nintendo - Game Boy (20230414-173400) fixdat.dat -├── Nintendo - Game Boy Advance (20230414-173400) fixdat.dat -└── Nintendo - Game Boy Color (20230414-173400) fixdat.dat -``` diff --git a/docs/input/file-scanning.md b/docs/input/file-scanning.md index d6bfe794e..d4fe587af 100644 --- a/docs/input/file-scanning.md +++ b/docs/input/file-scanning.md @@ -3,7 +3,7 @@ `igir` has a few options to specify input files, as well as files to exclude: - ROMs: `--input ` (required), `--input-exclude ` -- [DATs](dats.md): `--dat `, `--dat-exclude ` +- [DATs](../dats/overview.md): `--dat `, `--dat-exclude ` - [ROM patches](../roms/patching.md): `--patch `, `--patch-exclude ` ## Archive files diff --git a/docs/output/cleaning.md b/docs/output/cleaning.md index 45b21ef36..bae9f7a7a 100644 --- a/docs/output/cleaning.md +++ b/docs/output/cleaning.md @@ -2,8 +2,8 @@ The `igir clean` [command](../commands.md) can be used when writing (`igir copy`, `igir move`, and `igir symlink`) to delete files from the `--output ` directory that are either: -- Not contained in any provided [DAT](../input/dats.md) (the `--dat ` option). -- Contained in a [DAT](../input/dats.md) (the `--dat ` option), but the file is in the incorrect location. +- Not contained in any provided [DAT](../dats/overview.md) (the `--dat ` option). +- Contained in a [DAT](../dats/overview.md) (the `--dat ` option), but the file is in the incorrect location. ## The golden rule @@ -11,7 +11,7 @@ The golden rule of the `igir clean` command is it will _not_ delete files in any In practical terms, this means: -**1. If no file was written (i.e. no input file matched any ROM in any [DAT](../input/dats.md)), then `igir clean` will not delete any files.** +**1. If no file was written (i.e. no input file matched any ROM in any [DAT](../dats/overview.md)), then `igir clean` will not delete any files.** **2. If [tokens](tokens.md) are used with the `--output ` option, only subdirectories that are written to will be considered for cleaning.** diff --git a/docs/output/reporting.md b/docs/output/reporting.md index 3e720548b..21cba4e32 100644 --- a/docs/output/reporting.md +++ b/docs/output/reporting.md @@ -10,7 +10,7 @@ When using DATs (the `--dat ` option), the `igir report` [command](../comm - `UNUSED`: what input files didn't match to any ROM - `DELETED`: what output files were [cleaned](cleaning.md) (`igir clean` command) -At least one DAT is required for the `igir report` command to work, otherwise `igir` has no way to understand what input files are known ROMs and which aren't. See the [DAT docs](../input/dats.md) for more information about DATs. +At least one DAT is required for the `igir report` command to work, otherwise `igir` has no way to understand what input files are known ROMs and which aren't. See the [DAT docs](../dats/overview.md) for more information about DATs. The `igir report` command can be specified on its own without any [writing command](../commands.md) (i.e. `igir copy`, `igir move`, etc.) in order to report on an existing collection. This causes `igir` to operate in a _read-only_ mode, no files will be copied, moved, or deleted. For example: diff --git a/docs/output/tokens.md b/docs/output/tokens.md index 143638ffb..e611f9b4d 100644 --- a/docs/output/tokens.md +++ b/docs/output/tokens.md @@ -45,7 +45,7 @@ ROMs-Sorted/ ## DAT information -When using [DATs](../input/dats.md), you can make use of console & game information contained in them: +When using [DATs](../dats/overview.md), you can make use of console & game information contained in them: - `{datName}` the matching DAT's name, similar to how the `--dir-dat-name` option works - `{datDescription}` the matching DAT's description, similar to how the `--dir-dat-description` option works diff --git a/docs/overview.md b/docs/overview.md index 5033a2597..5d6ddec15 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -21,7 +21,7 @@ all additional features help serve these two purposes. Most ROM managers can automatically read & write many different ROM types including those in [archives](input/reading-archives.md) and those with [headers](roms/headers.md) so that you don't have to do much pre-work. -Most ROM managers rely on [DATs](input/dats.md), files that catalog every known ROM that exists per game system. DATs are published by release groups dedicated to keeping these catalogs accurate and up-to-date. DATs help ROM collectors name their ROMs in a consistent way as well as understand what ROMs may be missing from their collection. +Most ROM managers rely on [DATs](dats/overview.md), files that catalog every known ROM that exists per game system. DATs are published by release groups dedicated to keeping these catalogs accurate and up-to-date. DATs help ROM collectors name their ROMs in a consistent way as well as understand what ROMs may be missing from their collection. ## What is `igir`? diff --git a/docs/usage/arcade.md b/docs/usage/arcade.md index 74925cc17..a06061613 100644 --- a/docs/usage/arcade.md +++ b/docs/usage/arcade.md @@ -14,7 +14,7 @@ Unlike traditional console emulators that only have to emulate a small set of ha Due to arcade machines being more complicated and rarer, arcade ROM dumps are sometimes imperfect. Because of this, newer emulator versions may expect different ROM files than older versions. This makes ROM sets potentially incompatible between different emulator versions. -Because of all of these reasons, each arcade emulator version usually comes with a companion [DAT](../input/dats.md) that details the _exact_ set of ROM files supported by that _exact_ emulator version. Emulators such as [MAME](https://www.mamedev.org/) take this a step further and expect an _exact_ zip file name for each game. +Because of all of these reasons, each arcade emulator version usually comes with a companion [DAT](../dats/overview.md) that details the _exact_ set of ROM files supported by that _exact_ emulator version. Emulators such as [MAME](https://www.mamedev.org/) take this a step further and expect an _exact_ zip file name for each game. !!! danger @@ -27,7 +27,7 @@ Here is a chart of instructions for various setups: | Emulator | How to get DATs | Alternatives | |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Frontends ([Batocera](desktop/batocera.md), [EmulationStation](desktop/emulationstation.md), [Lakka](desktop/lakka.md), [Recalbox](desktop/recalbox.md), [RetroArch](desktop/retroarch.md), [RetroPie](desktop/retropie.md), etc.) | Each frontend's documentation should have instructions or links to download the appropriate DAT(s). For example, [RetroArch's arcade docs](https://docs.libretro.com/guides/arcade-getting-started/#step-3-use-the-correct-version-romsets-for-that-emulator) links to the exact DAT needed for each arcade core. | N/A | -| [MAME](https://www.mamedev.org/) | The easiest way to ensure you're using _exactly_ the right DAT for your MAME version is to provide the executable as `--dat mame`. See the [DATs page](../input/dats.md) for more information. | A standalone download of the latest MAME ListXML can be found on the [official site](https://www.mamedev.org/release.html). See the [DATs page](../input/dats.md) for other alternatives. | +| [MAME](https://www.mamedev.org/) | The easiest way to ensure you're using _exactly_ the right DAT for your MAME version is to provide the executable as `--dat mame`. See the [DATs page](../dats/overview.md) for more information. | A standalone download of the latest MAME ListXML can be found on the [official site](https://www.mamedev.org/release.html). See the [DATs page](../dats/overview.md) for other alternatives. | | [FinalBurn Neo](https://github.com/finalburnneo/FBNeo) | FinalBurn Neo doesn't provide an obvious way to find the correct DAT for each version. But it is likely that you are using FinalBurn Neo through a frontend, so use the above instructions. | N/A | | [FinalBurn Alpha](https://www.fbalpha.com/) | FinalBurn Alpha was forked into FinalBurn Neo, so you should use that if possible. Otherwise, hopefully your frontend's documentation has links to download the correct DAT. | N/A | @@ -35,7 +35,7 @@ Here is a chart of instructions for various setups: There are three broadly accepted types of ROM sets, with one extra variation, resulting in four types. -First, you will want to read up on [parent/clone](../input/dats.md#parentclone-pc-dats) sets and how DATs catalog them. +First, you will want to read up on [parent/clone](../dats/overview.md#parentclone-pc-dats) sets and how DATs catalog them. Here is a comparison chart: diff --git a/docs/usage/collection-sorting.md b/docs/usage/collection-sorting.md index d613f95ac..d2b6c6250 100644 --- a/docs/usage/collection-sorting.md +++ b/docs/usage/collection-sorting.md @@ -8,7 +8,7 @@ A walkthrough of an example way to sort your ROM collection. ## First time collection sort -First, you need to download a set of [DATs](../input/dats.md). For these examples I'll assume you downloaded a No-Intro daily P/C XML `.zip`. +First, you need to download a set of [DATs](../dats/overview.md). For these examples I'll assume you downloaded a No-Intro daily P/C XML `.zip`. Let's say that you have a directory named `ROMs/` that contains ROMs for many different systems, and it needs some organization. To make sure we're alright with the output, we'll have `igir` copy these files rather than move them. We'll also zip them to reduce disk space & speed up future scans. diff --git a/docs/usage/desktop/retroarch.md b/docs/usage/desktop/retroarch.md index af9ba9c12..50c130cdb 100644 --- a/docs/usage/desktop/retroarch.md +++ b/docs/usage/desktop/retroarch.md @@ -10,7 +10,7 @@ First, RetroArch needs a number of [BIOS files](https://docs.libretro.com/library/bios/). Thankfully, the libretro team maintains a DAT of these "system" files, so we don't have to guess at the correct filenames. -With `igir`'s support for [DAT URLs](../../input/dats.md) we don't even have to download the DAT! Locate your "System/BIOS" directory as configured in the RetroArch UI and use it as your output directory: +With `igir`'s support for [DAT URLs](../../dats/overview.md) we don't even have to download the DAT! Locate your "System/BIOS" directory as configured in the RetroArch UI and use it as your output directory: === ":simple-windowsxp: Windows (64-bit)" diff --git a/docs/usage/hardware/everdrive.md b/docs/usage/hardware/everdrive.md index 3088daca1..f73c729c7 100644 --- a/docs/usage/hardware/everdrive.md +++ b/docs/usage/hardware/everdrive.md @@ -2,7 +2,7 @@ ## ROMs -Because flash carts are specific to a specific console, you can provide specific input directories & [DATs](../../input/dats.md) when you run `igir`. For example: +Because flash carts are specific to a specific console, you can provide specific input directories & [DATs](../../dats/overview.md) when you run `igir`. For example: === ":simple-windowsxp: Windows" @@ -42,7 +42,7 @@ Because flash carts are specific to a specific console, you can provide specific you can then add some other output options such as `--dir-letter`, if desired. -Alternatively, `igir` supports [Hardware Target Game Database SMDB files](https://github.com/frederic-mahe/Hardware-Target-Game-Database/tree/master/EverDrive%20Pack%20SMDBs) as [DATs](../../input/dats.md). Unlike typical DATs, Hardware Target Game Database SMDBs typically have an opinionated directory structure to help sort ROMs by language, category, genre, and more. Example usage: +Alternatively, `igir` supports [Hardware Target Game Database SMDB files](https://github.com/frederic-mahe/Hardware-Target-Game-Database/tree/master/EverDrive%20Pack%20SMDBs) as [DATs](../../dats/overview.md). Unlike typical DATs, Hardware Target Game Database SMDBs typically have an opinionated directory structure to help sort ROMs by language, category, genre, and more. Example usage: === ":simple-windowsxp: Windows" diff --git a/docs/usage/personal.md b/docs/usage/personal.md index bec49983f..6dbc376d8 100644 --- a/docs/usage/personal.md +++ b/docs/usage/personal.md @@ -38,7 +38,7 @@ The file tree in that hard drive looks like this: └── igir_library_sync.sh ``` -The root directory has a DAT zip and subdirectory for each [DAT](../input/dats.md) release group. This helps separate differing quality of DATs and different DAT group ROM naming schemes. I then have one subdirectory for each game console, using the `--dir-dat-name` option. +The root directory has a DAT zip and subdirectory for each [DAT](../dats/overview.md) release group. This helps separate differing quality of DATs and different DAT group ROM naming schemes. I then have one subdirectory for each game console, using the `--dir-dat-name` option. The `igir_library_sync.sh` script helps me keep this collection organized and merge new ROMs into it. The complete source is: diff --git a/mkdocs.yml b/mkdocs.yml index 165b2dbc6..76e8a3ca1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -76,9 +76,12 @@ nav: - usage/console/ps2.md - usage/arcade.md - usage/personal.md + - DATs: + - dats/overview.md + - dats/fixdats.md + - dats/dir2dat.md - File Inputs: - input/file-scanning.md - - input/dats.md - input/reading-archives.md - ROM Processing: - roms/filtering-preferences.md @@ -119,10 +122,11 @@ plugins: - redirects: redirect_maps: 'archives.md': 'input/reading-archives.md' - 'dats.md': 'input/dats.md' + 'dats.md': 'dats/overview.md' 'examples.md': 'usage/collection-sorting.md' 'file-scanning.md': 'input/file-scanning.md' 'input/archives.md': 'input/reading-archives.md' + 'input/dats.md': 'dats/overview.md' 'output/arcade.md': 'usage/arcade.md' 'reporting.md': 'output/reporting.md' 'rom-filtering.md': 'roms/filtering-preferences.md' diff --git a/src/console/logger.ts b/src/console/logger.ts index 5afa3f07b..511c4623b 100644 --- a/src/console/logger.ts +++ b/src/console/logger.ts @@ -143,7 +143,7 @@ export default class Logger { .replace(/^(Usage:.+)/, chalk.bold('$1')) .replace(/(\[commands\.*\])/g, chalk.magenta('$1')) - .replace(new RegExp(`(${Constants.COMMAND_NAME}) (( ?[a-z])+)`, 'g'), `$1 ${chalk.magenta('$2')}`) + .replace(new RegExp(`(${Constants.COMMAND_NAME}) (( ?[a-z0-9])+)`, 'g'), `$1 ${chalk.magenta('$2')}`) .replace(/(\[options\.*\])/g, chalk.cyan('$1')) .replace(/([^a-zA-Z0-9-])(-[a-zA-Z0-9]+)/g, `$1${chalk.cyanBright('$2')}`) diff --git a/src/igir.ts b/src/igir.ts index ce9886597..00e4c72b1 100644 --- a/src/igir.ts +++ b/src/igir.ts @@ -18,6 +18,7 @@ import DATGameInferrer from './modules/datGameInferrer.js'; import DATMergerSplitter from './modules/datMergerSplitter.js'; import DATParentInferrer from './modules/datParentInferrer.js'; import DATScanner from './modules/datScanner.js'; +import Dir2DatCreator from './modules/dir2DatCreator.js'; import DirectoryCleaner from './modules/directoryCleaner.js'; import FileIndexer from './modules/fileIndexer.js'; import FixdatCreator from './modules/fixdatCreator.js'; @@ -75,7 +76,7 @@ export default class Igir { // Set up progress bar and input for DAT processing const datProcessProgressBar = await this.logger.addProgressBar(chalk.underline('Processing DATs'), ProgressBarSymbol.NONE, dats.length); if (dats.length === 0) { - dats = new DATGameInferrer(datProcessProgressBar).infer(roms); + dats = new DATGameInferrer(this.options, datProcessProgressBar).infer(roms); } const datsToWrittenFiles = new Map(); @@ -122,9 +123,19 @@ export default class Igir { .map((romWithFiles) => romWithFiles.getOutputFile())); datsToWrittenFiles.set(filteredDat, writtenRoms); + // Write a dir2dat + const dir2DatPath = await new Dir2DatCreator(this.options, progressBar) + .create(filteredDat); + if (dir2DatPath) { + datsToWrittenFiles.set(filteredDat, [ + ...(datsToWrittenFiles.get(filteredDat) ?? []), + await File.fileOf(dir2DatPath), + ]); + } + // Write a fixdat const fixdatPath = await new FixdatCreator(this.options, progressBar) - .write(filteredDat, parentsToCandidates); + .create(filteredDat, parentsToCandidates); if (fixdatPath) { datsToWrittenFiles.set(filteredDat, [ ...(datsToWrittenFiles.get(filteredDat) ?? []), @@ -133,9 +144,14 @@ export default class Igir { } // Write the output report - const datStatus = await new StatusGenerator(this.options, progressBar) + const datStatus = new StatusGenerator(this.options, progressBar) .generate(filteredDat, parentsToCandidates); datsStatuses.push(datStatus); + await progressBar.done([ + datStatus.toConsole(this.options), + dir2DatPath ? `dir2dat: ${dir2DatPath}` : undefined, + fixdatPath ? `Fixdat: ${fixdatPath}` : undefined, + ].filter((line) => line).join('\n')); // Progress bar cleanup const totalReleaseCandidates = [...parentsToCandidates.values()] @@ -167,6 +183,9 @@ export default class Igir { } private async processDATScanner(): Promise { + if (this.options.shouldDir2Dat()) { + return []; + } if (!this.options.usingDats()) { this.logger.warn('No DAT files provided, consider using some for the best results!'); return []; diff --git a/src/modules/argumentsParser.ts b/src/modules/argumentsParser.ts index d03eeea2c..df28985ac 100644 --- a/src/modules/argumentsParser.ts +++ b/src/modules/argumentsParser.ts @@ -2,6 +2,7 @@ import yargs, { Argv } from 'yargs'; import Logger from '../console/logger.js'; import Constants from '../constants.js'; +import ArrayPoly from '../polyfill/arrayPoly.js'; import ConsolePoly from '../polyfill/consolePoly.js'; import ROMHeader from '../types/files/romHeader.js'; import Internationalization from '../types/internationalization.js'; @@ -67,63 +68,78 @@ export default class ArgumentsParser { // Add every command to a yargs object, recursively, resulting in the ability to specify // multiple commands - const addCommands = (yargsObj: Argv): Argv => yargsObj - .command('copy', 'Copy ROM files from the input to output directory', (yargsSubObj) => { - addCommands(yargsSubObj); - }) - .command('move', 'Move ROM files from the input to output directory', (yargsSubObj) => { - addCommands(yargsSubObj); - }) - .command('symlink', 'Create symlinks in the output directory to ROM files in the input directory', (yargsSubObj) => { - addCommands(yargsSubObj); - }) - .command('extract', 'Extract ROM files in archives when copying or moving', (yargsSubObj) => { - addCommands(yargsSubObj); - }) - .command('zip', 'Create zip archives of ROMs when copying or moving', (yargsSubObj) => { - addCommands(yargsSubObj); - }) - .command('test', 'Test ROMs for accuracy after writing them to the output directory', (yargsSubObj) => { - addCommands(yargsSubObj); - }) - .command('fixdat', 'Generate a fixdat of any missing games for every DAT processed (requires --dat)', (yargsSubObj) => { - addCommands(yargsSubObj); - }) - .command('clean', 'Recycle unknown files in the output directory', (yargsSubObj) => { - addCommands(yargsSubObj); - }) - .command('report', 'Generate a CSV report on the known & unknown ROM files found in the input directories (requires --dat)', (yargsSubObj) => { - addCommands(yargsSubObj); - }) - .check((checkArgv) => { - if (checkArgv.help) { - return true; - } + const commands = [ + ['copy', 'Copy ROM files from the input to output directory'], + ['move', 'Move ROM files from the input to output directory'], + ['symlink', 'Create symlinks in the output directory to ROM files in the input directory'], + ['extract', 'Extract ROM files in archives when copying or moving'], + ['zip', 'Create zip archives of ROMs when copying or moving'], + ['test', 'Test ROMs for accuracy after writing them to the output directory'], + ['dir2dat', 'Generate a DAT from all input files'], + ['fixdat', 'Generate a fixdat of any missing games for every DAT processed (requires --dat)'], + ['clean', 'Recycle unknown files in the output directory'], + ['report', 'Generate a CSV report on the known & unknown ROM files found in the input directories (requires --dat)'], + ]; + const addCommands = ( + yargsObj: Argv, + commandsToAdd = commands.map((command) => command[0]), + ): Argv => { + commands + // Don't show duplicate commands, i.e. don't give `igir copy copy` as an option when + // specifying `igir copy --help`. + .filter(([command]) => commandsToAdd.includes(command)) + .forEach(([command, description]) => { + yargsObj.command(command, description, (yargsSubObj) => addCommands( + yargsSubObj, + commandsToAdd.filter((c) => c !== command), + )); + }); - const writeCommands = ['copy', 'move', 'symlink'].filter((command) => checkArgv._.includes(command)); - if (writeCommands.length > 1) { - throw new Error(`Incompatible commands: ${writeCommands.join(', ')}`); - } + if (commandsToAdd.length === 0) { + // Only register the check function once + return yargsObj; + } + return yargsObj + .middleware((middlewareArgv) => { + /* eslint-disable no-param-reassign */ + // Ignore duplicate commands + middlewareArgv._ = middlewareArgv._.reduce(ArrayPoly.reduceUnique(), []); + }, true) + .check((checkArgv) => { + if (checkArgv.help) { + return true; + } - const archiveCommands = ['symlink', 'extract', 'zip'].filter((command) => checkArgv._.includes(command)); - if (archiveCommands.length > 1) { - throw new Error(`Incompatible commands: ${archiveCommands.join(', ')}`); - } + const writeCommands = ['copy', 'move', 'symlink'].filter((command) => checkArgv._.includes(command)); + if (writeCommands.length > 1) { + throw new Error(`Incompatible commands: ${writeCommands.join(', ')}`); + } - ['extract', 'zip'].forEach((command) => { - if (checkArgv._.includes(command) && ['copy', 'move'].every((write) => !checkArgv._.includes(write))) { - throw new Error(`Command "${command}" also requires the commands copy or move`); + const archiveCommands = ['symlink', 'extract', 'zip'].filter((command) => checkArgv._.includes(command)); + if (archiveCommands.length > 1) { + throw new Error(`Incompatible commands: ${archiveCommands.join(', ')}`); } - }); - ['test', 'clean'].forEach((command) => { - if (checkArgv._.includes(command) && ['copy', 'move', 'symlink'].every((write) => !checkArgv._.includes(write))) { - throw new Error(`Command "${command}" requires one of the commands: copy, move, or symlink`); + const datWritingCommands = ['dir2dat', 'fixdat'].filter((command) => checkArgv._.includes(command)); + if (datWritingCommands.length > 1) { + throw new Error(`Incompatible commands: ${datWritingCommands.join(', ')}`); } - }); - return true; - }); + ['extract', 'zip'].forEach((command) => { + if (checkArgv._.includes(command) && ['copy', 'move'].every((write) => !checkArgv._.includes(write))) { + throw new Error(`Command "${command}" also requires the commands copy or move`); + } + }); + + ['test', 'clean'].forEach((command) => { + if (checkArgv._.includes(command) && ['copy', 'move', 'symlink'].every((write) => !checkArgv._.includes(write))) { + throw new Error(`Command "${command}" requires one of the commands: copy, move, or symlink`); + } + }); + + return true; + }); + }; const yargsParser = yargs([]) .parserConfiguration({ diff --git a/src/modules/candidateGenerator.ts b/src/modules/candidateGenerator.ts index f81822f19..02d069441 100644 --- a/src/modules/candidateGenerator.ts +++ b/src/modules/candidateGenerator.ts @@ -270,7 +270,6 @@ export default class CandidateGenerator extends Module { if (inputFile.getFileHeader() && this.options.canRemoveHeader(dat, path.extname(outputPathParsed.entryPath)) ) { - // TODO(cemmer): inputFile.getSizeWithoutHeader() ? outputFileCrc = inputFile.getCrc32WithoutHeader(); outputFileSize = inputFile.getSizeWithoutHeader(); } diff --git a/src/modules/datGameInferrer.ts b/src/modules/datGameInferrer.ts index 195b97fed..3126b8261 100644 --- a/src/modules/datGameInferrer.ts +++ b/src/modules/datGameInferrer.ts @@ -1,6 +1,10 @@ import path from 'node:path'; +import moment from 'moment'; + import ProgressBar from '../console/progressBar.js'; +import Constants from '../constants.js'; +import ArrayPoly from '../polyfill/arrayPoly.js'; import DAT from '../types/dats/dat.js'; import Game from '../types/dats/game.js'; import Header from '../types/dats/logiqx/header.js'; @@ -8,6 +12,7 @@ import LogiqxDAT from '../types/dats/logiqx/logiqxDat.js'; import ROM from '../types/dats/rom.js'; import ArchiveEntry from '../types/files/archives/archiveEntry.js'; import File from '../types/files/file.js'; +import Options from '../types/options.js'; import Module from './module.js'; /** @@ -18,8 +23,13 @@ import Module from './module.js'; * This class will not be run concurrently with any other class. */ export default class DATGameInferrer extends Module { - constructor(progressBar: ProgressBar) { + private static readonly DEFAULT_DAT_NAME = Constants.COMMAND_NAME; + + private readonly options: Options; + + constructor(options: Options, progressBar: ProgressBar) { super(progressBar, DATGameInferrer.name); + this.options = options; } /** @@ -28,28 +38,48 @@ export default class DATGameInferrer extends Module { infer(romFiles: File[]): DAT[] { this.progressBar.logInfo(`inferring DATs for ${romFiles.length.toLocaleString()} ROM${romFiles.length !== 1 ? 's' : ''}`); - const datNamesToRomFiles = romFiles.reduce((map, file) => { - const datName = DATGameInferrer.getDatName(file); - const datRomFiles = map.get(datName) ?? []; - datRomFiles.push(file); - map.set(datName, datRomFiles); + const normalizedInputPaths = this.options.getInputPaths() + .map((inputPath) => path.normalize(inputPath)) + // Try to strip out glob patterns + .map((inputPath) => inputPath.replace(/([\\/][?*]+)+$/, '')); + + const inputPathsToRomFiles = romFiles.reduce((map, file) => { + const normalizedPath = file.getFilePath().normalize(); + const matchedInputPaths = normalizedInputPaths + // `.filter()` rather than `.find()` because a file can be found in overlapping input paths, + // therefore it should be counted in both + .filter((inputPath) => normalizedPath.startsWith(inputPath)); + (matchedInputPaths.length > 0 ? matchedInputPaths : [DATGameInferrer.DEFAULT_DAT_NAME]) + .forEach((inputPath) => { + const datRomFiles = [...(map.get(inputPath) ?? []), file]; + map.set(inputPath, datRomFiles); + }); return map; }, new Map()); - this.progressBar.logDebug(`inferred ${datNamesToRomFiles.size.toLocaleString()} DAT${datNamesToRomFiles.size !== 1 ? 's' : ''}`); + this.progressBar.logDebug(`inferred ${inputPathsToRomFiles.size.toLocaleString()} DAT${inputPathsToRomFiles.size !== 1 ? 's' : ''}`); - const dats = [...datNamesToRomFiles.entries()] - .map(([datName, datRomFiles]) => DATGameInferrer.createDAT(datName, datRomFiles)); + const dats = [...inputPathsToRomFiles.entries()] + .map(([inputPath, datRomFiles]) => DATGameInferrer.createDAT(inputPath, datRomFiles)); this.progressBar.logInfo('done inferring DATs'); return dats; } - private static getDatName(file: File): string { - return path.dirname(file.getFilePath()).split(/[\\/]/).at(-1) ?? file.getFilePath(); - } - - private static createDAT(datName: string, romFiles: File[]): DAT { - const header = new Header({ name: datName }); + private static createDAT(inputPath: string, romFiles: File[]): DAT { + const datName = path.basename(inputPath); + const date = moment().format('YYYYMMDD-HHmmss'); + const header = new Header({ + name: datName, + description: datName, + version: date, + date, + author: Constants.COMMAND_NAME, + url: Constants.HOMEPAGE, + comment: [ + `dir2dat generated by ${Constants.COMMAND_NAME} v${Constants.COMMAND_VERSION}`, + `Input path: ${inputPath}`, + ].join('\n'), + }); const gameNamesToRomFiles = romFiles.reduce((map, file) => { const gameName = DATGameInferrer.getGameName(file); @@ -60,13 +90,18 @@ export default class DATGameInferrer extends Module { }, new Map()); const games = [...gameNamesToRomFiles.entries()].map(([gameName, gameRomFiles]) => { - const roms = gameRomFiles.map((romFile) => new ROM({ - name: path.basename(romFile.getExtractedFilePath()), - size: romFile.getSize(), - crc: romFile.getCrc32(), - })); + const roms = gameRomFiles + .map((romFile) => new ROM({ + name: path.basename(romFile.getExtractedFilePath()), + size: romFile.getSize(), + crc: romFile.getCrc32(), + md5: romFile.getMd5(), + sha1: romFile.getSha1(), + })) + .filter(ArrayPoly.filterUniqueMapped((rom) => rom.getName())); return new Game({ name: gameName, + description: gameName, rom: roms, }); }); @@ -83,6 +118,7 @@ export default class DATGameInferrer extends Module { } return path.basename(fileName) + // Chop off the extension .replace(/(\.[a-z0-9]+)+$/, '') .trim(); } diff --git a/src/modules/datScanner.ts b/src/modules/datScanner.ts index 57c5f6b39..5f5fa20d5 100644 --- a/src/modules/datScanner.ts +++ b/src/modules/datScanner.ts @@ -18,6 +18,7 @@ import LogiqxDAT from '../types/dats/logiqx/logiqxDat.js'; import MameDAT from '../types/dats/mame/mameDat.js'; import ROM from '../types/dats/rom.js'; import File from '../types/files/file.js'; +import { ChecksumBitmask } from '../types/files/fileChecksums.js'; import FileFactory from '../types/files/fileFactory.js'; import Options from '../types/options.js'; import Scanner from './scanner.js'; @@ -60,7 +61,11 @@ export default class DATScanner extends Scanner { await this.progressBar.reset(datFilePaths.length); this.progressBar.logDebug('enumerating DAT archives'); - const datFiles = await this.getFilesFromPaths(datFilePaths, this.options.getReaderThreads()); + const datFiles = await this.getUniqueFilesFromPaths( + datFilePaths, + this.options.getReaderThreads(), + ChecksumBitmask.NONE, + ); await this.progressBar.reset(datFiles.length); const downloadedDats = await this.downloadDats(datFiles); @@ -87,7 +92,7 @@ export default class DATScanner extends Scanner { this.progressBar.logTrace(`${datFile.toString()}: downloading`); const downloadedDatFile = await datFile.downloadToTempPath('dat'); this.progressBar.logTrace(`${datFile.toString()}: downloaded to ${downloadedDatFile.toString()}`); - return await FileFactory.filesFrom(downloadedDatFile.getFilePath()); + return await FileFactory.filesFrom(downloadedDatFile.getFilePath(), ChecksumBitmask.NONE); } catch (error) { this.progressBar.logWarn(`${datFile.toString()}: failed to download: ${error}`); return []; @@ -120,7 +125,7 @@ export default class DATScanner extends Scanner { }, )) .filter(ArrayPoly.filterNotNullish) - .map((dat) => DATScanner.sanitizeDat(dat)); + .map((dat) => this.sanitizeDat(dat)); return results .filter((dat) => { @@ -440,12 +445,12 @@ export default class DATScanner extends Scanner { }); } - private static sanitizeDat(dat: DAT): DAT { + private sanitizeDat(dat: DAT): DAT { const games = dat.getGames() .map((game) => { const roms = game.getRoms() // ROMs have to have filenames and sizes - .filter((rom) => rom.name && rom.size > 0); + .filter((rom) => this.options.shouldDir2Dat() || (rom.name && rom.size > 0)); return game.withProps({ rom: roms }); }); diff --git a/src/modules/dir2DatCreator.ts b/src/modules/dir2DatCreator.ts new file mode 100644 index 000000000..eef278987 --- /dev/null +++ b/src/modules/dir2DatCreator.ts @@ -0,0 +1,44 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import util from 'node:util'; + +import ProgressBar, { ProgressBarSymbol } from '../console/progressBar.js'; +import DAT from '../types/dats/dat.js'; +import Options from '../types/options.js'; +import OutputFactory from '../types/outputFactory.js'; +import Module from './module.js'; + +/** + * Write a DAT that was generated by {@link DATGameInferrer} to disk. + */ +export default class Dir2DatCreator extends Module { + private readonly options: Options; + + constructor(options: Options, progressBar: ProgressBar) { + super(progressBar, Dir2DatCreator.name); + this.options = options; + } + + /** + * Write the DAT. + */ + async create(dat: DAT): Promise { + if (!this.options.shouldDir2Dat()) { + return undefined; + } + + this.progressBar.logInfo(`${dat.getNameShort()}: writing dir2dat`); + await this.progressBar.setSymbol(ProgressBarSymbol.WRITING); + await this.progressBar.reset(1); + + const datDir = this.options.shouldWrite() + ? OutputFactory.getDir(this.options, dat) + : process.cwd(); + const datContents = dat.toXmlDat(); + const datPath = path.join(datDir, dat.getFilename()); + await util.promisify(fs.writeFile)(datPath, datContents); + + this.progressBar.logInfo(`${dat.getNameShort()}: done writing dir2dat`); + return datPath; + } +} diff --git a/src/modules/fixdatCreator.ts b/src/modules/fixdatCreator.ts index 4404a2ea9..53eaf560a 100644 --- a/src/modules/fixdatCreator.ts +++ b/src/modules/fixdatCreator.ts @@ -30,7 +30,7 @@ export default class FixdatCreator extends Module { /** * Create & write a fixdat. */ - async write( + async create( originalDat: DAT, parentsToCandidates: Map, ): Promise { @@ -67,7 +67,6 @@ export default class FixdatCreator extends Module { description: `${originalDat.getHeader().getDescription()} fixdat`.trim(), version: date, date, - author: Constants.AUTHOR, url: Constants.HOMEPAGE, comment: [ `fixdat generated by ${Constants.COMMAND_NAME} v${Constants.COMMAND_VERSION}`, @@ -84,7 +83,6 @@ export default class FixdatCreator extends Module { await util.promisify(fs.writeFile)(fixdatPath, fixdatContents); this.progressBar.logInfo(`${originalDat.getNameShort()}: done generating a fixdat`); - return fixdatPath; } } diff --git a/src/modules/patchScanner.ts b/src/modules/patchScanner.ts index 93e812a0b..650b6a835 100644 --- a/src/modules/patchScanner.ts +++ b/src/modules/patchScanner.ts @@ -2,6 +2,7 @@ import ProgressBar, { ProgressBarSymbol } from '../console/progressBar.js'; import DriveSemaphore from '../driveSemaphore.js'; import ArrayPoly from '../polyfill/arrayPoly.js'; import File from '../types/files/file.js'; +import { ChecksumBitmask } from '../types/files/fileChecksums.js'; import Options from '../types/options.js'; import Patch from '../types/patches/patch.js'; import PatchFactory from '../types/patches/patchFactory.js'; @@ -31,9 +32,10 @@ export default class PatchScanner extends Scanner { this.progressBar.logDebug(`found ${patchFilePaths.length.toLocaleString()} patch file${patchFilePaths.length !== 1 ? 's' : ''}`); await this.progressBar.reset(patchFilePaths.length); - const files = await this.getFilesFromPaths( + const files = await this.getUniqueFilesFromPaths( patchFilePaths, this.options.getReaderThreads(), + ChecksumBitmask.NONE, ); const patches = (await new DriveSemaphore(this.options.getReaderThreads()).map( diff --git a/src/modules/romScanner.ts b/src/modules/romScanner.ts index 0388d9d54..5cc651065 100644 --- a/src/modules/romScanner.ts +++ b/src/modules/romScanner.ts @@ -1,5 +1,6 @@ import ProgressBar, { ProgressBarSymbol } from '../console/progressBar.js'; import File from '../types/files/file.js'; +import { ChecksumBitmask } from '../types/files/fileChecksums.js'; import Options from '../types/options.js'; import Scanner from './scanner.js'; @@ -28,11 +29,12 @@ export default class ROMScanner extends Scanner { this.progressBar.logDebug(`found ${romFilePaths.length.toLocaleString()} ROM file${romFilePaths.length !== 1 ? 's' : ''}`); await this.progressBar.reset(romFilePaths.length); - const filterUnique = false; const files = await this.getFilesFromPaths( romFilePaths, this.options.getReaderThreads(), - filterUnique, + this.options.shouldDir2Dat() + ? ChecksumBitmask.CRC32 | ChecksumBitmask.MD5 | ChecksumBitmask.SHA1 + : ChecksumBitmask.CRC32, ); this.progressBar.logInfo('done scanning ROM files'); diff --git a/src/modules/scanner.ts b/src/modules/scanner.ts index bf31db202..7ce665bf0 100644 --- a/src/modules/scanner.ts +++ b/src/modules/scanner.ts @@ -2,6 +2,7 @@ import ProgressBar from '../console/progressBar.js'; import Constants from '../constants.js'; import DriveSemaphore from '../driveSemaphore.js'; import ElasticSemaphore from '../elasticSemaphore.js'; +import ArrayPoly from '../polyfill/arrayPoly.js'; import fsPoly from '../polyfill/fsPoly.js'; import File from '../types/files/file.js'; import FileFactory from '../types/files/fileFactory.js'; @@ -28,43 +29,42 @@ export default abstract class Scanner extends Module { protected async getFilesFromPaths( filePaths: string[], threads: number, - filterUnique = true, + checksumBitmask: number, ): Promise { - const foundFiles = (await new DriveSemaphore(threads).map( + return (await new DriveSemaphore(threads).map( filePaths, async (inputFile) => { await this.progressBar.incrementProgress(); const waitingMessage = `${inputFile} ...`; this.progressBar.addWaitingMessage(waitingMessage); - const files = await this.getFilesFromPath(inputFile); + const files = await this.getFilesFromPath(inputFile, checksumBitmask); this.progressBar.removeWaitingMessage(waitingMessage); await this.progressBar.incrementDone(); return files; }, - )) - .flat(); - if (!filterUnique) { - return foundFiles; - } + )).flat(); + } - // Limit to unique files - return [...foundFiles - .reduce((map, file) => { - const hashCodes = file.hashCodes().join(','); - if (!map.has(hashCodes)) { - map.set(hashCodes, file); - } - return map; - }, new Map()).values()]; + protected async getUniqueFilesFromPaths( + filePaths: string[], + threads: number, + checksumBitmask: number, + ): Promise { + const foundFiles = await this.getFilesFromPaths(filePaths, threads, checksumBitmask); + return foundFiles + .filter(ArrayPoly.filterUniqueMapped((file) => file.hashCodes().join(','))); } - private async getFilesFromPath(filePath: string): Promise { + private async getFilesFromPath( + filePath: string, + checksumBitmask: number, + ): Promise { try { const totalKilobytes = await fsPoly.size(filePath) / 1024; const files = await Scanner.FILESIZE_SEMAPHORE.runExclusive( - async () => FileFactory.filesFrom(filePath), + async () => FileFactory.filesFrom(filePath, checksumBitmask), totalKilobytes, ); diff --git a/src/modules/statusGenerator.ts b/src/modules/statusGenerator.ts index e6a4f8f3d..a0be94740 100644 --- a/src/modules/statusGenerator.ts +++ b/src/modules/statusGenerator.ts @@ -22,16 +22,14 @@ export default class StatusGenerator extends Module { /** * Generate a {@link DATStatus} for the {@link DAT}. */ - async generate( + generate( dat: DAT, parentsToReleaseCandidates: Map, - ): Promise { + ): DATStatus { this.progressBar.logInfo(`${dat.getNameShort()}: generating ROM statuses`); const datStatus = new DATStatus(dat, this.options, parentsToReleaseCandidates); - await this.progressBar.done(datStatus.toConsole(this.options)); - this.progressBar.logInfo(`${dat.getNameShort()}: done generating ROM statuses`); return datStatus; } diff --git a/src/types/dats/dat.ts b/src/types/dats/dat.ts index f10914dbf..6db194dfc 100644 --- a/src/types/dats/dat.ts +++ b/src/types/dats/dat.ts @@ -1,3 +1,6 @@ +import xml2js from 'xml2js'; + +import FsPoly from '../../polyfill/fsPoly.js'; import Game from './game.js'; import Header from './logiqx/header.js'; import Parent from './parent.js'; @@ -93,7 +96,7 @@ export default abstract class DAT { filename += ` (${this.getHeader().getVersion()})`; } filename += '.dat'; - return filename.trim(); + return FsPoly.makeLegal(filename.trim()); } /** @@ -134,7 +137,32 @@ export default abstract class DAT { } /** - * Return a short string representation of this {@link LogiqxDAT}. + * Serialize this {@link DAT} to the file contents of an XML file. + */ + toXmlDat(): string { + return new xml2js.Builder({ + renderOpts: { pretty: true, indent: '\t', newline: '\n' }, + xmldec: { version: '1.0' }, + doctype: { + pubID: '-//Logiqx//DTD ROM Management Datafile//EN', + sysID: 'http://www.logiqx.com/Dats/datafile.dtd', + }, + cdata: true, + }).buildObject(this.toXmlDatObj()); + } + + private toXmlDatObj(): object { + const parentNames = new Set(this.getParents().map((parent) => parent.getName())); + return { + datafile: { + header: this.getHeader().toXmlDatObj(), + game: this.getGames().map((game) => game.toXmlDatObj(parentNames)), + }, + }; + } + + /** + * Return a short string representation of this {@link DAT}. */ toString(): string { return `{"header": ${this.getHeader().toString()}, "games": ${this.getGames().length}}`; diff --git a/src/types/dats/game.ts b/src/types/dats/game.ts index 825aeb338..518b9b253 100644 --- a/src/types/dats/game.ts +++ b/src/types/dats/game.ts @@ -135,11 +135,18 @@ export default class Game implements GameProps { /** * Create an XML object, to be used by the owning {@link DAT}. */ - toXmlDatObj(): object { + toXmlDatObj(parentNames: Set): object { return { $: { name: this.getName(), - // NOTE(cemmer): explicitly not including `cloneof` + isbios: this.isBios() ? 'yes' : undefined, + isdevice: this.isDevice() ? 'yes' : undefined, + cloneof: this.getParent() && parentNames.has(this.getParent()) + ? this.getParent() + : undefined, + romof: this.getBios() && parentNames.has(this.getBios()) + ? this.getBios() + : undefined, }, description: { _: this.getDescription(), diff --git a/src/types/dats/logiqx/logiqxDat.ts b/src/types/dats/logiqx/logiqxDat.ts index fe7fc869b..ae35d7141 100644 --- a/src/types/dats/logiqx/logiqxDat.ts +++ b/src/types/dats/logiqx/logiqxDat.ts @@ -3,7 +3,6 @@ import 'reflect-metadata'; import { Expose, plainToInstance, Transform, Type, } from 'class-transformer'; -import xml2js from 'xml2js'; import DAT from '../dat.js'; import Game from '../game.js'; @@ -49,30 +48,6 @@ export default class LogiqxDAT extends DAT { .generateGameNamesToParents(); } - /** - * Serialize this {@link LogiqxDAT} to the file contents of an XML file. - */ - toXmlDat(): string { - return new xml2js.Builder({ - renderOpts: { pretty: true, indent: '\t', newline: '\n' }, - xmldec: { version: '1.0' }, - doctype: { - pubID: '-//Logiqx//DTD ROM Management Datafile//EN', - sysID: 'http://www.logiqx.com/Dats/datafile.dtd', - }, - cdata: true, - }).buildObject(this.toXmlDatObj()); - } - - private toXmlDatObj(): object { - return { - datafile: { - header: this.header.toXmlDatObj(), - game: this.getGames().map((game) => game.toXmlDatObj()), - }, - }; - } - // Property getters getHeader(): Header { diff --git a/src/types/dats/rom.ts b/src/types/dats/rom.ts index f9b8d64dc..3d5dab474 100644 --- a/src/types/dats/rom.ts +++ b/src/types/dats/rom.ts @@ -63,12 +63,12 @@ export default class ROM implements ROMProps { toXmlDatObj(): object { return { $: { - name: this.name, - size: this.size, - crc: this.crc, - md5: this.md5, - sha1: this.sha1, - status: this.status, + name: this.getName(), + size: this.getSize(), + crc: this.getCrc32(), + md5: this.getMd5(), + sha1: this.getSha1(), + status: this.getStatus(), }, }; } @@ -87,12 +87,12 @@ export default class ROM implements ROMProps { return (this.crc ?? '').toLowerCase().replace(/^0x/, '').padStart(8, '0'); } - getMd5(): string { - return (this.md5 ?? '').toLowerCase().replace(/^0x/, '').padStart(32, '0'); + getMd5(): string | undefined { + return this.md5?.toLowerCase().replace(/^0x/, '').padStart(32, '0'); } - getSha1(): string { - return (this.sha1 ?? '').toLowerCase().replace(/^0x/, '').padStart(40, '0'); + getSha1(): string | undefined { + return this.sha1?.toLowerCase().replace(/^0x/, '').padStart(40, '0'); } getChecksumProps(): ChecksumProps { diff --git a/src/types/files/archives/tar.ts b/src/types/files/archives/tar.ts index be59eb2f6..2d4444f41 100644 --- a/src/types/files/archives/tar.ts +++ b/src/types/files/archives/tar.ts @@ -36,8 +36,8 @@ export default class Tar extends Archive { highWaterMark: Constants.FILE_READING_CHUNK_SIZE, }).pipe(writeStream); + // Note: entries are read sequentially, so entry streams need to be fully read or resumed writeStream.on('entry', async (entry) => { - // TODO(cemmer): use ARCHIVE_ENTRY_SCANNER_THREADS const checksums = await FileChecksums.hashStream(entry, checksumBitmask); archiveEntryPromises.push(ArchiveEntry.entryOf( this, @@ -46,6 +46,8 @@ export default class Tar extends Archive { checksums, checksumBitmask, )); + // In case we didn't need to read the stream for hashes, resume the file reading + entry.resume(); }); // Wait for the tar file to be closed diff --git a/src/types/options.ts b/src/types/options.ts index b98d964a3..448825aff 100644 --- a/src/types/options.ts +++ b/src/types/options.ts @@ -502,6 +502,13 @@ export default class Options implements OptionsProps { )); } + /** + * Was the 'dir2dat' command provided? + */ + shouldDir2Dat(): boolean { + return this.getCommands().has('dir2dat'); + } + /** * Was the 'fixdat' command provided? */ diff --git a/test/igir.test.ts b/test/igir.test.ts index f33515420..f95993cb7 100644 --- a/test/igir.test.ts +++ b/test/igir.test.ts @@ -787,8 +787,8 @@ describe('with inferred DATs', () => { [path.join('onetwothree', 'one.rom'), 'f817a89f'], [path.join('onetwothree', 'three.rom'), 'ff46c5d8'], [path.join('onetwothree', 'two.rom'), '96170874'], - ['speed_test_v51.sfc', '8beffd94'], - ['speed_test_v51.smc', '9adca6cc'], + [path.join('speed_test_v51', 'speed_test_v51.sfc'), '8beffd94'], + [path.join('speed_test_v51', 'speed_test_v51.smc'), '9adca6cc'], ['three.rom', 'ff46c5d8'], ['two.rom', '96170874'], ['unknown.rom', '377a7727'], @@ -863,6 +863,7 @@ describe('with inferred DATs', () => { ['onetwothree.zip|one.rom', 'f817a89f'], ['onetwothree.zip|three.rom', 'ff46c5d8'], ['onetwothree.zip|two.rom', '96170874'], + ['speed_test_v51.zip|speed_test_v51.sfc', '8beffd94'], ['speed_test_v51.zip|speed_test_v51.smc', '9adca6cc'], ['three.zip|three.rom', 'ff46c5d8'], ['two.zip|two.rom', '96170874'], @@ -951,4 +952,54 @@ describe('with inferred DATs', () => { expect(result.cleanedFiles).toHaveLength(0); }); }); + + it('should dir2dat', async () => { + await copyFixturesToTemp(async (inputTemp, outputTemp) => { + const result = await runIgir({ + commands: ['dir2dat'], + dat: [path.join(inputTemp, 'dats')], + input: [path.join(inputTemp, 'roms')], + output: outputTemp, + dirDatName: true, + }); + + const writtenDir2Dats = result.cwdFilesAndCrcs + .map(([filePath]) => filePath) + .filter((filePath) => filePath.endsWith('.dat')); + + // Only the "roms" input path was provided + expect(writtenDir2Dats).toHaveLength(1); + expect(writtenDir2Dats[0]).toMatch(/^roms \([0-9]{8}-[0-9]{6}\)\.dat$/); + + expect(result.movedFiles).toHaveLength(0); + expect(result.cleanedFiles).toHaveLength(0); + }); + }); + + test.each([ + 'copy', + 'move', + 'symlink', + ])('should %s, dir2dat, and clean', async (command) => { + await copyFixturesToTemp(async (inputTemp, outputTemp) => { + const result = await runIgir({ + commands: [command, 'dir2dat', 'clean'], + dat: [path.join(inputTemp, 'dats')], + input: [path.join(inputTemp, 'roms')], + output: outputTemp, + dirDatName: true, + }); + + const writtenDir2Dats = result.outputFilesAndCrcs + .map(([filePath]) => filePath) + .filter((filePath) => filePath.endsWith('.dat')); + + // Only the "roms" input path was provided + expect(writtenDir2Dats).toHaveLength(1); + expect(writtenDir2Dats[0]).toMatch(/^roms[\\/]roms \([0-9]{8}-[0-9]{6}\)\.dat$/); + + expect(result.cwdFilesAndCrcs).toHaveLength(0); + expect(result.cleanedFiles).toHaveLength(0); + }); + }); }); diff --git a/test/modules/argumentsParser.test.ts b/test/modules/argumentsParser.test.ts index d70b9b42c..d084f03ab 100644 --- a/test/modules/argumentsParser.test.ts +++ b/test/modules/argumentsParser.test.ts @@ -47,19 +47,21 @@ describe('commands', () => { expect(argumentsParser.parse(['copy', ...dummyRequiredArgs]).shouldExtract()).toEqual(false); expect(argumentsParser.parse(['copy', ...dummyRequiredArgs]).shouldZip('')).toEqual(false); expect(argumentsParser.parse(['copy', ...dummyRequiredArgs]).shouldTest()).toEqual(false); + expect(argumentsParser.parse(['copy', ...dummyRequiredArgs]).shouldDir2Dat()).toEqual(false); expect(argumentsParser.parse(['copy', ...dummyRequiredArgs]).shouldFixdat()).toEqual(false); expect(argumentsParser.parse(['copy', ...dummyRequiredArgs]).shouldClean()).toEqual(false); expect(argumentsParser.parse(['copy', ...dummyRequiredArgs]).shouldReport()).toEqual(false); }); it('should parse multiple commands', () => { - const copyExtract = ['copy', 'extract', 'test', 'fixdat', 'clean', 'report', ...dummyRequiredArgs, '--dat', os.devNull]; + const copyExtract = ['copy', 'extract', 'test', 'dir2dat', 'clean', 'report', ...dummyRequiredArgs, '--dat', os.devNull]; expect(argumentsParser.parse(copyExtract).shouldCopy()).toEqual(true); expect(argumentsParser.parse(copyExtract).shouldMove()).toEqual(false); expect(argumentsParser.parse(copyExtract).shouldExtract()).toEqual(true); expect(argumentsParser.parse(copyExtract).shouldZip('')).toEqual(false); expect(argumentsParser.parse(copyExtract).shouldTest()).toEqual(true); - expect(argumentsParser.parse(copyExtract).shouldFixdat()).toEqual(true); + expect(argumentsParser.parse(copyExtract).shouldDir2Dat()).toEqual(true); + expect(argumentsParser.parse(copyExtract).shouldFixdat()).toEqual(false); expect(argumentsParser.parse(copyExtract).shouldClean()).toEqual(true); expect(argumentsParser.parse(copyExtract).shouldReport()).toEqual(true); @@ -69,6 +71,7 @@ describe('commands', () => { expect(argumentsParser.parse(moveZip).shouldExtract()).toEqual(false); expect(argumentsParser.parse(moveZip).shouldZip('')).toEqual(true); expect(argumentsParser.parse(moveZip).shouldTest()).toEqual(true); + expect(argumentsParser.parse(moveZip).shouldDir2Dat()).toEqual(false); expect(argumentsParser.parse(moveZip).shouldFixdat()).toEqual(true); expect(argumentsParser.parse(moveZip).shouldClean()).toEqual(true); expect(argumentsParser.parse(moveZip).shouldReport()).toEqual(true); @@ -93,6 +96,7 @@ describe('options', () => { expect(options.shouldSymlink()).toEqual(false); expect(options.shouldExtract()).toEqual(false); expect(options.canZip()).toEqual(false); + expect(options.shouldDir2Dat()).toEqual(false); expect(options.shouldFixdat()).toEqual(false); expect(options.shouldTest()).toEqual(false); expect(options.shouldClean()).toEqual(false); diff --git a/test/modules/candidateCombiner.test.ts b/test/modules/candidateCombiner.test.ts index 015f8faac..4524a9415 100644 --- a/test/modules/candidateCombiner.test.ts +++ b/test/modules/candidateCombiner.test.ts @@ -18,7 +18,7 @@ async function runCombinedCandidateGenerator( romFiles: File[], ): Promise> { // Run DATInferrer, but condense all DATs down to one - const datGames = new DATGameInferrer(new ProgressBarFake()).infer(romFiles) + const datGames = new DATGameInferrer(options, new ProgressBarFake()).infer(romFiles) .flatMap((dat) => dat.getGames()); const dat = new LogiqxDAT(new Header(), datGames); diff --git a/test/modules/candidatePatchGenerator.test.ts b/test/modules/candidatePatchGenerator.test.ts index e77640824..221691762 100644 --- a/test/modules/candidatePatchGenerator.test.ts +++ b/test/modules/candidatePatchGenerator.test.ts @@ -17,8 +17,8 @@ import ReleaseCandidate from '../../src/types/releaseCandidate.js'; import ProgressBarFake from '../console/progressBarFake.js'; // Run DATInferrer, but condense all DATs down to one -function buildInferredDat(romFiles: File[]): DAT { - const datGames = new DATGameInferrer(new ProgressBarFake()).infer(romFiles) +function buildInferredDat(options: Options, romFiles: File[]): DAT { + const datGames = new DATGameInferrer(options, new ProgressBarFake()).infer(romFiles) .flatMap((dat) => dat.getGames()); return new LogiqxDAT(new Header(), datGames); } @@ -56,10 +56,11 @@ it('should do nothing with no parents', async () => { describe('with inferred DATs', () => { it('should do nothing with no relevant patches', async () => { // Given - const romFiles = await new ROMScanner(new Options({ + const options = new Options({ input: [path.join('test', 'fixtures', 'roms', 'headered')], - }), new ProgressBarFake()).scan(); - const dat = buildInferredDat(romFiles); + }); + const romFiles = await new ROMScanner(options, new ProgressBarFake()).scan(); + const dat = buildInferredDat(options, romFiles); // When const parentsToCandidates = await runPatchCandidateGenerator(dat, romFiles); @@ -72,10 +73,11 @@ describe('with inferred DATs', () => { it('should create patch candidates with relevant patches', async () => { // Given - const romFiles = await new ROMScanner(new Options({ + const options = new Options({ input: [path.join('test', 'fixtures', 'roms', 'patchable')], - }), new ProgressBarFake()).scan(); - const dat = buildInferredDat(romFiles); + }); + const romFiles = await new ROMScanner(options, new ProgressBarFake()).scan(); + const dat = buildInferredDat(options, romFiles); // When const parentsToCandidates = await runPatchCandidateGenerator(dat, romFiles); diff --git a/test/modules/candidateWriter.test.ts b/test/modules/candidateWriter.test.ts index 350a91141..845916f2a 100644 --- a/test/modules/candidateWriter.test.ts +++ b/test/modules/candidateWriter.test.ts @@ -70,9 +70,9 @@ async function walkAndStat(dirPath: string): Promise<[string, Stats][]> { ); } -function datInferrer(romFiles: File[]): DAT { +function datInferrer(options: Options, romFiles: File[]): DAT { // Run DATInferrer, but condense all DATs down to one - const datGames = new DATGameInferrer(new ProgressBarFake()).infer(romFiles) + const datGames = new DATGameInferrer(options, new ProgressBarFake()).infer(romFiles) .flatMap((dat) => dat.getGames()); // TODO(cemmer): filter to unique games / remove duplicates return new LogiqxDAT(new Header({ name: 'ROMWriter Test' }), datGames); @@ -93,7 +93,7 @@ async function candidateWriter( output: outputTemp, }); const romFiles = await new ROMScanner(options, new ProgressBarFake()).scan(); - const dat = datInferrer(romFiles); + const dat = datInferrer(options, romFiles); const romFilesWithHeaders = await new ROMHeaderProcessor(options, new ProgressBarFake()) .process(romFiles); const indexedRomFiles = await new FileIndexer(options, new ProgressBarFake()) @@ -575,7 +575,8 @@ describe('zip', () => { [`ROMWriter Test.zip|${path.join('onetwothree', 'one.rom')}`, 'f817a89f'], [`ROMWriter Test.zip|${path.join('onetwothree', 'three.rom')}`, 'ff46c5d8'], [`ROMWriter Test.zip|${path.join('onetwothree', 'two.rom')}`, '96170874'], - ['ROMWriter Test.zip|speed_test_v51.sfc', '8beffd94'], + [`ROMWriter Test.zip|${path.join('speed_test_v51', 'speed_test_v51.sfc')}`, '8beffd94'], + [`ROMWriter Test.zip|${path.join('speed_test_v51', 'speed_test_v51.smc')}`, '9adca6cc'], ['ROMWriter Test.zip|three.rom', 'ff46c5d8'], ['ROMWriter Test.zip|two.rom', '96170874'], ['ROMWriter Test.zip|unknown.rom', '377a7727'], diff --git a/test/modules/datGameInferrer.test.ts b/test/modules/datGameInferrer.test.ts index 0b4b17175..27295898d 100644 --- a/test/modules/datGameInferrer.test.ts +++ b/test/modules/datGameInferrer.test.ts @@ -4,30 +4,32 @@ import Options from '../../src/types/options.js'; import ProgressBarFake from '../console/progressBarFake.js'; test.each([ - ['test/fixtures/roms/**/*', { - '7z': 5, + // No input paths + [[], {}], + // One input path + [['test/fixtures/roms/**/*'], { roms: 27 }], + [['test/fixtures/roms/7z/*'], { '7z': 5 }], + [['test/fixtures/roms/rar/*'], { rar: 5 }], + [['test/fixtures/roms/raw/*'], { raw: 10 }], + [['test/fixtures/roms/tar/*'], { tar: 5 }], + [['test/fixtures/roms/zip/*'], { zip: 6 }], + // Multiple input paths + [[ + 'test/fixtures/roms/headered', + 'test/fixtures/roms/patchable/*', + 'test/fixtures/roms/unheadered/**/*', + ], { headered: 6, patchable: 9, - rar: 5, - raw: 10, - roms: 5, - tar: 5, unheadered: 1, - zip: 6, }], - ['test/fixtures/roms/7z/*', { '7z': 5 }], - ['test/fixtures/roms/rar/*', { rar: 5 }], - ['test/fixtures/roms/raw/*', { raw: 10 }], - ['test/fixtures/roms/tar/*', { tar: 5 }], - ['test/fixtures/roms/zip/*', { zip: 6 }], -])('should infer DATs: %s', async (inputGlob, expected) => { +])('should infer DATs: %s', async (input, expected) => { // Given - const romFiles = await new ROMScanner(new Options({ - input: [inputGlob], - }), new ProgressBarFake()).scan(); + const options = new Options({ input }); + const romFiles = await new ROMScanner(options, new ProgressBarFake()).scan(); // When - const dats = new DATGameInferrer(new ProgressBarFake()).infer(romFiles); + const dats = new DATGameInferrer(options, new ProgressBarFake()).infer(romFiles); // Then const datNameToGameCount = Object.fromEntries( diff --git a/test/modules/dir2DatCreator.test.ts b/test/modules/dir2DatCreator.test.ts new file mode 100644 index 000000000..4563234c8 --- /dev/null +++ b/test/modules/dir2DatCreator.test.ts @@ -0,0 +1,74 @@ +import 'jest-extended'; + +import DATGameInferrer from '../../src/modules/datGameInferrer.js'; +import DATScanner from '../../src/modules/datScanner.js'; +import Dir2DatCreator from '../../src/modules/dir2DatCreator.js'; +import ROMScanner from '../../src/modules/romScanner.js'; +import FsPoly from '../../src/polyfill/fsPoly.js'; +import DAT from '../../src/types/dats/dat.js'; +import Options from '../../src/types/options.js'; +import ProgressBarFake from '../console/progressBarFake.js'; + +it('should do nothing if dir2dat command not provided', async () => { + // Given some input ROMs + const options = new Options({ + // No command provided + input: ['test/fixtures/roms'], + }); + const files = await new ROMScanner(options, new ProgressBarFake()).scan(); + + // And a DAT + const inferredDats = new DATGameInferrer(options, new ProgressBarFake()).infer(files); + expect(inferredDats).toHaveLength(1); + const [inferredDat] = inferredDats; + + // When writing the DAT to disk + const dir2dat = await new Dir2DatCreator(options, new ProgressBarFake()).create(inferredDat); + + // Then the DAT wasn't written + expect(dir2dat).toBeUndefined(); +}); + +it('should write a valid DAT', async () => { + // Given some input ROMs + const options = new Options({ + commands: ['dir2dat'], + input: ['test/fixtures/roms'], + }); + const files = await new ROMScanner(options, new ProgressBarFake()).scan(); + + // And a DAT + const inferredDats = new DATGameInferrer(options, new ProgressBarFake()).infer(files); + expect(inferredDats).toHaveLength(1); + const [inferredDat] = inferredDats; + + // When writing the DAT to disk + const dir2dat = await new Dir2DatCreator(options, new ProgressBarFake()).create(inferredDat); + + // Then the written DAT exists + if (dir2dat === undefined) { + throw new Error('failed to create dir2dat'); + } + + // And the written DAT can be parsed + let writtenDat: DAT; + try { + await expect(FsPoly.exists(dir2dat)).resolves.toEqual(true); + const writtenDats = await new DATScanner(new Options({ + ...options, + dat: [dir2dat], + }), new ProgressBarFake()).scan(); + expect(writtenDats).toHaveLength(1); + [writtenDat] = writtenDats; + } finally { + await FsPoly.rm(dir2dat, { force: true }); + } + + // And the written DAT matches the inferred DAT + expect(writtenDat.getHeader().toString()) + .toEqual(inferredDat.getHeader().toString()); + expect(writtenDat.getParents().map((parent) => parent.getName())) + .toIncludeAllMembers(inferredDat.getParents().map((parent) => parent.getName())); + expect(writtenDat.getGames().map((game) => game.hashCode())) + .toIncludeAllMembers(inferredDat.getGames().map((game) => game.hashCode())); +}); diff --git a/test/modules/fixdatCreator.test.ts b/test/modules/fixdatCreator.test.ts index ce444d4fb..7628d12fe 100644 --- a/test/modules/fixdatCreator.test.ts +++ b/test/modules/fixdatCreator.test.ts @@ -55,7 +55,7 @@ async function runFixdatCreator( parentsToCandidates: Map, ): Promise { const fixdatPath = await new FixdatCreator(new Options(optionsProps), new ProgressBarFake()) - .write(dat, parentsToCandidates); + .create(dat, parentsToCandidates); if (!fixdatPath) { return undefined; } diff --git a/test/modules/statusGenerator.test.ts b/test/modules/statusGenerator.test.ts index 9f5874059..3bd8d2427 100644 --- a/test/modules/statusGenerator.test.ts +++ b/test/modules/statusGenerator.test.ts @@ -104,35 +104,35 @@ describe('toConsole', () => { preferParent: true, }); const map = await candidateGenerator(options, []); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, map); expect(stripAnsi(datStatus.toConsole(options))).toEqual('2/6 games, 0/1 BIOSes, 1/1 devices, 2/5 retail releases found'); }); - it('should not print BIOS count when noBios:true', async () => { + it('should not print BIOS count when noBios:true', () => { const options = new Options({ ...defaultOptions, noBios: true }); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, parentsToReleaseCandidatesWithoutFiles); expect(stripAnsi(datStatus.toConsole(options))).toEqual('2/6 games, 1/1 devices, 2/5 retail releases found'); }); - it('should only print BIOS count when onlyBios:true', async () => { + it('should only print BIOS count when onlyBios:true', () => { const options = new Options({ ...defaultOptions, onlyBios: true }); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, parentsToReleaseCandidatesWithoutFiles); expect(stripAnsi(datStatus.toConsole(options))).toEqual('0/1 BIOSes found'); }); - it('should not print device count when noDevice:true', async () => { + it('should not print device count when noDevice:true', () => { const options = new Options({ ...defaultOptions, noDevice: true }); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, parentsToReleaseCandidatesWithoutFiles); expect(stripAnsi(datStatus.toConsole(options))).toEqual('2/6 games, 0/1 BIOSes, 2/5 retail releases found'); }); - it('should not print device count when onlyDevice:true', async () => { + it('should not print device count when onlyDevice:true', () => { const options = new Options({ ...defaultOptions, onlyDevice: true }); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, parentsToReleaseCandidatesWithoutFiles); expect(stripAnsi(datStatus.toConsole(options))).toEqual('1/1 devices found'); }); @@ -142,7 +142,7 @@ describe('toConsole', () => { it('should print games without ROMS and BIOSes as found', async () => { const options = new Options(defaultOptions); const map = await candidateGenerator(options, [gameNameBios]); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, map); expect(stripAnsi(datStatus.toConsole(options))).toEqual('3/6 games, 1/1 BIOSes, 1/1 devices, 3/5 retail releases found'); }); @@ -150,7 +150,7 @@ describe('toConsole', () => { it('should print prototypes as found', async () => { const options = new Options(defaultOptions); const map = await candidateGenerator(options, [gameNamePrototype]); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, map); expect(stripAnsi(datStatus.toConsole(options))).toEqual('3/6 games, 0/1 BIOSes, 1/1 devices, 2/5 retail releases found'); }); @@ -158,7 +158,7 @@ describe('toConsole', () => { it('should print the game with single rom as found', async () => { const options = new Options(defaultOptions); const map = await candidateGenerator(options, [gameNameSingleRom]); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, map); expect(stripAnsi(datStatus.toConsole(options))).toEqual('3/6 games, 0/1 BIOSes, 1/1 devices, 3/5 retail releases found'); }); @@ -180,7 +180,7 @@ describe('toConsole', () => { ]); const options = new Options(defaultOptions); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, map); expect(stripAnsi(datStatus.toConsole(options))).toEqual('2/6 games, 0/1 BIOSes, 1/1 devices, 2/5 retail releases, 1 patched games found'); }); @@ -193,7 +193,7 @@ describe('toConsole', () => { gameNameSingleRom, gameNameMultipleRoms, ]); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, map); expect(stripAnsi(datStatus.toConsole(options))).toEqual('6/6 games, 1/1 BIOSes, 1/1 devices, 5/5 retail releases found'); }); @@ -211,7 +211,7 @@ describe('toConsole', () => { gameNameMultipleRoms, ]); map = await new CandidatePreferer(options, new ProgressBarFake()).prefer(dummyDat, map); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, map); expect(stripAnsi(datStatus.toConsole(options))).toEqual('5/5 games, 1/1 BIOSes, 1/1 devices, 5/5 retail releases found'); }); @@ -226,7 +226,7 @@ describe('toCSV', () => { preferParent: true, }); const map = await candidateGenerator(options, []); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, map); await expect(datStatus.toCsv(options)).resolves.toEqual(`DAT Name,Game Name,Status,ROM Files,Patched,BIOS,Retail Release,Unlicensed,Debug,Demo,Beta,Sample,Prototype,Test,Aftermarket,Homebrew,Bad dat,bios,MISSING,,false,true,true,false,false,false,false,false,false,false,false,false,false @@ -240,7 +240,7 @@ dat,no roms,FOUND,,false,false,true,false,false,false,false,false,false,false,fa // NOTE(cemmer): the BIOS game shows here because DATFilter is never run, and this is fine it('should not report BIOSes when noBios:true', async () => { const options = new Options({ noBios: true }); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, parentsToReleaseCandidatesWithoutFiles); await expect(datStatus.toCsv(options)).resolves.toEqual(`DAT Name,Game Name,Status,ROM Files,Patched,BIOS,Retail Release,Unlicensed,Debug,Demo,Beta,Sample,Prototype,Test,Aftermarket,Homebrew,Bad dat,bios,MISSING,,false,true,true,false,false,false,false,false,false,false,false,false,false @@ -253,7 +253,7 @@ dat,no roms,FOUND,,false,false,true,false,false,false,false,false,false,false,fa it('should only report BIOSes when onlyBios:true', async () => { const options = new Options({ ...defaultOptions, onlyBios: true }); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, parentsToReleaseCandidatesWithoutFiles); await expect(datStatus.toCsv(options)).resolves.toEqual(`DAT Name,Game Name,Status,ROM Files,Patched,BIOS,Retail Release,Unlicensed,Debug,Demo,Beta,Sample,Prototype,Test,Aftermarket,Homebrew,Bad dat,bios,MISSING,,false,true,true,false,false,false,false,false,false,false,false,false,false`); @@ -262,7 +262,7 @@ dat,bios,MISSING,,false,true,true,false,false,false,false,false,false,false,fals // NOTE(cemmer): the device game shows here because DATFilter is never run, and this is fine it('should not report devices when noDevice:true', async () => { const options = new Options({ ...defaultOptions, noDevice: true }); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, parentsToReleaseCandidatesWithoutFiles); await expect(datStatus.toCsv(options)).resolves.toEqual(`DAT Name,Game Name,Status,ROM Files,Patched,BIOS,Retail Release,Unlicensed,Debug,Demo,Beta,Sample,Prototype,Test,Aftermarket,Homebrew,Bad dat,bios,MISSING,,false,true,true,false,false,false,false,false,false,false,false,false,false @@ -275,7 +275,7 @@ dat,no roms,FOUND,,false,false,true,false,false,false,false,false,false,false,fa it('should not report devices when onlyDevice:true', async () => { const options = new Options({ ...defaultOptions, onlyDevice: true }); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, parentsToReleaseCandidatesWithoutFiles); await expect(datStatus.toCsv(options)).resolves.toEqual(`DAT Name,Game Name,Status,ROM Files,Patched,BIOS,Retail Release,Unlicensed,Debug,Demo,Beta,Sample,Prototype,Test,Aftermarket,Homebrew,Bad dat,device,FOUND,,false,false,true,false,false,false,false,false,false,false,false,false,false`); @@ -286,7 +286,7 @@ dat,device,FOUND,,false,false,true,false,false,false,false,false,false,false,fal it('should report the BIOS as found', async () => { const options = new Options(defaultOptions); const map = await candidateGenerator(options, [gameNameBios]); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, map); await expect(datStatus.toCsv(options)).resolves.toEqual(`DAT Name,Game Name,Status,ROM Files,Patched,BIOS,Retail Release,Unlicensed,Debug,Demo,Beta,Sample,Prototype,Test,Aftermarket,Homebrew,Bad dat,bios,FOUND,bios.rom,false,true,true,false,false,false,false,false,false,false,false,false,false @@ -326,7 +326,7 @@ dat,no roms,FOUND,,false,false,true,false,false,false,false,false,false,false,fa return [parent, releaseCandidatesWithFiles]; }))); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, map); await expect(datStatus.toCsv(options)).resolves.toEqual(`DAT Name,Game Name,Status,ROM Files,Patched,BIOS,Retail Release,Unlicensed,Debug,Demo,Beta,Sample,Prototype,Test,Aftermarket,Homebrew,Bad dat,bios,MISSING,,false,true,true,false,false,false,false,false,false,false,false,false,false @@ -340,7 +340,7 @@ dat,no roms,FOUND,,false,false,true,false,false,false,false,false,false,false,fa it('should report the prototype as found', async () => { const options = new Options(defaultOptions); const map = await candidateGenerator(options, [gameNamePrototype]); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, map); await expect(datStatus.toCsv(options)).resolves.toEqual(`DAT Name,Game Name,Status,ROM Files,Patched,BIOS,Retail Release,Unlicensed,Debug,Demo,Beta,Sample,Prototype,Test,Aftermarket,Homebrew,Bad dat,bios,MISSING,,false,true,true,false,false,false,false,false,false,false,false,false,false @@ -354,7 +354,7 @@ dat,no roms,FOUND,,false,false,true,false,false,false,false,false,false,false,fa it('should report the game with a single ROM as found', async () => { const options = new Options(defaultOptions); const map = await candidateGenerator(options, [gameNameSingleRom]); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, map); await expect(datStatus.toCsv(options)).resolves.toEqual(`DAT Name,Game Name,Status,ROM Files,Patched,BIOS,Retail Release,Unlicensed,Debug,Demo,Beta,Sample,Prototype,Test,Aftermarket,Homebrew,Bad dat,bios,MISSING,,false,true,true,false,false,false,false,false,false,false,false,false,false @@ -382,7 +382,7 @@ dat,no roms,FOUND,,false,false,true,false,false,false,false,false,false,false,fa ]); const options = new Options(defaultOptions); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, map); await expect(datStatus.toCsv(options)).resolves.toEqual(`DAT Name,Game Name,Status,ROM Files,Patched,BIOS,Retail Release,Unlicensed,Debug,Demo,Beta,Sample,Prototype,Test,Aftermarket,Homebrew,Bad dat,bios,MISSING,,false,true,true,false,false,false,false,false,false,false,false,false,false @@ -402,7 +402,7 @@ dat,patched game,FOUND,patched.rom,true,false,true,false,false,false,false,false gameNameSingleRom, gameNameMultipleRoms, ]); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, map); await expect(datStatus.toCsv(options)).resolves.toEqual(`DAT Name,Game Name,Status,ROM Files,Patched,BIOS,Retail Release,Unlicensed,Debug,Demo,Beta,Sample,Prototype,Test,Aftermarket,Homebrew,Bad dat,bios,FOUND,bios.rom,false,true,true,false,false,false,false,false,false,false,false,false,false @@ -425,7 +425,7 @@ dat,no roms,FOUND,,false,false,true,false,false,false,false,false,false,false,fa gameNameSingleRom, gameNameMultipleRoms, ]); - const datStatus = await new StatusGenerator(options, new ProgressBarFake()) + const datStatus = new StatusGenerator(options, new ProgressBarFake()) .generate(dummyDat, map); await expect(datStatus.toCsv(options)).resolves.toEqual(`DAT Name,Game Name,Status,ROM Files,Patched,BIOS,Retail Release,Unlicensed,Debug,Demo,Beta,Sample,Prototype,Test,Aftermarket,Homebrew,Bad dat,bios,FOUND,bios.rom,false,true,true,false,false,false,false,false,false,false,false,false,false