Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: dir2dat command #740

Merged
merged 41 commits into from
Dec 7, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
9ac02f9
Draft
emmercm Aug 26, 2023
ab4abdc
Merge remote-tracking branch 'origin/main' into emmercm/file-md5-sha1
emmercm Aug 28, 2023
5d13638
Refactor
emmercm Aug 28, 2023
d8ef9d4
Merge remote-tracking branch 'origin/main' into emmercm/file-md5-sha1
emmercm Aug 30, 2023
0f23a56
Update
emmercm Sep 6, 2023
b43e39b
Update
emmercm Sep 10, 2023
faba1b5
Get archive entry CRC32 when testing
emmercm Sep 10, 2023
77c82f7
Linter
emmercm Sep 10, 2023
6641256
Merge remote-tracking branch 'origin/main' into emmercm/file-md5-sha1
emmercm Sep 11, 2023
2763562
Merge fix
emmercm Sep 11, 2023
67348af
Merge remote-tracking branch 'origin/main' into emmercm/file-md5-sha1
emmercm Sep 29, 2023
ef77be6
Lint fix
emmercm Sep 29, 2023
9a1ac6d
Merge fix
emmercm Sep 29, 2023
ad29d3c
Tests
emmercm Sep 30, 2023
b114512
Merge remote-tracking branch 'origin/main' into emmercm/file-md5-sha1
emmercm Sep 30, 2023
bffa2d0
Merge fix
emmercm Sep 30, 2023
4d71236
Merge branch 'main' into emmercm/file-md5-sha1
emmercm Oct 3, 2023
006e130
Draft
emmercm Oct 3, 2023
9e11397
Merge remote-tracking branch 'origin/main' into emmercm/dir2dat
emmercm Oct 3, 2023
58f9dd2
Merge remote-tracking branch 'origin/main' into emmercm/dir2dat
emmercm Oct 17, 2023
1fcc0b9
Linter
emmercm Oct 17, 2023
7766b9a
Fix argument parsing
emmercm Oct 17, 2023
726e6cc
Address TODOs
emmercm Oct 17, 2023
ec42ab7
Update
emmercm Oct 17, 2023
4e797ea
Merge remote-tracking branch 'origin/main' into emmercm/dir2dat
emmercm Oct 21, 2023
c7dc0bf
Update
emmercm Oct 21, 2023
c2eb64a
Update
emmercm Oct 21, 2023
dba5a54
Logging
emmercm Oct 21, 2023
30ffa8f
Test
emmercm Oct 21, 2023
da9a0be
Merge remote-tracking branch 'origin/main' into emmercm/dir2dat
emmercm Oct 21, 2023
ee7c7c5
Docs
emmercm Oct 21, 2023
5797ffa
Merge remote-tracking branch 'origin/main' into emmercm/dir2dat
emmercm Oct 21, 2023
cc7460d
Docs
emmercm Oct 22, 2023
75851cb
Merge remote-tracking branch 'origin/main' into emmercm/dir2dat
emmercm Oct 23, 2023
4609598
Merge remote-tracking branch 'origin/main' into emmercm/dir2dat
emmercm Nov 20, 2023
ff6e676
Merge remote-tracking branch 'origin/main' into emmercm/dir2dat
emmercm Dec 4, 2023
6c7c54f
Merge fix
emmercm Dec 4, 2023
777a57e
E2E test
emmercm Dec 4, 2023
379a550
Docs
emmercm Dec 4, 2023
318632c
Merge remote-tracking branch 'origin/main' into emmercm/dir2dat
emmercm Dec 7, 2023
3ff0381
Fixes
emmercm Dec 7, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,14 @@ A video of an example use case:
With `igir` you can manage a ROM collection of any size:

- 🔍 Scan for DATs, ROMs, and ROM patches - including those in archives (see [scanning](https://igir.io/input/file-scanning) & [archive docs](https://igir.io/input/reading-archives))
- 📂 Organize ROM files by console (see [DAT docs](https://igir.io/input/dats))
- 🪄 Name ROM files consistently, including the right extension (see [DAT docs](https://igir.io/input/dats))
- 📂 Organize ROM files by console (see [DAT docs](https://igir.io/dats/overview))
- 🪄 Name ROM files consistently, including the right extension (see [DAT docs](https://igir.io/dats/overview))
- ✂️ Filter out duplicate ROMs, or ROMs in languages you don't understand (see [filtering docs](https://igir.io/roms/filtering-preferences))
- 🗜️ Extract or archive ROMs in mass (see [archive docs](https://igir.io/output/writing-archives))
- 🩹 Patch ROMs automatically in mass (see [scanning](https://igir.io/input/file-scanning) & [patching docs](https://igir.io/roms/patching))
- 🎩 Parse ROMs with headers, and optionally remove them (see [header docs](https://igir.io/roms/headers))
- ↔️ Build & re-build (un-merge, split, or merge) MAME ROM sets (see [arcade docs](https://igir.io/usage/arcade))
- 🔮 Report on what ROMs are present or missing for each console, and create fixdats for missing ROMs (see [reporting](https://igir.io/output/reporting) & [DAT docs](https://igir.io/input/dats))
- 🔮 Report on what ROMs are present or missing for each console, and create fixdats for missing ROMs (see [reporting](https://igir.io/output/reporting) & [DAT docs](https://igir.io/dats/overview))

## How do I run `igir`?

Expand Down
6 changes: 3 additions & 3 deletions docs/alternatives.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@ There are a few different popular ROM managers that have similar features:
| App: OS compatibility | ✅ anything [Node.js supports](https://nodejs.org/en/download) | ⚠️ Windows, macOS & Linux via [Wine](https://www.winehq.org/) | ⚠️ Windows, Linux via [Mono](https://www.mono-project.com/) | ❌ Windows only |
| App: UI or CLI | CLI only by design | UI only | Separate UI & CLI versions | UI only |
| App: required setup steps | ✅ no setup required | ❌ requires "profile" setup per DAT | ⚠️ if specifying DAT & ROM dirs | ❌ requires per-DAT DB setup |
| DATs: supported formats | Logiqx XML, MAME ListXML, CMPro, HTGD SMDB ([DATs docs](input/dats.md)) | Logiqx XML, MAME ListXML, CMPro | Logiqx XML, MAME ListXML, CMPro, RomCenter, HTGD SMDB | Logiqx XML, CMPro, RomCenter |
| DATs: supported formats | Logiqx XML, MAME ListXML, CMPro, HTGD SMDB ([DATs docs](dats/overview.md)) | Logiqx XML, MAME ListXML, CMPro | Logiqx XML, MAME ListXML, CMPro, RomCenter, HTGD SMDB | Logiqx XML, CMPro, RomCenter |
| DATs: process multiple at once | ✅ | ⚠️ via the batcher | ✅ | ❌ |
| DATs: infer parent/clone info | ✅ | ❌ | ❌ | ❌ |
| DATs: built-in download manager | ❌ | ❌ | ⚠️ via [DatVault](https://www.datvault.com/) | ❌ |
| DATs: supports DAT URLs | ✅ | ❌ | ❌ | ❌ |
| DATs: create from files (dir2dat) | | ✅ | | ❌ |
| DATs: create from files (dir2dat) | ✅ [dir2dat docs](dats/dir2dat.md) | ✅ | | ❌ |
| DATs: combine multiple | ❌ | ❌ | ✅ | ❌ |
| Archives: extraction formats | ✅ many formats ([reading archives docs](input/reading-archives.md)) | ✅ `.zip`, `.7z`, `.rar` | ⚠️ `.zip`, `.7z` | ⚠️ `.zip`, `.7z` |
| Archives: creation formats | ❌ `.zip` only by design ([writing archives docs](output/writing-archives.md)) | ✅ `.zip`, `.7z`, `.rar` | ⚠️ `.zip`, `.7z` | ⚠️ `.zip`, `.7z` |
Expand All @@ -31,7 +31,7 @@ There are a few different popular ROM managers that have similar features:
| Output: separate input & output dirs | ✅ | ❌ | ⚠️ yes but files are always moved | ❌ |
| Output: subdirectory customization | ✅ | ❌ | ⚠️ depends on DAT organization | ❌ |
| Output: create single archive for DAT | ✅ | ❌ | ✅ | ❌ |
| Output: fixdat creation | ✅ [DATs docs](input/dats.md) | ✅ | ✅ | ❌ |
| Output: fixdat creation | ✅ [Fixdat docs](dats/fixdats.md) | ✅ | ✅ | ❌ |

!!! note

Expand Down
8 changes: 4 additions & 4 deletions docs/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ Files in the input directories will be left alone, they will _not_ be modified o

### `move`

Move ROMs from an input directory to the output directory. The same directory can be specified for both input & output, resulting in ROMs being renamed as their names change in [DATs](input/dats.md).
Move ROMs from an input directory to the output directory. The same directory can be specified for both input & output, resulting in ROMs being renamed as their names change in [DATs](dats/overview.md).

ROMs will be deleted from their input directory after _all_ ROMs for _every_ [DAT](input/dats.md) have been written.
ROMs will be deleted from their input directory after _all_ ROMs for _every_ [DAT](dats/overview.md) have been written.

### `symlink`

Expand Down Expand Up @@ -63,14 +63,14 @@ After performing one of the ROM writing commands, verify that the file was writt

### `clean`

Files in the output directory that do not match any ROM in any [DAT](input/dats.md) will be deleted.
Files in the output directory that do not match any ROM in any [DAT](dats/overview.md) will be deleted.

See the [output cleaning page](output/cleaning.md) for more information.

## ROM reporting

### `report`

A report will be generated of what input files were matched by what DAT, and what games in what [DATs](input/dats.md) have missing ROMs.
A report will be generated of what input files were matched by what DAT, and what games in what [DATs](dats/overview.md) have missing ROMs.

See the [reporting page](output/reporting.md) for more information.
84 changes: 84 additions & 0 deletions docs/dats/dir2dat.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# dir2dat

"dir2dat" refers to DATs that have been automatically created based on files in an input directory. DATs generated this way are not typically useful as-is, they usually require some hand editing after creation.

`igir` has the ability to create these DATs with the `igir dir2dat` command. Example:

```shell
igir dir2dat --input <path> [--input <path>..]
```

## dir2dat rules

`igir` uses the following rules when creating dir2dat DAT files:

- **A DAT file will be created for every input path.**

If multiple input paths overlap, such as:

=== ":simple-windowsxp: Windows"

```batch
igir dir2dat ^
--input "C:\ROMs" ^
--input "C:\ROMs\NES"
```

=== ":simple-apple: macOS"

```shell
igir dir2dat \
--input ~/ROMs/ \
--input ~/ROMs/NES
```

=== ":simple-linux: Linux"

```shell
igir dir2dat \
--input ~/ROMs/ \
--input ~/ROMs/NES
```

then ROMs can appear in multiple resulting dir2dat files.

- **Each input path's [basename](https://linux.die.net/man/1/basename) will be used for the DAT's name.**

Here are some examples:

| Input path | DAT name |
|----------------------------|----------|
| `--input "ROMs"` | `ROMs` |
| `--input "ROMs/NES"` | `NES` |
| `--input "ROMs/SNES/*"` | `SNES` |
| `--input "ROMs/SNES/**/*"` | `SNES` |

- **Archive files will be treated as a single game, with every archive entry being a separate ROM.**

This is consistent with how the [`igir zip` command](../output/writing-archives.md) works, and with what [MAME expects](../usage/arcade.md).

- **The input file's [basename](https://linux.die.net/man/1/basename) (without extension) will be used for the game name.**

!!! warning

This will cause input files with the same basename to be grouped together!

## Combining with other options

Once DATs have been generated from input files, they are processed the same as any other DAT file. That means:

- **Parent/clone information may be [inferred](overview.md#parentclone-inference) from game names.**

If your input files are in some kind of standard naming convention (e.g. [No-Intro](https://wiki.no-intro.org/index.php?title=Naming_Convention), [Redump](https://datomatic.no-intro.org/stuff/The%20Official%20No-Intro%20Convention%20(20071030).pdf), or [TOSEC](https://www.tosecdev.org/tosec-naming-convention)), then parent/clone information can be inferred for [1G1R preferences](../roms/filtering-preferences.md).

Parent/clone information also allows for [merging & splitting](../usage/arcade.md) of ROM sets.

- **[ROM filter options](../roms/filtering-preferences.md) can be applied.**

If your input files are in some kind of standard naming convention (e.g. [No-Intro](https://wiki.no-intro.org/index.php?title=Naming_Convention), [Redump](https://datomatic.no-intro.org/stuff/The%20Official%20No-Intro%20Convention%20(20071030).pdf), or [TOSEC](https://www.tosecdev.org/tosec-naming-convention)) that contains region, language, or other tags, then [ROM filter options](../roms/filtering-preferences.md) can be applied.

## Alternative tools

It is unlikely that any ROM tool, including `igir`, will ever meet every person's exact DAT creation needs.

[SabreTools](https://github.com/SabreTools/SabreTools) is a great tool for DAT management that offers many complex options for DAT creation, filtering, merging, and splitting.
52 changes: 52 additions & 0 deletions docs/dats/fixdats.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Fixdats

"Fixdats" are DATs that contain only ROMs that are missing from your collection. Fixdats are derived from some other DAT (see the [DATs overview docs](overview.md) for how to obtain DATs), containing only a subset of the ROMs. Fixdats are specific to the state of each person's ROM collection, so they aren't necessarily meaningful to other people.

Fixdats help you find files missing from your collection, and they can be used to generate a collection of those files once you've found them. This sub-collection of files can then be merged back into your main collection.

The `fixdat` command creates a [Logiqx XML](http://www.logiqx.com/DatFAQs/) DAT for every input DAT (the `--dat <path>` option) that is missing ROMs. When writing (`copy`, `move`, and `symlink` [commands](../commands.md)), the fixdat will be written to the output directory, otherwise it will be written to the working directory.

For example:

=== ":simple-windowsxp: Windows"

```batch
igir copy zip fixdat ^
--dat "Nintendo - Game Boy.dat" ^
--dat "Nintendo - Game Boy Advance.dat" ^
--dat "Nintendo - Game Boy Color.dat" ^
--input ROMs\ ^
--output ROMs-Sorted\ ^
--fixdat
```

=== ":simple-apple: macOS"

```shell
igir copy zip fixdat \
--dat "Nintendo - Game Boy.dat" \
--dat "Nintendo - Game Boy Advance.dat" \
--dat "Nintendo - Game Boy Color.dat" \
--input ROMs/ \
--output ROMs-Sorted/
```

=== ":simple-linux: Linux"

```shell
igir copy zip fixdat \
--dat "Nintendo - Game Boy.dat" \
--dat "Nintendo - Game Boy Advance.dat" \
--dat "Nintendo - Game Boy Color.dat" \
--input ROMs/ \
--output ROMs-Sorted/
```

may produce some fixdats in the `ROMs-Sorted/` directory, if any of the input DATs have ROMs that weren't found in the `ROMs/` input directory:

```text
ROMs-Sorted/
├── Nintendo - Game Boy (20230414-173400) fixdat.dat
├── Nintendo - Game Boy Advance (20230414-173400) fixdat.dat
└── Nintendo - Game Boy Color (20230414-173400) fixdat.dat
```
57 changes: 2 additions & 55 deletions docs/input/dats.md → docs/dats/overview.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# DATs
# DAT Overview

## Overview

Expand Down Expand Up @@ -80,7 +80,7 @@ There have been a few DAT-like formats developed over the years. `igir` supports

## DAT input options

The `--dat <path>` supports files, archives, directories, and globs like any of the other file options. See the [file scanning page](file-scanning.md) for more information.
The `--dat <path>` supports files, archives, directories, and globs like any of the other file options. See the [file scanning page](../input/file-scanning.md) for more information.

`igir` also supports URLs to DAT files and archives. This is helpful to make sure you're always using the most up-to-date version of a DAT hosted on sites such as GitHub. For example:

Expand Down Expand Up @@ -162,56 +162,3 @@ The rule-of-thumb with DATs and arcade emulation is: your emulator probably has
If you are using a desktop frontend such as [RetroArch](../usage/desktop/retroarch.md), it may come with multiple versions of the same emulator, and it is unlikely that any of them is the most recent version. Follow the frontend's documentation to location or download the correct DAT to use with each emulator.

See the [arcade page](../usage/arcade.md) for more information on building & re-building arcade ROM sets.

## Fixdats

"Fixdats" are DATs that contain only ROMs that are missing from your collection. Fixdats are derived from some other DAT (see above for obtaining DATs), containing only a subset of the ROMs. Fixdats are specific to the state of each person's ROM collection, so they aren't necessarily meaningful to other people.

Fixdats help you find files missing from your collection, and they can be used to generate a collection of those files once you've found them. This sub-collection of files can then be merged back into your main collection.

The `fixdat` command creates a [Logiqx XML](http://www.logiqx.com/DatFAQs/) DAT for every input DAT (the `--dat <path>` option) that is missing ROMs. When writing (`copy`, `move`, and `symlink` commands), the fixdat will be written to the output directory, otherwise it will be written to the working directory.

For example:

=== ":simple-windowsxp: Windows"

```batch
igir copy zip fixdat ^
--dat "Nintendo - Game Boy.dat" ^
--dat "Nintendo - Game Boy Advance.dat" ^
--dat "Nintendo - Game Boy Color.dat" ^
--input ROMs\ ^
--output ROMs-Sorted\ ^
--fixdat
```

=== ":simple-apple: macOS"

```shell
igir copy zip fixdat \
--dat "Nintendo - Game Boy.dat" \
--dat "Nintendo - Game Boy Advance.dat" \
--dat "Nintendo - Game Boy Color.dat" \
--input ROMs/ \
--output ROMs-Sorted/
```

=== ":simple-linux: Linux"

```shell
igir copy zip fixdat \
--dat "Nintendo - Game Boy.dat" \
--dat "Nintendo - Game Boy Advance.dat" \
--dat "Nintendo - Game Boy Color.dat" \
--input ROMs/ \
--output ROMs-Sorted/
```

may produce some fixdats in the `ROMs-Sorted/` directory, if any of the input DATs have ROMs that weren't found in the `ROMs/` input directory:

```text
ROMs-Sorted/
├── Nintendo - Game Boy (20230414-173400) fixdat.dat
├── Nintendo - Game Boy Advance (20230414-173400) fixdat.dat
└── Nintendo - Game Boy Color (20230414-173400) fixdat.dat
```
2 changes: 1 addition & 1 deletion docs/input/file-scanning.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
`igir` has a few options to specify input files, as well as files to exclude:

- ROMs: `--input <path>` (required), `--input-exclude <path>`
- [DATs](dats.md): `--dat <path>`, `--dat-exclude <path>`
- [DATs](../dats/overview.md): `--dat <path>`, `--dat-exclude <path>`
- [ROM patches](../roms/patching.md): `--patch <path>`, `--patch-exclude <path>`

## Archive files
Expand Down
6 changes: 3 additions & 3 deletions docs/output/cleaning.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@

The `igir clean` [command](../commands.md) can be used when writing (`igir copy`, `igir move`, and `igir symlink`) to delete files from the `--output <path>` directory that are either:

- Not contained in any provided [DAT](../input/dats.md) (the `--dat <path>` option).
- Contained in a [DAT](../input/dats.md) (the `--dat <path>` option), but the file is in the incorrect location.
- Not contained in any provided [DAT](../dats/overview.md) (the `--dat <path>` option).
- Contained in a [DAT](../dats/overview.md) (the `--dat <path>` option), but the file is in the incorrect location.

## The golden rule

The golden rule of the `igir clean` command is it will _not_ delete files in any directory tree that it did not write to.

In practical terms, this means:

**1. If no file was written (i.e. no input file matched any ROM in any [DAT](../input/dats.md)), then `igir clean` will not delete any files.**
**1. If no file was written (i.e. no input file matched any ROM in any [DAT](../dats/overview.md)), then `igir clean` will not delete any files.**

**2. If [tokens](tokens.md) are used with the `--output <path>` option, only subdirectories that are written to will be considered for cleaning.**

Expand Down
Loading
Loading