Implement different DLA methods

[boeckmann]

Currently there is an incompatibility between the drive letter assignment of the (E)DR-DOS kernel and FDISK, as FDISK follows the FreeDOS (and Microsoft) way of assigning drive letters to partitions. But DR-DOS handles this differently.

We should detect if running under DR-DOS and adapt to its drive letter assignment.

A new FDISK config variable DLA will be introduced, defaulting to zero. This means that FDISK will try to auto-adapt its DLA to the operating system. DLA=1 means explicit FreeDOS DLA algorithm, while DLA=2 means explicit DR-DOS DLA algorithm.

Notice: The DLA of the FreeDOS kernel may be changed via a kernel config variable. There is currently no easy way to detect which DLA the FreeDOS kernel uses. But the default is to match the Microsoft way of doing it, and this is what FDISK assumes.

[boeckmann]

implemented by 1c9271f

[PerditionC]

For increased compatibility you could use the method described in DOS Internals book. I.e. the drivparam/driver.sys interface, INT 2F/08xx - GET DRIVE DATA TABLE LIST *1

based on snippet from DOS Internals DRVINFO tool:

/* takes DOS drive where 2=C:,3=D:,... and returns # to use for BIOS int 13h call (FFh if no match) */
static BYTE GetBIOSDriveNumber(BYTE dosdrive)
{
    struct ddtstruct FAR *pddt; /* see hdr\device.h in FreeDOS kernel source */

    pddt = GetDOSDriveTable();

    /*	Follow the chain (each table contains a far pointer to another - not
	necessarily in ascending drive order) to the table for the specified
	DOS drive number.  Another field in the table gives the BIOS drive
	number that is used for int 13h access.  */

    for(; (WORD)pddt != 0xFFFF; pddt = pddt->ddt_next) {
	if (pddt->ddt_logdriveno == dosdrive) 
            return pddt->ddt_driveno;
    }

    /* if made it here, then either invalid DOS drive or not in DOS's 
       drive table, i.e. not handled by DOS BIOS block disk driver. 
       Return unknown - at this point the only way to map would be
       to hook int 13h and make a call to DOS to force it to access
       drive and see which call matches.  We just return failure sentinel.
    */

    return 0xFF;
}

/* validate call should be available, query for internal DOS structure with DOS drive mappings */
struct ddtstruct FAR * GetDOSDriveTable(void)
{
    if (_osmajor < 4) {
	nomatch:
	return ((struct ddtstruct FAR *) -1);
    }

    /* DS:DI -> first drive data table in list
       Supported by Novell DOS 7+ (using the MS-DOS 4+ data table format)
    */
    _asm {
	    mov     ax,0800h
	    int     2Fh
	    cmp     al,0FFh
	    jnz     nomatch

	    push    ds
	    mov     ax,0803h
	    int     2Fh
	    mov     ax,di
	    mov     dx,ds
	    pop     ds
    }
}

[*1] the FreeDOS kernel doesn't currently implement this, so I'd need to actually finish my work on it and publish it.

[PerditionC]

For fdisk, you may need the reverse function, pass in BIOS# and return DOS drive# (ie swap compare and return ddt_driveno/ddt_logdriveno).

[boeckmann]

@PerditionC I could use this to make sure the user deletes the correct drive. But I cannot use this to detect die DLA method if the user starts with an empty partition table.

Is there any benefit of using INT2F instead simply peeking at the LoL for getting a pointer to the first ddt?

[boeckmann]

There is also the problem that drive letters change with the first modification and are "out of sync" with the letters known to the OS.

[ecm-pushbx]

Is there any benefit of using INT2F instead simply peeking at the LoL for getting a pointer to the first ddt?

1. It's more portable, 2. I don't think the List of Lists, being a DOS structure, has any pointers to the chain of DDTs / UDSCs / UBPBs, being a BIO structure.

[PerditionC]

at least FreeDOS kernel doesn't, it's pointed to by dyn variable

[boeckmann]

You are right, I confused the DDT / UDSC structure with the DPB / DDSC structure. I need the DDT to get the partition offset.

[PerditionC]

makes sense, I was thinking of a good interface the kernel could expose current drive letter assignment and realized it already existed, so wanted to mention it.

[boeckmann]

Some time ago I think Jerome asked me if partition activation based on drive letter could be implemented. This INT2Fh,08 would be quite useful for it.

Also it would be very useful if kernel config like DLA method could be queried by applications.

[boeckmann]

relates to #81

[boeckmann]

I am in the process of changing FDISK to make use of volume labels instead the volume id found in the BPB, because this is what Microsoft FDISK does.

This depends on FDISK and DOS agreeing on the drive letters to query the right drive for the file with the volume attribute set. I came up with the following algorithm:

• Let FDISK do its drive letter assignment
• For every supported FAT partition FDISK found: query the serial number via INT21,69 based on the drive letter FDISK thinks it is
• If this returns a serial number, compare it with the one stored in the BPB
• If they match, use the drive letter to query the volume label
• If it does not match, set dla_incompatible = 1 and print a big fat warning on FDISK main screen
• Optionally: fallback to BPB volume id if the above fails

This does not catch all possible cases that can go wrong, but I think it is way better than not verifying anything. And it does not peek into internal DOS data structures.

[ecm-pushbx]

Why not add code to fdisk to scan the root directory itself, which would allow to always find the correct volume label file?

[boeckmann]

Why not add code to fdisk to scan the root directory itself, which would allow to always find the correct volume label file?

Could be done, and is most probably the more reliable solution (if implemented properly). Would allow FDISK to read the volume file if the drive letters disagree. The question is if it is worth implementing FAT support inside FDISK, or continue to use the BPB volume id (and instead modify FreeDOS and EDR kernels to keep the field in sync, like MS-DOS does). I tend to the latter, because it is a cleaner separation of responsibility.

Regarding DLA verification, I am quite unsure if this is even beneficial to the user. If a warning is issued on FDISK start, that would also show up if the user altered the partitions and FDISK is started a second time etc. If it is implemented it should be a help and not a disturbance.

[andrewbird]

The question is if it is worth implementing FAT support inside FDISK, or continue to use the BPB volume id (and instead modify FreeDOS and EDR kernels to keep the field in sync, like MS-DOS does).

Or both?

[boeckmann]

Or both?

Thought about it. But I do not like it, because then it is not obvious to the user where the volume name originates from. It should at least be indicated somehow, for example by prepending the label with a star or similar if it originates from the BPB. And that star should be explained to the user.

[andrewbird]

But I do not like it, because then it is not obvious to the user where the volume name originates from.

Agreed, but given that a disk/volume can be formatted by tools and altered by kernels other than those with FreeDOS, I think you'll need to display the volume name's source somehow anyway if they are different. I think you can see an example of this need here https://gitlab.com/FreeDOS/issue-reporting/-/issues/31

I'm pretty sure that the BPB label really needs to be written if the field exists in the BPB:

• at filesystem creation time (format)
• at the same time as a volume name is created/deleted on the filesystem by the kernel(int21 13/16).

I can see there is one caveat to this, at present (with FreeDOS) the filesystem volume name and normal files / directories share the same namespace, consequently you cannot have a file or directory with the same name as the volume. I'm sure you are aware of these two issues, but I'll reference them anyway FDOS/label#17 and https://gitlab.com/FreeDOS/base/label/-/issues/1 . I had a stab at fixing this collision in the FreeDOS kernel but ran into trouble and out of enthusiasm. I had better luck and a lot of help with the FDPP version see https://github.com/dosemu2/fdpp/pull/202/files.

After seeing an issue/comment/post somewhere this week (it may have been yours?) about int21/69 https://fd.lod.bz/rbil/interrup/dos_kernel/2169.html#3212, I did a little digging into FreeDOS/FDPP and there may already be code to set the label in the BPB without messing about determining the correct location to write. In any case I now know that interface presents another vector for filesystem volume name, and BPB volume name to get out of sync so I guess I'll have to write dosemu tests to cover that, and perhaps see about closing it somehow.

[andrewbird]

From reading the code only I have a little follow up to int21/69, at present it seems the kernel will get the volume name, but not set it.

Get

https://github.com/FDOS/kernel/blob/1cc00e194dd969d30c78775c67a1df44af307abf/kernel/dsk.c#L708-L735

Set

https://github.com/FDOS/kernel/blob/1cc00e194dd969d30c78775c67a1df44af307abf/kernel/dsk.c#L775-L788

I'm off to write some tests now to see what MS-DOS 6.22 does.

[boeckmann]

I'm off to write some tests now to see what MS-DOS 6.22 does.

How do you write these tests and what is needed to run them? Would it be possible to open source them so I can also check the EDR kernel against them?

[andrewbird]

Written using the python unittest framework to create a clean environment for each test, compile any binaries necessary, and usually provide the logic to validate output. Already open source as part of dosemu2, for instance here's the current label test source https://github.com/andrewbird/dosemu2/blob/ci-115/test/func_label_create.py, which provides 10 tests. It requires dosemu2 to run, and currently runs those tests regularly against DR-DOS 7.01 (opendos), MS-DOS 6.22, FreeDOS 1.3, and FDPP current git. Here's a couple of sample runs

ajb@calypso:/clients/common/dosemu2.git$ test/test_dos.py --require-attr=labeltest PPDOSGITTestCase
 
Test PP-DOS-GIT  FAT FCB label create BPB FAT12                                  ... ok (  1.86s)
Test PP-DOS-GIT  FAT FCB label create BPB FAT16                                  ... ok (  1.86s)
Test PP-DOS-GIT  FAT FCB label create BPB FAT32                                  ... ok (  2.21s)
Test PP-DOS-GIT  FAT FCB label create no duplicate                               ... ok (  1.65s)
Test PP-DOS-GIT  FAT FCB label create on top of LFNs                             ... ok (  1.87s)
Test PP-DOS-GIT  FAT FCB label create directory afterwards                       ... ok (  1.74s)
Test PP-DOS-GIT  FAT FCB label create file afterwards                            ... ok (  1.72s)
Test PP-DOS-GIT  FAT FCB label create directory beforehand                       ... ok (  1.71s)
Test PP-DOS-GIT  FAT FCB label create file beforehand                            ... ok (  1.58s)
Test PP-DOS-GIT  FAT FCB label create simple                                     ... ok (  1.75s)

----------------------------------------------------------------------
Ran 10 tests in 18.158s

OK
ajb@calypso:/clients/common/dosemu2.git$ test/test_dos.py --require-attr=labeltest FRDOS130TestCase
 
Test FR-DOS-1.30 FAT FCB label create BPB FAT12                                  ... skipped 'known failure'
Test FR-DOS-1.30 FAT FCB label create BPB FAT16                                  ... skipped 'known failure'
Test FR-DOS-1.30 FAT FCB label create BPB FAT32                                  ... skipped 'known failure'
Test FR-DOS-1.30 FAT FCB label create no duplicate                               ... skipped 'known failure'
Test FR-DOS-1.30 FAT FCB label create on top of LFNs                             ... ok (  0.80s)
Test FR-DOS-1.30 FAT FCB label create directory afterwards                       ... ok (  0.72s)
Test FR-DOS-1.30 FAT FCB label create file afterwards                            ... ok (  0.70s)
Test FR-DOS-1.30 FAT FCB label create directory beforehand                       ... skipped 'known failure'
Test FR-DOS-1.30 FAT FCB label create file beforehand                            ... skipped 'known failure'
Test FR-DOS-1.30 FAT FCB label create simple                                     ... ok (  0.73s)

----------------------------------------------------------------------
Ran 10 tests in 3.031s

OK (skipped=6)

Here's the scheduled weekly test run that does all the default flavours https://github.com/dosemu2/dosemu2/actions/runs/10985259035/job/30496852273. Since it takes a long time, and not wanting to abuse github actions too much, we skip some of the expensive tests there like the 120 libi86, pcmos and freecom build ones.

If a test fails, then you'll obviously see a fail message, and you'll get two files, one the dosemu2 log file, and also the interaction transcript. The test framework mostly uses python-pexpect which is like traditional expect in that you 'send' command text, then 'expect' a set of responses to drive dosemu as if you were typing the commands yourself. So this naturally restricts the type of programs you can capture output from. So I think you'd have difficulty driving a graphical program, even text programs that are driven heavily by mouse could be difficult, but most DOS command utilities and obviously our self written test binaries should be fine.

check the EDR kernel against them?

That should be possible, by creating a class file for it which I suspect is easiest by modifying the one for DR-DOS 7.01 https://github.com/dosemu2/dosemu2/blob/fbf64cee350d67aafa02c18dd5fa26b006f1870f/test/test_dos.py#L5016. In an ideal world the classes action dict would be empty as that details the known fails and unsupported features.

First step is to get a Dosemu2 to boot EDR kernel, and its command.com from its virtual fatfs directory. Note booting from a floppy image is no good for the test.

Got to break off now, going to watch the SpaceX IFT5 launch and 'catch'!

[boeckmann]

Thanks for the detailed explanation!

Got to break off now, going to watch the SpaceX IFT5 launch and 'catch'!

Seems it was catched quite well :) Lets see how re-entry of Spaceship goes... :)

First step is to get a Dosemu2 to boot EDR kernel, and its command.com from its virtual fatfs directory. Note booting from a floppy image is no good for the test.

First step for me would be to get Dosem2 running on my laptop (Mac). Probably I better try on my OpenSUSE Linux machine at home...

I had a quick look at the structure of the tests. Did you consider loading .c files from Python instead inserting the C source as a string into the Python files? Would make it easier to "fetch" the C tests and compile them independent from the test infrastructure.

[andrewbird]

Seems it was catched quite well :)

Well I was skeptical that they'd catch it, but so happy to be wrong!

Lets see how re-entry of Spaceship goes... :)

Less flap burnthrough than last time, still too much, however it was on target, not 6km off this time

First step for me would be to get Dosem2 running on my laptop (Mac)

If you can run a linux distro (ubuntu) in a VM that would work fine. I never tried in a mac, but even KVM nesting works on linux host/linux guest.

Did you consider loading .c files from Python instead inserting the C source as a string into the Python files?

Yes, I started out that way, but many of them are built dynamically from the python. I do output the resultant .c files into the test directory as the test is built/run, so you could grab then from there. However mostly the pass/fail logic is in the python as I can do regex etc.

Have to go out now.