Weirdness hooking calls to open() in terminal apps

[sdwannfv]

hook lib

int Hooked_open(const char *path, int flags, ...)
{
    int ret;
    int mode = 0;

    if (flags & O_CREAT) {
        va_list ap;
        va_start(ap, flags);
        mode = va_arg(ap, int);
        va_end(ap);
    }

    ret = open(path, flags, mode);
    LogWithFormat(true, "[open](%s, 0x%x)-->%d", path, flags, ret);
    
    return ret;
}

INTERPOSE_FUNCTION(open),

command

HC_INSERT_LIBRARY=/path/to/hook.dylib  /System/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal

[steven-michaud]

I just tried your testcase on macOS 12.6.3 and 13.2, with HookCase 7.1.1 (the current version), and had no problems.

Remember that you need to codesign hook.dylib if you haven't completely disabled SIP -- if you've used csrutil enable --without kext instead of csrutil disable.

[sdwannfv]

hook.dylib is codesigned, the Terminal process self not quit, but i can't input any commmand, here is log

(Wed Feb  1 14:25:37 2023) /usr/bin/login[1702] [0x11130f600] [open](/usr/bin, 0x0)-->3
(Wed Feb  1 14:25:37 2023) /usr/bin/login[1702] [0x11130f600] [open](/usr/bin/Info.plist, 0x0)-->-1
(Wed Feb  1 14:25:37 2023) /usr/bin/login[1702] [0x11130f600] [open](/etc/nologin, 0x0)-->-1
(Wed Feb  1 14:25:37 2023) /usr/bin/login[1702] [0x11130f600] [open](/System/Library/OpenDirectory/request-schema.plist, 0x0)-->3
Last login: Wed Feb  1 14:25:14 on ttys004

[Process completed]

[steven-michaud]

Now I see your problem. It seems that every time you click on the second Terminal window (the one you just spawned), the whole Terminal app restarts and the (second) window disappears.

I haven't found out why this happens, but I do have a workaround. Add HC_NOKIDS=1 to the command you use to start a second instance of Terminal:

HC_NOKIDS=1 HC_INSERT_LIBRARY=/path/to/hook.dylib  /System/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal

I don't understand why this works.

It's entirely possible that your problem isn't a bug in HookCase. But I'll leave this issue open until I know more. It will probably stay open for quite a long time.

Possible causes that I've already ruled out:

1. That it happens because Terminal doesn't like running in multiple instances: But the problem still happens launching (and hooking) Terminal from another terminal app, Alacritty.
2. That it happens because LogWithFormat() gets re-entered: But preventing this doesn't get rid of the problem.
3. That it's a bug in LogWithFormat(): But it also happens with printf().
4. That it's a bug in HookCase's implementation of interpose hooks: But the problem still happens with a patch hook on __open().

One more thing: The problem goes away if the hook library doesn't do any logging from its open() hook (using either LogWithFormat() or printf()). This is a clue to what's going wrong. But clearly it isn't a viable solution.

[steven-michaud]

This problem may be specific to open(). It doesn't happen hooking some other system calls, like read() and proc_pidinfo().

And, interestingly, it also happens with Alacritty.

[sdwannfv]

it might have had something to do with LogWithFormat, when VIRTUAL_SERIAL_PORT is not defined and LogWithFormat to stdout, it will cause xpc Connection interrupted, it looked like that

2023-02-02 15:42:49.150 TextEdit[2774:66898] +[NSXPCSharedListener endpointForReply:withListenerName:]: an error occurred while attempting to obtain endpoint for listener '': Connection interrupted
2023-02-02 15:42:49.150 TextEdit[2774:66898] failed to warm up class NSOpenAndSavePanelService in service com.apple.appkit.xpc.openAndSavePanelService due to Error Domain=NSCocoaErrorDomain Code=4099 "<dictionary: 0x7ff856cee6c0> { count = 1, transaction: 0, voucher = 0x0, contents =
	"XPCErrorDescription" => <string: 0x7ff856cee8b0> { length = 22, contents = "Connection interrupted" }
} for listener for whole-service management; BEFORE ASSIGNING A BUG TO ANYONE ELSE, search backward for earlier logging regarding service/extension 'com.apple.appkit.xpc.openAndSavePanelService'" UserInfo={NSDebugDescription=<dictionary: 0x7ff856cee6c0> { count = 1, transaction: 0, voucher = 0x0, contents =
	"XPCErrorDescription" => <string: 0x7ff856cee8b0> { length = 22, contents = "Connection interrupted" }
} for listener for whole-service management; BEFORE ASSIGNING A BUG TO ANYONE ELSE, search backward for earlier logging regarding service/extension 'com.apple.appkit.xpc.openAndSavePanelService'}

meanwhile when VIRTUAL_SERIAL_PORT is defined and LogWithFormat to PySerialPortLogger opened tty, xpc Connection interrupted will not occur

[steven-michaud]

Thanks for this information. I'll keep it in mind.