Standardize File Handling with std::filesystem and Expand Cross-Platform Utilities #260
Conversation
Refactored N3Utils.h to introduce various platform-agnostic string utilities, improving code consistency and cross-platform compatibility. Changes include: - Added concepts for CharType, StringType, StringViewType, and PathType. - Introduced case-insensitive comparison functions (iequals, istarts_with, iends_with) for strings and filesystem paths, using a custom ToLower functor to handle case transformations across char types. - Defined platform-specific mappings for string functions, improving portability across Windows, macOS, and Linux. - Updated comments and instructions for verifying compatibility with C++20 compilers (GCC, Clang, MSVC) on godbolt.org. These improvements enhance code reuse and ensure compatibility across multiple platforms, supporting the goal of a more robust, standardized codebase.
stevewgr
requested review from
srmeier,
ForcePower,
DACRepair,
UTengine,
xGuTeK,
cansnoxd,
ko4life-psko,
GiRx8 and
JanekCode
|
Build passing: https://github.com/ko4life-net/ko/actions/runs/11810866975 Let's wait for atleast 2 maintainers (@ko4life-net/stevemaintainers) and 1-2 @ko4life-net/steveteam members to approve this pull-request. 🚀 |
stevewgr
force-pushed
the
std-filesystem
branch
from
975c42f to
397bd2f
Compare
|
Further notes: Although this introduces minor overhead on Windows, it is preferable to using raw It’s also worth mentioning that calling Approach for Identifying Replacements with
|
There was a problem hiding this comment.
Here’s a summary of what I’ve tested:
Compilation: No issues encountered
Server startup and testing: Everything works flawlessly
Client:
- Loading Models/Objects: All good
- Object collisions: Functioning properly
- Loading textures/images: No issues
- Loading sounds: Working perfectly
- Loading UIF files: Everything loads fine
- Loading TBL files: No problems
- Loading skill effects: Appears to be fine—level-up effects are working, but since skills aren't fully functional yet, I can’t test all skill effects. We can revisit this later.
Launcher: No issues
Options: No issues
N3M3: No issues
Kscviewer: No issues
This is a large PR that requires thorough testing. Even if we approve it now, additional testing will be necessary in the future. While it's possible that we may encounter issues later, ideally, they should not arise given that the changes themselves were not overly complex, though there were many of them. Amazing work @stevewgr 🚀 🚀🚀
|
We need to check maps with ponds and rivers |
Ponds and rivers works in terms of loading. I tested with a debugger. The old maps still got some ponds, but no rivers. |
There was a problem hiding this comment.
Great work, @stevewgr! Unfortunately I didn’t have time to review the changes under the tool directory or run the application locally, so I focused on skimming through the code changes, and it looks good to me. 🚀
My comments are just feedback or points for your attention in case you want to further improve it, but overall, I think it’s ready to merge.
This change sets MSSQL 2019 as the minimum SQL Server version. For older versions, the collation can be reverted to `SQL_Latin1_General_CP1_CI_AS` by modifying the `import.ps1` script. However, UTF-8 compatibility is required for proper functionality. UTF-8 ensures consistent behavior regardless of the host system's settings. On Unix, UTF-8 is typically enabled by default. On Windows, it can be enabled under Region settings with "Beta: Use Unicode UTF-8 for worldwide language support." For details, see this discussion: ko4life-net/ko#260 (comment)
This commit introduces `fs::pathx`, a custom wrapper around `std::filesystem::path`, to address limitations in the standard implementation. This extended class provides additional functionality and optimizations while maintaining compatibility with existing code. Key features include: - **Automatic Path Normalization**: FS_AUTO_NORMALIZE automatically standardizes path separators during concatenation with `operator/` or `operator/=`, simplifying path handling, especially on Windows. - **Extended Path Manipulation**: Added functions for case-insensitive path comparisons (`icontains`), relative path construction, and converting paths to lowercase, improving flexibility in handling paths. - **Operator Overloading**: Implemented `operator/` and `operator+=` to streamline path concatenation, along with other convenient overloads. - **Temporary File Creation**: Added `mktemp_file`, a function for creating temporary files with customizable prefixes, suffixes, and directory locations. - **Namespace Redirection**: The new class uses namespace redirection and aliasing to remain unobtrusive in the codebase, making it behave like a native `std::filesystem::path` object. These enhancements offer a more versatile and efficient approach to path management, particularly on Windows, and allow for a cleaner, more maintainable codebase. This approach should remain effective until future standard updates to `std::filesystem` address current limitations.
Added a console log output in CLogWriter::Write to display log messages directly in the console in addition to file output. This enhancement helps improve real-time debugging by making logs immediately visible.
stevewgr
force-pushed
the
std-filesystem
branch
from
57bf029 to
37f5024
Compare
This commit replaces numerous platform-specific WinAPI functions and custom implementations with C++17’s std::filesystem, streamlining and modernizing path and file management across the codebase. The refactor enhances code readability, reduces platform-specific dependencies, and improves cross-platform compatibility. Summary of changes: - **Path Handling**: Replaced custom and WinAPI-based path handling across files, standardizing operations like path concatenation, normalization, and extraction using std::filesystem methods. - **File I/O Operations**: Migrated file reading, writing, and existence checks to use std::filesystem, ensuring cleaner and more uniform file operations across platforms. - **Legacy Code Removal**: Removed obsolete components such as `CAVIPlayer` and other WinAPI-based implementations no longer required with std::filesystem support. - **Bug Fixes**: Addressed existing path-related bugs and inconsistencies that stemmed from the manual path handling methods previously used. - **Code Cleanup**: Deleted unnecessary code comments and unused functions, including those related to deprecated path handling, for a cleaner, more maintainable codebase. This update provides a solid foundation for improved cross-platform support and prepares the codebase for future C++ standard upgrades.
The `branch` option in `.gitmodules` is only relevant when tracking remote changes to a specific branch, such as a dedicated development branch. However, we use `master` as both the default development branch and the tagging mechanism for releases. As we don’t rely on a complex branching strategy, the `branch` tags are unnecessary for our use case.
Note that we released a new 1.2.0 version, as well as added updates related to the changes in this PR where it affects how we read the VERSION table.
Note that ko-db now includes its own submodule for archived migration scripts. To speed-up clonning time, we don't init the submodules recursively.
C++26 plans to make `std::filesystem::path` compatible with `std::format`, but why wait for the future when we can build it ourselves today? This change brings tomorrow's functionality to life. If C++26 eventually delivers the same support, we can gracefully retire our custom solution and let the future take over.
This update improves the `CLogWriter` class by enhancing portability and performance: - Replaced WinAPI's `CreateFile` with `std::fstream` for file operations, ensuring compatibility across platforms. - Updated timestamp generation to use `std::chrono`, improving code clarity and modernizing time handling. - Introduced a `std::mutex` for thread-safe logging to address potential concurrency issues. - Simplified and standardized log messages using `std::format`, improving readability and maintainability. These changes modernize the logging implementation, making it more portable and easier to maintain while adhering to modern C++ standards.
stevewgr
force-pushed
the
std-filesystem
branch
from
37f5024 to
cf24965
Compare
|
Build passing: https://github.com/ko4life-net/ko/actions/runs/11922099338 Thanks everyone for the support and reviews, you're awesome! 💥 Let's ship the ship 🚀
|
Description
This pull request represents a major improvement and modernization of the project’s file handling, path manipulation, string utilities, and logging mechanisms. Key changes include replacing legacy and platform-specific implementations with C++17 and C++20 standards, adding versatile utility functions, and enhancing debugging output for better developer experience and platform compatibility.
It also addresses variable naming inconsistencies within the codebase, particularly when migrating to
std::filesystem. Recommended naming conventions for file paths and their components are as follows:C:/hello/world/file.txt->fsFileorfsFileAbsdepending on the contextC:/hello/world->fsDirworld/file.txt->fsFileorfsFileReldepending on the contextfile.txt->fsFileNamefile->fsFileStemorfsStemdepending on the contextNote: Avoid using
fsFilePath, except as a function name (e.g.,CN3BaseFileAccess::FilePath()). Including “Path” in variable names is redundant, as thefs::pathtype andfsprefix already indicate the variable represents a filesystem path.Highlights
Standardize File Handling with
std::filesystemstd::filesystem, replacing numerous WinAPI and custom functions.Extended Filesystem Wrapper Class
fs::pathxfs::pathx, a custom wrapper aroundstd::filesystem::path, to address limitations in the standard.mktemp_filefor creating temporary files, streamlining file-handling tasks.Refactored Logging with
CLogWriterCLogWriterto usestd::fstreamfor portability,std::chronofor timestamps, andstd::mutexfor thread safety.std::format.Enhanced Utilities with Platform-Agnostic String Functions
N3Utils.hto include case-insensitive string utilities and cross-platform compatibility mappings.Improved Submodule Handling
ko-dbsubmodule to include its latest release (v1.2.0), aligning it with the changes in this PR.Benefits
These updates modernize the codebase, making it more efficient, portable, and easier to maintain. By adopting modern C++ standards and enhancing utilities, the project is better equipped for future growth and cross-platform support.
Testing and Validation
Benefits
These updates provide a more modern and efficient codebase, improving cross-platform support, developer experience, and maintainability. By standardizing utilities and using modern C++ features, this refactor prepares the project for future growth and adherence to industry standards.