Resolve symbolic links to mags and store them in db #8350
Conversation
|
Important Review skippedReview was skipped due to path filters ⛔ Files ignored due to path filters (1)
CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including You can disable this status message by setting the 📝 WalkthroughWalkthroughThis pull request introduces comprehensive changes to dataset path management and metadata handling in the WEBKNOSSOS application. The modifications span multiple components, including controllers, services, and database schemas. Key additions include new methods for retrieving and updating dataset paths, introducing case classes to represent dataset magnification information, and extending the database schema to support real paths and local data tracking. The changes enhance the system's ability to manage and track dataset metadata more precisely. Changes
Assessment against linked issues
Possibly related PRs
Suggested labels
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
frcroth
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🔭 Outside diff range comments (1)
webknossos-datastore/app/com/scalableminds/webknossos/datastore/storage/RemoteSourceDescriptorService.scala (1)
Line range hint
49-64: Consider using canonical paths and typed exceptions for security and clarity.While this logic properly checks remote vs. local schemes, relying on
startsWithfor whitelisting can be susceptible to path traversal (e.g., embedded"../"). Additionally, throwing rawExceptioncan obscure the root cause. Use a canonical path check (e.g.,toRealPath) and a specialized exception type to enhance security and clarity.
🧹 Nitpick comments (13)
webknossos-datastore/app/com/scalableminds/webknossos/datastore/services/DataSourceService.scala (3)
91-105: Enhance error logging for better debugging.While the error handling is good, the error messages could be more descriptive to aid in debugging.
Consider adding more context to error messages:
- logger.error(s"Failed to determine real paths for mags of $ds: $msg") + logger.error(s"Failed to determine real paths for mags of dataset ${ds.id}: $msg. This may affect path resolution.") - logger.error(s"Failed to determine real paths for mags of $ds") + logger.error(s"Failed to determine real paths for mags of dataset ${ds.id}. This may affect path resolution.")
153-158: Add input validation and documentation.The method would benefit from input validation and clear documentation of its purpose and behavior.
Consider these improvements:
+ /** + * Resolves a potentially relative path against a base path. + * If the relative path is already absolute, returns it unchanged. + * Otherwise, resolves it against the base path and normalizes the result. + * + * @param basePath The base path to resolve against + * @param relativePath The path to resolve + * @return The resolved absolute path + * @throws NullPointerException if either path is null + */ private def resolveRelativePath(basePath: Path, relativePath: Path): Path = { + require(basePath != null, "basePath cannot be null") + require(relativePath != null, "relativePath cannot be null") if (relativePath.isAbsolute) { relativePath } else { basePath.resolve(relativePath).normalize().toAbsolutePath } }
91-158: Consider implementing a dedicated PathResolver service.The new path resolution functionality is well-implemented but could benefit from being extracted into a dedicated service. This would:
- Improve separation of concerns
- Make the path resolution logic more testable
- Reduce the complexity of the DataSourceService class
- Make it easier to modify path resolution strategies in the future
Consider creating a new
PathResolverServiceclass to handle all path resolution logic, including symlink resolution and URI generation. This would make the codebase more maintainable and modular.webknossos-datastore/app/com/scalableminds/webknossos/datastore/storage/RemoteSourceDescriptorService.scala (2)
77-83: Check for invalid local directories and handle them explicitly.If neither
localDirWithScalarMagnorlocalDirWithVec3Magexists, the code silently defaults tolocalDirWithVec3Mag. Updating the logic or adding a warning/error for missing directories can avoid unintentional fallbacks and help debug user misconfiguration.
86-98: Consolidate path resolution logic in a single place.
uriForMagLocatorcallsresolveMagPathand then applies an additionaltoAbsolutePath.toUriif not remote. Consider directly returning an absolute path fromresolveMagPathfor consistency, or unifying their logic to avoid duplication and reduce potential path resolution discrepancies.webknossos-datastore/app/com/scalableminds/webknossos/datastore/services/DataSourceRepository.scala (1)
49-53: Add error handling and logging around real path reporting.This method simply calls
reportRealPathswithout any additional handling for potential failures or partial updates. Consider adding logging or structured error handling to ensure visibility into path reporting issues.webknossos-datastore/app/com/scalableminds/webknossos/datastore/services/DSRemoteWebknossosClient.scala (1)
40-50: Document or validate new path-related fields.The new case classes
DataSourcePathInfoandMagPathInfointroduce important fields (e.g.,layerName,mag,path,realPath,hasLocalData). Brief documentation or validation logic would help ensure correct usage across the codebase, especially formagor path fields.app/controllers/WKRemoteDataStoreController.scala (1)
224-238: Consider adding request logging and error details.The method implementation is correct but could benefit from additional logging to aid in debugging:
- Log the number of path infos being processed
- Include more details in the warning message for invalid JSON
Apply this diff to enhance logging:
def updatePaths(name: String, key: String): Action[JsValue] = Action.async(parse.json) { implicit request => dataStoreService.validateAccess(name, key) { _ => request.body.validate[List[DataSourcePathInfo]] match { case JsSuccess(infos, _) => + logger.info(s"Updating paths for ${infos.length} data sources from data store '$name'") for { _ <- datasetService.updateRealPaths(infos)(GlobalAccessContext) } yield { JsonOk } case e: JsError => - logger.warn("Data store reported invalid json for data source paths.") + logger.warn(s"Data store '$name' reported invalid json for data source paths: ${JsError.toJson(e)}") Fox.successful(JsonBadRequest(JsError.toJson(e))) } } }app/models/dataset/DatasetService.scala (1)
343-356: Consider adding error logging for dataset not found case.The error handling is correct, but adding logging would help track issues in production.
Apply this diff to enhance error logging:
private def updateRealPath(pathInfo: DataSourcePathInfo)(implicit ctx: DBAccessContext): Fox[Unit] = if (pathInfo.magPathInfos.isEmpty) { Fox.successful(()) } else { val dataset = datasetDAO.findOneByDataSourceId(pathInfo.dataSourceId).futureBox dataset.flatMap { case Full(dataset) => datasetMagsDAO.updateMagPathsForDataset(dataset._id, pathInfo.magPathInfos) case Empty => // Dataset reported but ignored (non-existing/forbidden org) + logger.debug(s"Ignoring path update for non-existing dataset: ${pathInfo.dataSourceId}") Fox.successful(()) case e: EmptyBox => + logger.warn(s"Failed to find dataset: ${pathInfo.dataSourceId}", e) Fox.failure("dataset.notFound", e) } }app/controllers/DatasetController.scala (1)
137-149: Consider adding pagination for large datasets.The method correctly implements the required functionality but might need pagination for datasets with many mags.
Apply this diff to add pagination:
-def getLinkedMags(datasetId: String, dataLayerName: String): Action[AnyContent] = +def getLinkedMags(datasetId: String, dataLayerName: String, limit: Option[Int], offset: Option[Int]): Action[AnyContent] = sil.SecuredAction.async { implicit request => for { datasetIdValidated <- ObjectId.fromString(datasetId) _ <- datasetDAO.findOne(datasetIdValidated) ?~> notFoundMessage(datasetId) ~> NOT_FOUND _ <- Fox.bool2Fox(request.identity.isAdmin) ?~> "notAllowed" ~> FORBIDDEN magsAndLinkedMags <- datasetService.getPathsForDatalayer(datasetIdValidated, dataLayerName) - returnValues = magsAndLinkedMags.map { + returnValues = magsAndLinkedMags.drop(offset.getOrElse(0)).take(limit.getOrElse(Int.MaxValue)).map { case (mag, linkedMags) => MagLinkInfo(mag, linkedMags) } } yield Ok(Json.toJson(returnValues)) }conf/webknossos.latest.routes (1)
94-94: Update route documentation.The routes are correctly defined but should be documented in the API documentation.
Add comments above each route to describe their purpose:
+# Get linked magnifications for a dataset layer GET /datasets/:datasetId/layers/:layer/linkedMags controllers.DatasetController.getLinkedMags(datasetId: String, layer: String) +# Update data source paths in the datastore PUT /datastores/:name/datasources/paths controllers.WKRemoteDataStoreController.updatePaths(name: String, key: String)Also applies to: 112-112
app/models/dataset/Dataset.scala (1)
767-784: Consider improving the handling of uninitialized paths.Instead of using a magic string "uninitialized" for null paths, consider:
- Using an Option type to represent potentially uninitialized paths
- Handling the null case at the application level where the context is clearer
- row.path.getOrElse("uninitialized"), - row.realpath.getOrElse("uninitialized"), + row.path.getOrElse(""), + row.realpath.getOrElse(""),conf/evolutions/126-mag-real-paths.sql (1)
1-12: Well-structured SQL evolution with proper safeguards!The SQL evolution:
- Uses transaction for atomicity
- Includes schema version check
- Adds columns with appropriate types
- Sets sensible default for hasLocalData
However, consider adding NOT NULL constraints to path and realPath columns since they are required fields in the DatasetMagInfo case class.
-ALTER TABLE webknossos.dataset_mags ADD COLUMN realPath TEXT; -ALTER TABLE webknossos.dataset_mags ADD COLUMN path TEXT; +ALTER TABLE webknossos.dataset_mags ADD COLUMN realPath TEXT NOT NULL DEFAULT ''; +ALTER TABLE webknossos.dataset_mags ADD COLUMN path TEXT NOT NULL DEFAULT '';
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (13)
MIGRATIONS.unreleased.md(1 hunks)app/controllers/DatasetController.scala(2 hunks)app/controllers/WKRemoteDataStoreController.scala(2 hunks)app/models/dataset/Dataset.scala(4 hunks)app/models/dataset/DatasetService.scala(3 hunks)conf/evolutions/126-mag-real-paths.sql(1 hunks)conf/evolutions/reversions/126-mag-real-paths.sql(1 hunks)conf/webknossos.latest.routes(2 hunks)tools/postgres/schema.sql(2 hunks)webknossos-datastore/app/com/scalableminds/webknossos/datastore/services/DSRemoteWebknossosClient.scala(3 hunks)webknossos-datastore/app/com/scalableminds/webknossos/datastore/services/DataSourceRepository.scala(1 hunks)webknossos-datastore/app/com/scalableminds/webknossos/datastore/services/DataSourceService.scala(3 hunks)webknossos-datastore/app/com/scalableminds/webknossos/datastore/storage/RemoteSourceDescriptorService.scala(2 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: circleci_build
🔇 Additional comments (16)
conf/evolutions/reversions/126-mag-real-paths.sql (3)
1-3: LGTM! Transaction and schema version check are properly implemented.The transaction block ensures atomicity, and the schema version assertion provides a critical safety check against incorrect migrations.
9-11: LGTM! Schema version update and transaction commit are correct.The schema version is properly decremented to 125, and the transaction is committed to ensure all changes are applied atomically.
5-7: Verify the impact of dropping these columns.The column drops align with the PR objectives for managing symlinked mags. However, ensure that:
- Any existing data in these columns has been properly migrated or is safe to discard
- All application code references to these columns are removed
- The corresponding forward migration (126-mag-real-paths.sql) correctly creates these columns
webknossos-datastore/app/com/scalableminds/webknossos/datastore/services/DataSourceService.scala (3)
11-11: LGTM: Required imports added for new functionality.The new imports for
MagLocatorandDataVaultServicealign with the PR's objective to handle mag symlinks.Also applies to: 15-15
81-81: LGTM: Path resolution integrated into inbox scanning.The addition of
sendRealPathscall ensures that real paths are processed and published whenever inbox sources are scanned.
143-151: LGTM: Clean and focused implementation.The method follows the single responsibility principle and clearly handles mag URI resolution.
webknossos-datastore/app/com/scalableminds/webknossos/datastore/storage/RemoteSourceDescriptorService.scala (1)
65-70: Raise an error if neither dataset nor layer path exists.Currently, if
magPathRelativeToDatasetdoes not exist, the code falls back tomagPathRelativeToLayer. Consider raising an error or logging a warning if neither path is found, ensuring graceful handling of invalid file structures.app/controllers/WKRemoteDataStoreController.scala (1)
10-10: LGTM!The import statement correctly includes the new
DataSourcePathInfotype needed for the path update functionality.app/models/dataset/DatasetService.scala (2)
17-17: LGTM!The imports are correctly organized and include all necessary types for the new functionality.
Also applies to: 23-23
357-361: LGTM!The
updateRealPathsmethod correctly serializes updates to maintain consistency.app/controllers/DatasetController.scala (1)
74-78: LGTM!The case class and its JSON format are correctly defined.
app/models/dataset/Dataset.scala (3)
24-24: LGTM!The import for
MagPathInfois correctly added to support the new mag path handling functionality.
712-721: Well-structured data model for mag information!The
DatasetMagInfocase class and its companion object are well-designed:
- Encapsulates all necessary fields for mag path information
- Provides JSON serialization support
- Aligns with the PR objectives for storing symbolic link information
748-765: Robust implementation with proper transaction handling!The implementation includes:
- Proper SQL escaping and type handling
- Transaction with serializable isolation
- Retry logic for handling serialization failures
- Batch updates for efficiency
MIGRATIONS.unreleased.md (1)
21-22: LGTM!The migration guide is correctly updated to include the new SQL evolution.
tools/postgres/schema.sql (1)
23-23: LGTM!The schema changes:
- Correctly update the schema version
- Add the new columns matching the SQL evolution
- Set appropriate default for hasLocalData
Also applies to: 181-183
| private def determineMagRealPathsForDataSource(dataSource: InboxDataSource) = tryo { | ||
| val organizationPath = dataBaseDir.resolve(dataSource.id.organizationId) | ||
| val datasetPath = organizationPath.resolve(dataSource.id.directoryName) | ||
| dataSource.toUsable match { | ||
| case Some(usableDataSource) => { | ||
| usableDataSource.dataLayers.flatMap { dataLayer => | ||
| val rawLayerPath = datasetPath.resolve(dataLayer.name) | ||
| val absoluteLayerPath = if (Files.isSymbolicLink(rawLayerPath)) { | ||
| resolveRelativePath(datasetPath, Files.readSymbolicLink(rawLayerPath)) | ||
| } else { | ||
| rawLayerPath.toAbsolutePath | ||
| } | ||
| dataLayer.mags.map { mag => | ||
| val (magURI, isRemote) = getMagURI(datasetPath, absoluteLayerPath, mag) | ||
| if (isRemote) { | ||
| MagPathInfo(dataLayer.name, mag.mag, magURI.toString, magURI.toString, hasLocalData = false) | ||
| } else { | ||
| val magPath = Paths.get(magURI) | ||
| val realPath = magPath.toRealPath() | ||
| // Does this dataset have local data, i.e. the data that is referenced by the mag path is within the dataset directory | ||
| val isLocal = realPath.startsWith(datasetPath.toAbsolutePath) | ||
| val unresolvedPath = | ||
| rawLayerPath.toAbsolutePath.resolve(absoluteLayerPath.relativize(magPath)).normalize() | ||
| MagPathInfo(dataLayer.name, | ||
| mag.mag, | ||
| unresolvedPath.toUri.toString, | ||
| realPath.toUri.toString, | ||
| hasLocalData = isLocal) | ||
| } | ||
| } | ||
| } | ||
| } | ||
| case None => List() | ||
| } | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add error handling for symlink resolution and improve code structure.
The method handles complex path resolution logic but has some areas for improvement:
- Missing error handling for symlink resolution
- Complex nested structure could be simplified
- Missing documentation for the business logic
Consider these improvements:
private def determineMagRealPathsForDataSource(dataSource: InboxDataSource) = tryo {
+ // Resolves paths for each mag in the data source, handling both local and remote data
+ // Returns a list of MagPathInfo containing the original and resolved paths
val organizationPath = dataBaseDir.resolve(dataSource.id.organizationId)
val datasetPath = organizationPath.resolve(dataSource.id.directoryName)
dataSource.toUsable match {
case Some(usableDataSource) => {
usableDataSource.dataLayers.flatMap { dataLayer =>
val rawLayerPath = datasetPath.resolve(dataLayer.name)
- val absoluteLayerPath = if (Files.isSymbolicLink(rawLayerPath)) {
- resolveRelativePath(datasetPath, Files.readSymbolicLink(rawLayerPath))
- } else {
- rawLayerPath.toAbsolutePath
- }
+ val absoluteLayerPath = try {
+ if (Files.isSymbolicLink(rawLayerPath)) {
+ resolveRelativePath(datasetPath, Files.readSymbolicLink(rawLayerPath))
+ } else {
+ rawLayerPath.toAbsolutePath
+ }
+ } catch {
+ case e: Exception =>
+ logger.error(s"Failed to resolve layer path for ${dataLayer.name}: ${e.getMessage}")
+ rawLayerPath.toAbsolutePath
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| private def determineMagRealPathsForDataSource(dataSource: InboxDataSource) = tryo { | |
| val organizationPath = dataBaseDir.resolve(dataSource.id.organizationId) | |
| val datasetPath = organizationPath.resolve(dataSource.id.directoryName) | |
| dataSource.toUsable match { | |
| case Some(usableDataSource) => { | |
| usableDataSource.dataLayers.flatMap { dataLayer => | |
| val rawLayerPath = datasetPath.resolve(dataLayer.name) | |
| val absoluteLayerPath = if (Files.isSymbolicLink(rawLayerPath)) { | |
| resolveRelativePath(datasetPath, Files.readSymbolicLink(rawLayerPath)) | |
| } else { | |
| rawLayerPath.toAbsolutePath | |
| } | |
| dataLayer.mags.map { mag => | |
| val (magURI, isRemote) = getMagURI(datasetPath, absoluteLayerPath, mag) | |
| if (isRemote) { | |
| MagPathInfo(dataLayer.name, mag.mag, magURI.toString, magURI.toString, hasLocalData = false) | |
| } else { | |
| val magPath = Paths.get(magURI) | |
| val realPath = magPath.toRealPath() | |
| // Does this dataset have local data, i.e. the data that is referenced by the mag path is within the dataset directory | |
| val isLocal = realPath.startsWith(datasetPath.toAbsolutePath) | |
| val unresolvedPath = | |
| rawLayerPath.toAbsolutePath.resolve(absoluteLayerPath.relativize(magPath)).normalize() | |
| MagPathInfo(dataLayer.name, | |
| mag.mag, | |
| unresolvedPath.toUri.toString, | |
| realPath.toUri.toString, | |
| hasLocalData = isLocal) | |
| } | |
| } | |
| } | |
| } | |
| case None => List() | |
| } | |
| } | |
| private def determineMagRealPathsForDataSource(dataSource: InboxDataSource) = tryo { | |
| // Resolves paths for each mag in the data source, handling both local and remote data | |
| // Returns a list of MagPathInfo containing the original and resolved paths | |
| val organizationPath = dataBaseDir.resolve(dataSource.id.organizationId) | |
| val datasetPath = organizationPath.resolve(dataSource.id.directoryName) | |
| dataSource.toUsable match { | |
| case Some(usableDataSource) => { | |
| usableDataSource.dataLayers.flatMap { dataLayer => | |
| val rawLayerPath = datasetPath.resolve(dataLayer.name) | |
| val absoluteLayerPath = try { | |
| if (Files.isSymbolicLink(rawLayerPath)) { | |
| resolveRelativePath(datasetPath, Files.readSymbolicLink(rawLayerPath)) | |
| } else { | |
| rawLayerPath.toAbsolutePath | |
| } | |
| } catch { | |
| case e: Exception => | |
| logger.error(s"Failed to resolve layer path for ${dataLayer.name}: ${e.getMessage}") | |
| rawLayerPath.toAbsolutePath | |
| } | |
| dataLayer.mags.map { mag => | |
| val (magURI, isRemote) = getMagURI(datasetPath, absoluteLayerPath, mag) | |
| if (isRemote) { | |
| MagPathInfo(dataLayer.name, mag.mag, magURI.toString, magURI.toString, hasLocalData = false) | |
| } else { | |
| val magPath = Paths.get(magURI) | |
| val realPath = magPath.toRealPath() | |
| // Does this dataset have local data, i.e. the data that is referenced by the mag path is within the dataset directory | |
| val isLocal = realPath.startsWith(datasetPath.toAbsolutePath) | |
| val unresolvedPath = | |
| rawLayerPath.toAbsolutePath.resolve(absoluteLayerPath.relativize(magPath)).normalize() | |
| MagPathInfo(dataLayer.name, | |
| mag.mag, | |
| unresolvedPath.toUri.toString, | |
| realPath.toUri.toString, | |
| hasLocalData = isLocal) | |
| } | |
| } | |
| } | |
| } | |
| case None => List() | |
| } | |
| } |
| def reportRealPaths(dataSourcePaths: List[DataSourcePathInfo]): Fox[_] = | ||
| rpc(s"$webknossosUri/api/datastores/$dataStoreName/datasources/paths") | ||
| .addQueryString("key" -> dataStoreKey) | ||
| .silent | ||
| .putJson(dataSourcePaths) | ||
|
|
There was a problem hiding this comment.
🛠️ Refactor suggestion
Consider explicit result type and user feedback.
reportRealPaths returns Fox[_] and uses .silent. In case of partial or complete failure, there is no user-facing feedback or logs. Specifying a clearer return type (e.g., Fox[ReportPathsResult]) and/or logging errors can facilitate troubleshooting.
| def getPathsForDatalayer(datasetId: ObjectId, layerName: String): Fox[List[(DatasetMagInfo, List[DatasetMagInfo])]] = | ||
| for { | ||
| magInfos <- datasetMagsDAO.findPathsForDatasetAndDatalayer(datasetId, layerName) | ||
| magInfosAndLinkedMags <- Fox.serialCombined(magInfos)(magInfo => | ||
| for { | ||
| pathInfos <- datasetMagsDAO.findAllByRealPath(magInfo.realPath) | ||
| } yield (magInfo, pathInfos.filter(!_.equals(magInfo)))) | ||
|
|
||
| } yield magInfosAndLinkedMags | ||
|
|
There was a problem hiding this comment.
🛠️ Refactor suggestion
Consider adding input validation for layer name.
The method should validate the layer name before querying the database.
Apply this diff to add validation:
def getPathsForDatalayer(datasetId: ObjectId, layerName: String): Fox[List[(DatasetMagInfo, List[DatasetMagInfo])]] =
for {
+ _ <- assertValidLayerNameLax(layerName)
magInfos <- datasetMagsDAO.findPathsForDatasetAndDatalayer(datasetId, layerName)
magInfosAndLinkedMags <- Fox.serialCombined(magInfos)(magInfo =>
for {
pathInfos <- datasetMagsDAO.findAllByRealPath(magInfo.realPath)
} yield (magInfo, pathInfos.filter(!_.equals(magInfo))))
} yield magInfosAndLinkedMagsCommittable suggestion skipped: line range outside the PR's diff.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
app/models/dataset/Dataset.scala (1)
765-782: Remove unnecessary empty Fox.successful.The empty
Fox.successful(())on line 767 is unnecessary and can be removed.def updateMagPathsForDataset(datasetId: ObjectId, magPaths: List[MagPathInfo]): Fox[Unit] = for { - _ <- Fox.successful(()) updateQueries = magPaths.map(pathInfo => {
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
app/models/dataset/Dataset.scala(4 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: circleci_build
🔇 Additional comments (4)
app/models/dataset/Dataset.scala (4)
712-721: LGTM! Well-structured case class implementation.The
DatasetMagInfocase class and its companion object are well-designed, providing a clean representation of mag path metadata with proper JSON serialization support. The implementation aligns perfectly with the PR objectives for storing mag paths.
736-738: Replace "uninitialized" with empty string as default value.Using "uninitialized" as a default value for paths is not ideal. Consider using an empty string instead, as suggested in previous reviews.
Apply this diff:
- row.path.getOrElse("uninitialized"), - row.realpath.getOrElse("uninitialized"), + row.path.getOrElse(""), + row.realpath.getOrElse(""),
784-794: LGTM! Well-implemented query and mapping logic.The method correctly retrieves and maps mag paths for a dataset and layer, with proper error handling using Fox combinators.
795-810: Replace "uninitialized" with empty string and consider adding an index.
- Replace "uninitialized" with empty string as default value.
- Consider adding an index on the
realPathcolumn since we're querying by it.Apply this diff for the default values:
- row.path.getOrElse("uninitialized"), - row.realpath.getOrElse("uninitialized"), + row.path.getOrElse(""), + row.realpath.getOrElse(""),Consider adding the following index to improve query performance:
CREATE INDEX IF NOT EXISTS dataset_mags_realpath_idx ON webknossos.dataset_mags(realPath);
URL of deployed dev instance (used for testing):
Steps to test:
Overview
In the datastore we now scan for mags and symlinks. Symlinks are resolved for layer directories and mag directories. The result is send to WK via a new route (subject to discussion).
In WK the database stores per mag:
So that
Using hasLocalData and realPath we will be able to handle deletions.
I added a route to get a list of mags that use a mags path, this is for testing purposes and can be deleted before merge.
TODOs:
Issues:
(Please delete unneeded items, merge only when none are left open)