Package SchemaCrawler's core types (Catalog/Schema/Table/Relationship etc) for use as generic library, allows for implementing non-JDBC sources

[GavinRay97]

The tool I work on relies heavily on collecting schema information about datasources

For JDBC sources, we can use SchemaCrawler, everything is great.
But lets say that you want to present a schema for a CSV file, or MongoDB, or anything else

This requires coming up with your own generic DTO types and writing mapping code

I think that SchemaCrawler has very mature DTO types and architecture model for representing any kind of hierarchical datasource
It would be great to be able to have a library like schemacrawler.models which exposed these types

Then, users could contribute libraries which can map any datasource into SchemaCrawler schemas
This might be a bit difficult though, as there are some parts which assume JDBC is used, like:

public interface Catalog extends NamedObject, AttributedObject, DescribedObject, Reducible {

  JdbcDriverInfo getJdbcDriverInfo();
}

But maybe it'd be possible to do something like

interface JdbcCatalog extends Catalog {}

What do people think? 🤔

[GavinRay97]

For an example of what I mean, here is code that converts an internal type into a SchemaCrawler catalog representation:

fun queryRequestToCatalog(request: QueryRequest): Catalog {
    val catalog = MutableCatalog("mycatalog", null)

    val relationships = request.table_relationships.associateBy { it.source_table }

    val tableNames = relationships.entries
        .map { it.key + it.value.relationships.flatMap { it.value.target_table } }
        .distinct()

    val schemaNames = tableNames.map { it.first() }.distinct()
    val schemas = schemaNames.associateWith { 
		SchemaReference("mycatalog", it)
	}

    val tables = tableNames.associateWith {
        val (schema, table) = it
        MutableTable(schemas[schema], table)
    }

    catalog.schemas += schemas.values
    catalog.tables += tables.values

    return catalog
}

[sualeh]

@GavinRay97 - SchemaCrawler's object model is heavily influenced by the ISO standards for SQL, and how JDBC views the metadata. I am not sure that this would apply to other models. For example, the standards define the meaning of table constraints, triggers, privileges, and so on, which may not apply as is to other types of models.

That being said, in recent SchemaCrawler releases, I have started opening up pieces of the model to modification. The first opening up was attributes for every object. Next, I introduced the WeakAssociationBuilder to allow external and programmatic specification of relationships. I could open up other parts of the model to modification, but I have been wary of allowing free-form change.