Blacklight and Marc views

[xhero]

Currently Muscat includes a controller for Blacklight to have an in-app OPAC. While Blacklight served us very well in the past it has several drawbacks:

• It is rather difficult to customise to the specific Muscat needs
• It conflicts with the Bootstrap version used, making it not possible to update to newer version
• It has a set of dependencies that we cannot easily keep to date for the above point.

It was then decided that in the future we will move away from Blacklight for our search interface, and in turn develop our own, Muscat+, which is tailored specifically to the data we have in Muscat. The reasons for this are various, but not the least we need a tailored indexer and search interface for the very peculiar data that we have in Muscat.

So I'm opening up this discussion because it can be a good moment to discuss about the future of Blacklight. Since other projects are using Muscat for non musical data or specific installations, we are working on the idea to scorporate Blacklight from the main Muscat application and possibly have it as a separate module that installations can use an customise. This would have the advantage that changes in the Blacklight installation will not modify the cataloguing interface, and also that BL can run on its own indexer separate from the one used in the cataloguing part of the application.

The module or separate installation could be made so some parts are still shared with Muscat - such as the Marc rendering views if needed.

[fjorba]

The truth is that, for me, one of the big selling points for Muscat is that it has Blacklight as end user search interface. And as Blacklight has become a de facto standard in (mostly American) academic library catalogs and digital repositories, I've been using this fact to point that Muscat is quite mainstream, and we are just jumping in.

I think I need more information about your plans to understand its consequences.

[fjorba]

For the library community, Blacklight is more than a piece of software. It is the return of community self-esteem after many years of abandoning the initiative to the commercial vendors, and demonstrating that free software and interinstitutional collaborations pays off, and is a benefit for everybody. You may know this 2015 presentation that explains it very well:

https://academiccommons.columbia.edu/doi/10.7916/D8J38S9M

But, beyond this very important fact, Blacklight is the basis for other important public services, like alternative bibliographic formats and exports, links to external services, specific Marc21 indices, OAI server (I was counting on it for my #1109 proposal), etc. And as a community, they are friendly (in their American friendly way) and receptive; during the last 2020 Blacklight Summit (https://trln.org/event/blacklight-summit/) I could confirm it because they eagerly accepted my Catalan translation (projectblacklight/blacklight#2353), even though they are hardly interested in it.

So yes, I have been selling Muscat due to its Blacklight public interface, because with Blacklight we have most of the features we need; and if there is a way that Muscat can be compatible with Blacklight and its future developments, for us it is of utmost importance. And if I can help, please count on it, as it is my interest to do it.

From my external point of view, I would even suggest to sync Muscat version with Blacklight version (as they do it with Rails), so issues like this Bootstrap incompatibily would be solved. But yes, talk is cheap as I don't know the inner details of this problem.

[ahankinson]

I have some experience in developing, running, and managing several (~10) Blacklight systems in production for a large academic library. As a "product" it is very important to some libraries, as it gives them a sense of community and a "brand name" to sell to their management, since (in my experience) the management people need to know what product they are defending to their bosses. I understand this can be very important, politically. And the people who develop Blacklight it are very friendly -- I know and have worked with many of them quite closely. (I have observed, however, that once you start critiquing their work and questioning their assumptions, some of them can get a bit unfriendly and dismissive...)

Blacklight also has a very good 'out-of-the-box' experience. You install it, plug it in, and it works. This is important for confidence boosting in a new team, and also for selling to the larger organization.

Beyond the initial configuration and demos, however, every Blacklight installation I have seen is highly customized to meet local needs. Once you get into the details of a project, the "out of the box" templates and partials and configuration are rarely suitable for the projects they are used for, so every local installation is highly specific to the institution and the project -- sometimes beyond the point where other members of the Blacklight community can help. I've seen Blacklight projects take over a year to complete, as the complexity of the customizations grew. The end result is that you generally end up removing, disabling, or rewriting large chunks of the application you don't need to meet the local requirements, instead of building a simple solution that addresses what you do need. In hindsight, I would often look back and wonder whether it would have been faster to not have used Blacklight.

Given our current discussions in other threads, I would imagine that the "default" RISM configuration for Blacklight would not be suitable for your needs, @fjorba. You would want to customize it in lots of different ways that would not be compatible with our vision of how we want to present searches to our users. So you would likely be faced with the question of how you want to change Blacklight anyway, and would have local customizations that wouldn't be accepted by the Muscat project. Your choices then are to either a) maintain a separate fork of Muscat or b) install a custom Blacklight instance anyway.

So just because we're planning on removing Blacklight from the Muscat distribution doesn't mean that you can't continue to sell Muscat as "Blacklight compatible." In fact, not having to build your customizations on top of our customizations is probably quite a bit easier, in the long run. The compatibility area of Blacklight and Muscat is very small: if you can put stuff into Solr, you can put Blacklight on top of it. How it gets into Solr is your call -- you can use Traject if that meets your needs better, for example, while we can continue to use our own indexers.

A second advantage of decoupling the two products is that you can run multiple Blacklight instances off of a single Muscat installation -- the same data, framed in different ways. You could, for example, run an internal OPAC that has all of the data available for staff, and an external one that has only the published data for the public. You can keep the MARC editing interface on a private, firewalled server, and only expose the data to the public on a server that has tighter security. These are valuable things that will be easier when the two products are decoupled.

[fjorba]

I understand that this movement will affect (at least) the two Muscat systems that are currently being used as Opac, like the Swiss and the UK ones, right?

• http://www.rism-ch.org/catalog
• https://uk.rism-ch.org/catalog

Out of curiosity, and to better understand your plans and try to align my efforts with you, which are your intentions? To use pure Blacklight? Are you planing to publish the source code in Github as well, so that us, with less experience, can learn about those interfaces between the two systems?

Thanks for sharing your ideas.

[xhero]

Yes simplifying the installation process is one of the goals. Let's say you want to add a facet to the UK RISM site. Right now this is the process:

• Commit a change to Muscat
• Create a new Muscat release
• Update the required system

This works if the changes are sparse, but it we have say 15 different customised sites it becomes very hard very quickly - and it also pollutes a lot the code and commits in Muscat. We have 4 different Muscat installations and managing them is not always easy, for example many times small graphical changes get postponed for months waiting for a release to be ready.

On the other hand if we remove the BL hard dependency on Muscat, all the changes to the catalog can be done without having to touch che cataloguing part and they can be independent. This will also help to keep the Muscat and BL code and commit clean and defined for each project.

The technical details are still fuzzy (and this is why we have the discussion) but the idea is to basically distribute the Muscat BL as a separate component that can be run without the whole Muscat, but still sharing parts of Muscat like the Marc rendering partials. In this way we can more easily have a central Muscat installation and various BLs feeding from it, and each of these BLs can be stored it its own repository so it is easier to customise them. All these Muscat Blacklights will be based on Muscat + Blacklight, basically moving out of the cataloguing part the catalog part we currently have.

[fjorba]

In a way, I'm trying to do a similar thing. Assuming that we'll go with Blacklight, as it provides some of the features we need (as an OAI server, among others), we want to extend the current Invenio collection search to allow isolated searches per branch. I understand that, ignoring the geographical and record sizes, it would be like have each collection tree we have in https://ifmuc.uab.cat/ for the RISM catalog but with countries, right? Of course we do have a similar case for another of our current Invenio systems, like https://ddd.uab.cat/.

So, as I learn about Rails engines (being Blacklight a Rails engine, as I've read), I envision a Blacklight instance where each of the branches of the tree adds a sticky collection-specific field as the field that narrows the search. This way, I think that, even for you, a single BL instance would allow global or specific search for the whole RISM catalog. As Blacklight allows removing the sticky search field, it is very easy to continue searching in a specific branch or go global. Does it make sense?

In my case, I'm already able to export our Invenio 900+ collection definitions, in 3 languages, for our 3 systems and import most of its information to each equivalent test Muscat instances, as Liturgical Feasts records in to start with (as currently I have no specific use for them). Then, my idea would be to automatically transfer them to the Blacklight-based opac system. This is what I'm working on, but I go slow, as I cannot work on it full time.

[ahankinson]

Phasing out those two complete Muscat installations, which we have just so that we could have different public-facing catalogue, is one of the main driving forces behind the move to decouple Blacklight from Muscat. Every night we need to download 6GB SQL dumps, import them into two different MySQL servers, and then index them in two different Solr cores, just so we can drive two different Blacklight installations. This is inefficient and error-prone. Muscat is not designed for “multi-tenant” site delivery.

It is better to use a single Muscat MySQL server, and hook up the public-facing views through dedicated Solr cores for each catalogue. We plan on expanding country-specific search significantly — our users really want the ability to search in any country’s catalogue. Maintaining over 50 separate installations to support this is just not possible.

There really isn’t much magic happening with Blacklight. It is just a friendly UI for Solr. It takes what it finds in the Catalogue controller and looks for those fields in a Solr response. I have previously set up indexing pipelines that take TEI XML, create Solr docs through an XSLT, pass them to Solr, and then have the catalogue controller configured to expect the fields that were indexed. Blacklight is then quite happy to serve and facet as needed.

We will be open sourcing our new catalogue replacement, once it is out of the initial development phase. It is structured quite differently from Blacklight, though. It is built on a micro service architecture and consists of three main components: an indexing service that is responsible for moving the data from MySQL and MARC to Solr, an API server that delivers a public API serving JSON-LD from Solr, and a JavaScript UI that renders the JSON-LD data client-side.

The interface with Muscat is very small. The indexing service is the only part that “knows” anything about MARC or Muscat. If Muscat went offline, or we did some major work that necessitated complex database or record changes that took a long time to do, the indexer is the only layer that would need to change — the rest of the public-facing service would continue working independent of Muscat. We might not be able to index new data, but we could run the service indefinitely for the public while we made the changes.

The indexer service is built with Python, but that doesn’t really matter much. It could be built with any language that reads from MySQL and outputs to Solr. Our current performance is that it can index 5 million Solr documents in under half an hour (but this may change as we index more complex data). The “shape” of the data in Solr is much more friendly to search, rather than simply mimicking MySQL records in Solr — we can tailor the data in the Solr records to meet specific user requirements. One example is that we are using some NLP to parse date ranges from our messy data — we get it out of MySQL, do some NLP analysis, and then store it as a numeric date range field in Solr.

You mentioned using Traject in #1110 — I don’t see why this same process wouldn’t be possible for you: Take your data from Muscat, run it through traject based indexes, and send it to a Solr core. Then just tell any Blacklight installation about the fields and the shape of the data in the core, and it should then provide you with a UI for it. If you want to run two installations, configure a traject indexer to deliver different data to Solr.

We have already made it possible to decouple Solr from Muscat. We can run the latest Solr server for our own data and bypass the built-in service. @xhero can confirm, but I think that eventually we would like to move away from a hard dependency on Solr as well. This would vastly simplify the Muscat installation, but right now there are still lookup fields in the catalogue interface that depend on Solr values, so the timeline on that is still open.

[fjorba]

During the last weeks I have created a draft Rails application that uses the Blacklight engine and indexes the records for my sites, so now I understand it better, how it behaves and its possibilities. It also allows collection navigation like Invenio classic. Although it is not finished and the functionality is very basic, this experience has helped me to conclude that, in this case, having two different systems has more advantages than disadvantages

So yes, please, for me go for it.