ideas for an "indexes" endpoint #167

PietroLiuzzo · 2019-10-19T15:29:38Z

Dear all, I have been thinking about a possible additional API, which I will temptatively call indexes Endpoint which I am quite sure has been planned many times and perhaps discarded or deferred to the future. I am not sure and I thought it harmless in any case to copy here my thoughts.

I was thinking of a syntax like the following, to have an API which is index-like and allows a navigation of the text which is not only the structure of it, just like in most cases we would anyway have, and be generic enough not to restrict it self to any particular type of index.

/dts/api/indexes?{&id}{&name}{&page}

{
  "@context": {
        "@vocab": "https://www.w3.org/ns/hydra/core#",
        "dc": "http://purl.org/dc/terms/",
        "dts": "https://w3id.org/dts/api#"
  },
  "@type": "IriTemplate",
  "template": "/dts/api/collections/?id={collection_id}&name={index_name}&page={index_page}",
  "variableRepresentation": "BasicRepresentation",
  "mapping": [
    {
      "@type": "IriTemplateMapping",
      "variable": "collection_id",
      "required": false
    },
    {
      "@type": "IriTemplateMapping",
      "variable": "index_name",
      "required": false
    },
    {
      "@type": "IriTemplateMapping",
      "variable": "index_page",
      "required": false
    }
  ]
}

/dts/api/indexes?{&id}

would lists the available supported indexes with their names, depending on what one has marked for such indexes, for example terms, persons, dates, whatever.

Here is a made-up example for a non-readable collection

/api/dts/indexes?id=urn:dts:betmas

{
    "@context": {
        "@vocab": "https://www.w3.org/ns/hydra/core#",
        "dc": "http://purl.org/dc/terms/",
        "dts": "https://w3id.org/dts/api#"
    },
    "@id":"urn:dts:betmas",
    "member": [
      {"name": "persons"},
      {"name": "places"}
    ],
"dts:collection": "/api/dts/collections?id=urn:dts:betmas"
           
}

and one for a readable collection, where additionally I would get direct access to other endpoints, and may find more indexes specific to that only.

/api/dts/indexes?id=urn:dts:betmas:LIT3122Galaw

{
    "@context": {
        "@vocab": "https://www.w3.org/ns/hydra/core#",
        "dc": "http://purl.org/dc/terms/",
        "dts": "https://w3id.org/dts/api#"
    },
    "@id":"urn:dts:betmas:LIT3122Galaw",
    "member": [
      {"name": "persons"},
      {"name": "places"},
      {"name": "terms"}
    ],
"dts:passage": "/api/dts/documents?id=urn:dts:betmas:LIT3122Galaw",
"dts:references": "/api/dts/navigation?id=urn:dts:betmas:LIT3122Galaw",
"dts:collection": "/api/dts/collections?id=urn:dts:betmas:LIT3122Galaw"
           
}

Once one has picked the wanted index, this would have the structure of a normal book index with a key and the passages it is found at, optionally also the actual string annotated, if relevant (it would not mean anything for an index of topics for example)

/dts/api/indexes?{&id}{&name}

would return the desired index. for example, something like

/api/dts/indexes?id=urn:dts:betmas:LIT3122Galaw&name=persons

would have pagination has the collection endpoint and then a number of "attestations" one for each indexed thing. Each of these attestations would come with a series of members, like in the navigation endpoint, for each occurrence in the collection.

{
  "@context" : {
    "dts" : "https://w3id.org/dts/api#",
    "@vocab" : "https://www.w3.org/ns/hydra/core#",
    },
"view" : {
    "first" : "/api/dts/indexes?id=urn:dts:betmas:LIT3122Galaw&page=1",
    "next" : "/api/dts/indexes?id=urn:dts:betmas:LIT3122Galaw&page=2",
    "last" : "/api/dts/indexes?id=urn:dts:betmas:LIT3122Galaw&page=10",
    "previous" : null,
    "@type" : "PartialIndexView",
    "@id" : "/api/dts/indexes?id=urn:dts:betmas:LIT3122Galaw&page=1",
    "totalItems": 200,
    "title" : "index of persons",
 "name" : "persons"
    "description": "index of persons attested in collection",
   
  },
"dts:attestation" :[
{"@id" : "https://betamasaheft.eu/PRS2377Bakaffa",
"@value" : "Bakaffa"
 "member": [
      {"ref": "1.1", 
       "@value" : "መሢሕ፡ ሰገድ፡"},
      {"ref": "1.2", "@value" : "በካፋ፡"},
      {"start": "1.1.1", "end": "1.1.2"}
    ]
},
{"@id" : "https://betamasaheft.eu/PRS3827Ephrem",
 "member": [
      {"ref": "1.3"},
      {"ref": "1.4"},
      {"start": "1.1.1", "end": "1.1.2"}
    ]
}
],
"dts:passage": "/api/dts/documents?id=urn:dts:betmas:LIT3122Galaw",
"dts:references": "/api/dts/navigation?id=urn:dts:betmas:LIT3122Galaw",
"dts:collection": "/api/dts/collections?id=urn:dts:betmas:LIT3122Galaw"
}

I am not sure if it would be useful to include also, &level=2 and &groupSize=2 to determin t
the format of ref and return them as "dts:citeDepth" and "dts:level": 3.

In the special case of places, I think it would be also useful to link to pelagios annotations, with a property for that like in the following

/api/dts/indexes?id=urn:dts:betmas: LIT4633Chronicle&name=places

{
  "@context" : {
    "dts" : "https://w3id.org/dts/api#",
    "@vocab" : "https://www.w3.org/ns/hydra/core#",
    },
"view" : {
    "first" : "/api/dts/indexes?id=urn:dts:betmas:LIT4633Chronicle&page=1",
    "next" : "/api/dts/indexes?id=urn:dts:betmas:LIT4633Chronicle&page=2",
    "last" : "/api/dts/indexes?id=urn:dts:betmas:LIT4633Chronicle&page=10",
    "previous" : null,
    "@type" : "PartialIndexView",
    "@id" : "/api/dts/indexes?id=urn:dts:betmas:LIT4633Chronicle&page=1",
    "totalItems": 200,
    "title" : "index of places",
 "name" : "places"
    "description": "index of persons attested in collection",
   
  },
"dts:pelagios" : "https://betamasaheft.eu/api/placeNames/works/LIT4633Chronicle",
"dts:attestation" :[
{"@id" : "https://betamasaheft.eu/LOC6157WahniA",
 "member": [
      {"ref": "2", 
       "@value" : "ደብረ፡ ወህኒ፡"}
    ]
},
{"@id" : "https://betamasaheft.eu/LOC6376Yebab",
 "member": [
      {"ref": "2", 
        "@value": "ይባባ፡"}
    ]
}
],
"dts:passage": "/api/dts/documents?id=urn:dts:betmas:LIT4633Chronicle",
"dts:references": "/api/dts/navigation?id=urn:dts:betmas:LIT4633Chronicle",
"dts:collection": "/api/dts/collections?id=urn:dts:betmas:LIT4633Chronicle"
}

I think here @value could also be used also if the document api supports CTS like syntax to get passages like urn:dts:betmas:LIT4633Chronicle:2@ደብረ፡ ወህኒ፡[1], or eventually if one, starting from this wants to get directly to the forms attested. I have not added totals for the number of attestations, although it would be useful perhaps.

I think this type of API would allow a very flexible access to the text in one of the most common ways to access texts normally in books or web resources.

The text was updated successfully, but these errors were encountered:

mromanello · 2020-02-28T13:18:08Z

Hi @PietroLiuzzo, many thanks for this proposal. As someone who has recently developed a completely custom JSON API for the index locorum (i.e. index of cited passages from primary sources) of a scholarly publication, I'm very interested in defining ways to standardize how indexes can be exposed and queried.

Here some initial thoughts/questions triggered while reading your ideas above:

are the indexes global (= at collection level) or rather one index per dts document? Would it be possible to allow both? (which does not mean a DTS provider must necessarily implement)
in dts:attestation it would be useful to allow for custom RDF statements, for example to assert that "my" Bakaffa has a corresponding entry in e.g. Wikidata, or that a given place has a corresponding entry in Geonames or Pleiades. The use case for this request is the aggregation of indexes across several DTS endpoints; common identifiers provide one possible way to automatically establish identity of entries in multiple indexes (Bakaffa in the index of names of doc A is the same Bakaffa as in the index of names in doc B).
always for the same use case of aggregation, it'd be useful to expose index types as terms defined in the dts namespace (working as a controlled vocabulary) rather than leaving this as a free text string. Well, at least the most common index types (index of place names, index of people, index locorum, etc.)
are nested structures allowed in dts:attestation? For an index of citations, it'd be useful to group all document passages citing lines of the Iliad under Homer > Iliad, and then by book, etc.

PietroLiuzzo · 2020-06-17T18:36:04Z

Ideally I think indexes could be at any collection or passage level:
index for the collection, for the entire document, for a section of if. However, if an index for a resource does need only dts:ref, what would then such a reference look like at collection level?

{
    "dts:attestations": [
        {
            "member": {
                "@lang": "gez",
                "@value": "ደብረ፡ ዳሞ፡",
                "dts:ref": "152"
            },
            "@id": "https://betamasaheft.eu/INS0105DD"
        },
        {
            "member": [
                {
                    "@lang": "gez",
                    "@value": "ደብረ፡ ሊባኖስ፡",
                    "dts:ref": "109"
                },
                {
                    "@lang": "gez",
                    "@value": "ደብረ፡ ሊባኖስ፡",
                    "dts:ref": "231"
                }
            ],
            "@id": "https://betamasaheft.eu/INS0341DL"
        }
    ]
}

in the above example I have in dts:ref the same I would have in the navigation endpoint for the deepest possible level of reference for the attestation. But if the request is for a collection, and I want to group by attestation, the dts:ref would need to be a full URI or I should we group by source within the attestation and specify a "@source" kind of thing?

{
    "dts:attestations": [
        {
            "@id": "https://betamasaheft.eu/INS0341DL",
            "source": [
                {
                    "@id": "https://betamasaheft.eu/LIT1234",
                    "member": [
                        {
                            "@lang": "gez",
                            "@value": "ደብረ፡ ሊባኖስ፡",
                            "dts:ref": "109"
                        },
                        {
                            "@lang": "gez",
                            "@value": "ደብረ፡ ሊባኖስ፡",
                            "dts:ref": "231"
                        }
                    ]
                },
                {
                    "@id": "https://betamasaheft.eu/LIT5678",
                    "member": [
                        {"dts:ref": "1.3.4"},
                        {"dts:ref": "2.3.1"}
                    ]
                }
            ]
        }
    ]
}

on your second point I fully agree, again I am not sure where that optional information may be better placed. skos:*Match ? or simply making an array of the value of @id

{
    "@id": "https://betamasaheft.eu/INS0341DL",
    "skos:exactMatch": "https://www.wikidata.org/entity/Q12345678",
    "member": [
        {"dts:ref": "1.a"},
        {"dts:ref": "2.c.i"}
    ]
}

on the index types defined in dts, I am not actually sure, but I think it can do no harm, especially if it does no mean that these are the only ones one can make.

I also agree on the last point, and I think I would opt for the same structure of the navigation api with citeType. if I understand well here we want to state the structure of the citation. let's say that, in an ideal word, the value of my index locorum is also a DTS uri, e.g. in my resource at 1.a and 2.c.i there is a citation of Gen. 1.1., I could have something like this

{
    "@id": "https://betamasaheft.eu/LIT1546Genesi&ref=1.1&level=2",   
    "label": "Gen. 1.1",
    "type": "index locorum",
    "member": [
        {"dts:ref": "1.a"},
        {"dts:ref": "2.c.i"}
    ]
}

but I would need to lookup in the collection API for dts:citeStructure, perhaps this could be served directly?

{
    "@id": "https://betamasaheft.eu/LIT1546Genesi",
    "dts:ref" : "1.1",
    "dts:level" : 2 ,
    "dts:citeDepth": 2,
    "dts:citeStructure": [
        {
            "dts:citeType": "chapter",
            "dts:citeStructure": [
                {"dts:citeType": "verse"}
            ]
        }
    ],
    "label": "Gen. 1.1",
    "type": "index locorum",
    "member": [
        {"dts:ref": "1.a"},
        {"dts:ref": "2.c.i"}
    ]
}

mromanello · 2020-06-18T08:28:23Z

Some additional thoughts from today's chat:

need to specify the exact location of the index text within its context

Examples:

a place name mentioned within one single line of text
a person name mentioned in a longer paragraph (300 words long).

To complicate things, the same indexed entity (i.e. Homer) may be mentioned several times within the same context. The index will need to contain each individual attestation, and provide a way to unambiguously find this attestation in its context (use case: client application displays the highlighted indexed term in its context).

The solution to this depends on the type of content:

for TEI/XML content it can be an xpath to the indexed passage
for JSON or plain text it will have to be an absolute character offset

(@PietroLiuzzo can you mock up an example for the xpath solution based on what you already have?)

behaviour of index endpoint at collection vs document level

2a) One option that would work well at different levels (i.e. keeping same structure of the response) is to move @id inside members like in the following

{
    "dts:attestations": [
        {
            "member": {
                "@lang": "gez",
                "@value": "ደብረ፡ ዳሞ፡",
                "dts:ref": "152",
                "@id": "https://betamasaheft.eu/INS0105DD"
            }
        }
       ... here the list continues ...
    ]
}

2b. The other option, sketched by @PietroLiuzzo above, is to use @source inside attestations to group indexed elements by containing document. The "problem" with this, is that the response structure changes depending on the index level and the client will have to guess which one it received (a document index or a collection index?).

This way, the response structure does not change when the index level changes (collection vs document).

embed dts:citeStructure information in the index endpoint response

From an implementation point of view this seems desirable, as to limit the number of calls to diff. endpoints that the client will have to make. However, this is in conflict with one of the two options for point 2 above. It can work with 2b (@source used to group attestations at collection level) but it will become super redundant with 2a (@id gets moved inside members)

PonteIneptique · 2020-06-18T08:35:36Z

While I am not against it in general, I would heavily recommend that we do not reinvent the wheel: I really think we should, if we do this, build onto the WebAnnotation framework : http://webannotation.org/

mromanello · 2020-06-18T09:08:32Z

@PonteIneptique yes, Web Annotation was exactly what we were thinking of concerning point #1 above. Or is it a more general point (along the lines of "an index is just a set of annotations with the purpose of indexing")?

PonteIneptique · 2020-06-18T09:15:24Z

It's kinda both, I have to admit. I really think we should try to frame this functionality in the context of both Hydra and WebAnnotation, except if there is a spec for a webannotation API.

But it looks to me like Index is a collection of annotations, so you could have, in terms of APIs:

AnnotationCollection -(n)> WebAnnotation where things are derived from both. The nice thing about it, is that it would go beyond indexes...

PonteIneptique · 2020-06-18T09:17:47Z

And I'd probably allow @source in target of WebAnnotation to be a Navigation object, ie {ref, start, end, documentid, level?, citationtype?}

PietroLiuzzo · 2020-06-18T10:20:08Z

Sorry @PonteIneptique could I please ask you to unpack some of this with an example? thanks!

PonteIneptique · 2020-06-18T12:37:54Z

@PietroLiuzzo I'll try to do that in the following days, please remind me if I don't L(

balmas · 2020-06-18T14:53:36Z

I'm not sure but it could also be useful to pay attention to the work of the LD4LT group here, it seems they are also thinking about issues of interoperability of linguistic annotations https://github.com/ld4lt/linguistic-annotation

PonteIneptique · 2020-06-19T15:07:18Z

Example of navigation:

foobar.com/dts/annotations

{
    "@context": {
        "@vocab": "https://www.w3.org/ns/hydra/core#",
        "dc": "http://purl.org/dc/terms/",
        "dts": "https://w3id.org/dts/api#"
    },
    "@id": "//foobar.com/foo/annotations",
    "@type": "AnnotationCollection",
    "totalItems": 2,
    "dts:totalParents": 0,
    "dts:totalChildren": 2,
    "title": "Annotations Root Collection",
    "dts:dublincore": {
        "dc:publisher": ["École Nationale des Chartes"],
    },
    "member": [
        {
             "@id" : "//foobar.com/foo/annotations/indexes",
             "title" : "Index collection",
             "description": "Collection of Index",
             "@type" : "AnnotationCollection",
             "totalItems" : 2,
             "dts:totalParents": 1,
             "dts:totalChildren": 2
        },
        {
             "@id" : "//foobar.com/foo/annotations/linguistic_annotation",
             "title" : "Linguistic Annotations Collection",
             "description": "Collection of Linguistic Annotations",
             "@type" : "Collection",
             "totalItems" : 2,
             "dts:totalParents": 1,
             "dts:totalChildren": 2
        }
    ]
}

foobar.com/dts/annotations?id=//foobar.com/foo/annotations/indexes

{
    "@context": {
        "@vocab": "https://www.w3.org/ns/hydra/core#",
        "dc": "http://purl.org/dc/terms/",
        "dts": "https://w3id.org/dts/api#"
    },
    "@id" : "//foobar.com/foo/annotations/indexes",
    "title" : "Index collection",
    "description": "Collection of Index",
    "@type" : "AnnotationCollection",
    "totalItems": 2,
    "dts:totalParents": 0,
    "dts:totalChildren": 2,
    "dts:dublincore": {
        "dc:publisher": ["École Nationale des Chartes"],
    },
    "member": [
        {
             "@id" : "//foobar.com/foo/annotations/indexes/texts",
             "title" : "Texts Indexes",
             "description": "Collection of indexes for each book",
             "@type" : "AnnotationCollection",
             "totalItems" : 2,
             "dts:totalParents": 1,
             "dts:totalChildren": 2
        },
        {
             "@id" : "//foobar.com/foo/annotations/indexes/general",
             "title" : "General Index",
             "description": "General Index",
             "@type" : "AnnotationCollection",
             "totalItems" : 2,
             "dts:totalParents": 1,
             "dts:totalChildren": 2
        }
    ]
}

foobar.com/dts/annotations?id=//foobar.com/foo/annotations/indexes

{
    "@context": {
        "@vocab": "https://www.w3.org/ns/hydra/core#",
        "dc": "http://purl.org/dc/terms/",
        "dts": "https://w3id.org/dts/api#"
    },
	"@id" : "//foobar.com/foo/annotations/indexes/general/general",	
	"title" : "General Index",
	"description": "General Index",
    "@type" : "AnnotationCollection",
    "totalItems": 2,
    "dts:totalParents": 0,
    "dts:totalChildren": 2,
    "dts:dublincore": {
        "dc:publisher": ["École Nationale des Chartes"],
    },
    "member": [
        {
		  "@context": "http://www.w3.org/ns/anno.jsonld",
		  "id": "http://example.org/anno3",
		  "type": "Annotation",
		  "creator": {
		    "id": "http://example.org/user1",
		    "name": "Foo",
		    "nick": "Bar"
		  },
		  "body": [
		    {
		      "role": "identifying",
		      "text": "Place1",
		      "id": "http://example.org/place1"
		    }
		  ],  
		  "target": {
		    "source": {
				// Optional level?, citationtype?
				"type": "Resource",
				"ref": "Passage1",
				"id": "foobar.com/Document2",
				"passage" : "foobar.com/dts/document?id={id}&ref={ref}"
		    },
		    "selector": {
		      "type": "TextQuoteSelector", // Or any other selection
		      //....
		    }
		  }
		}
        // Other Annotation
    ],
    "view": {
    	// Pagination information
    }
}

PietroLiuzzo · 2020-06-19T15:22:34Z

thanks @PonteIneptique !
I think I understand a bit more. I will try it out and see how far I get in managing my indexes. basically what I had as named indexes will be annotation collections has you already said and the web annotation will be there to model each single member. grouping I guess would be for the client to do?

PonteIneptique · 2020-06-19T15:25:58Z

I think it can be up to the client, but the provider should probably, if he is doing indexes, have multiple target with the same body. Note that this is an idea, and in no way vouched by the Technical Committee. Le ven. 19 juin 2020 à 17:22, Pietro Liuzzo <[email protected]> a écrit :

…

thanks @PonteIneptique <https://github.com/PonteIneptique> ! I think I understand a bit more. I will try it out and see how far I get in managing my indexes. basically what I had as named indexes will be annotation collections has you already said and the web annotation will be there to model each single member. grouping I guess would be for the client to do? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#167 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOXEZTES4A7ZQV4HHAZ2VDRXN7ETANCNFSM4JCQIONA> .

PietroLiuzzo · 2020-06-19T15:27:19Z

webannotation Technical Committee or DTS?

PonteIneptique · 2020-06-19T15:32:39Z

DTS :) Le ven. 19 juin 2020 à 17:27, Pietro Liuzzo <[email protected]> a écrit :

…

webannotation Technical Committee or DTS? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#167 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOXEZRQ5KTRWSLF6QOHKI3RXN7WLANCNFSM4JCQIONA> .

PietroLiuzzo · 2020-06-24T13:09:15Z

Soooo I have given it a go with my DTS allocated time, and I think it does work decently.

/api/dts/annotations

this fetches the top collection of annotations, which I limit to three which are related to collections of TEI files + one which has them all, i.e. all our annotations for any given indexed thing.

{
  "dts:totalChildren": 4,
  "@context": {...},
  "dts:dublincore": {...},
  "title": "Annotations Root Collection",
  "member": [
    {
      "dts:totalChildren": 7,
      "title": "Annotations of works Root Collection",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works",
      "dts:totalParents": 1,
      "totalItems": 7
    },
    {
      "dts:totalChildren": 7,
      "title": "Annotations of mss Root Collection",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/mss",
      "dts:totalParents": 1,
      "totalItems": 7
    },
    {
      "dts:totalChildren": 7,
      "title": "Annotations of narr Root Collection",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/narr",
      "dts:totalParents": 1,
      "totalItems": 7
    },
    {
      "dts:totalChildren": 7,
      "title": "Annotations of all Root Collection",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/all",
      "dts:totalParents": 1,
      "totalItems": 7
    }
  ],
  "@type": "AnnotationCollection",
  "@id": "https://betamasaheft.eu/api/dts/annotations",
  "dts:totalParents": 0,
  "totalItems": 4
}

/api/dts/annotations/works

this has a collection of annotation collections, one for each available index, which I also decide ahead of time. beside these another collection of annotation collection, /items/ is just organized differently, and has a collection of annotation collections one for each item. this way one can drill down to a more specific context.

{
  "dts:totalChildren": 6,
  "@context": {...},
  "dts:dublincore": {...},
  "title": "Annotations of works Root Collection",
  "member": [
    {
      "dts:totalChildren": 154,
      "title": "Index of persons for works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/persons",
      "dts:totalParents": 1,
      "totalItems": 154
    },
    {
      "dts:totalChildren": 25,
      "title": "Index of places for works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/places",
      "dts:totalParents": 1,
      "totalItems": 25
    },
    {
      "dts:totalChildren": 7249,
      "title": "Index of keywords for works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/keywords",
      "dts:totalParents": 1,
      "totalItems": 7249
    },
    {
      "dts:totalChildren": 3,
      "title": "Index of loci for works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/loci",
      "dts:totalParents": 1,
      "totalItems": 3
    },
    {
      "dts:totalChildren": 7,
      "title": "Index of works for works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/works",
      "dts:totalParents": 1,
      "totalItems": 7
    },
    {
      "dts:totalChildren": 7262,
      "title": "Annotations of each item in works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/items",
      "dts:totalParents": 1,
      "totalItems": 7262
    }
  ],
  "@type": "AnnotationCollection",
  "@id": "https://betamasaheft.eu/api/dts/annotations/works",
  "dts:totalParents": 1,
  "totalItems": 6
}

/api/dts/annotations/works/items

While the previous ones do not really need to be paginated, there are always less than 8 annotation collections, there are a lot of items in a resource collection, thus, here they are paginated.

{
  "view": {
    "name": "items",
    "first": "/api/dts/annotations/works/items?page=1",
    "next": "/api/dts/annotations/works/items?page=2",
    "last": "/api/dts/annotations/works/items?page=727",
    "previous": null,
    "title": null,
    "@type": "PartialIndexView",
    "@id": "/api/dts/annotations/works/items?page=1",
    "totalItems": 7262
  },
  "dts:totalChildren": 7262,
  "@context": {...},
  "dts:dublincore": {...},
  "title": "Annotations of each item in works",
  "member": [
    {
      "dts:totalChildren": 1,
      "title": "Annotations of فصل في أصول علم الرمل in works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT3551IHA",
      "dts:totalParents": 3,
      "totalItems": 1
    },
    {
      "dts:totalChildren": 1,
      "title": "Annotations of Collection of Praises of the Prophet in works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT1527IHA",
      "dts:totalParents": 3,
      "totalItems": 1
    },
...
  ],
  "@type": "AnnotationCollection",
  "@id": "https://betamasaheft.eu/api/dts/annotations/works/items",
  "dts:totalParents": 3,
  "totalItems": 7262
}

/api/dts/annotations/works/places

will return the index of places for the resource collection works and will return one annotation collection for each referenced entity

{
  "view": {
    "name": "places",
    "first": "/api/dts/annotations/works/places?page=1",
    "next": "/api/dts/annotations/works/places?page=2",
    "last": "/api/dts/annotations/works/places?page=91",
    "previous": null,
    "title": "Index of places",
    "@type": "PartialIndexView",
    "@id": "/api/dts/annotations/works/places?page=1",
    "totalItems": 902
  },
  "dts:totalChildren": 25,
  "@context": {...},
  "dts:dublincore": {...}
    ]
  },
  "title": "Index of places for works",
  "member": [
    {
      "dts:totalChildren": "127",
      "title": "Annotations of Goǧǧām in places index.",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/LOC3549Gojjam",
      "dts:totalParents": 3,
      "totalItems": "127"
    },
    {
      "dts:totalChildren": "59",
      "title": "Annotations of ʾAbbāy in places index.",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/LOC1022Abbay",
      "dts:totalParents": 3,
      "totalItems": "59"
    },
..... 
  ],
  "@type": "AnnotationCollection",
  "@id": "https://betamasaheft.eu/api/dts/annotations/works/places",
  "dts:totalParents": 1,
  "totalItems": 25
}

/api/dts/annotations/works/places?id=https://betamasaheft.eu/LOC3549Gojjam

the id here can be used as a parameter (so that it need not to be hierarchical) and finally we get some annotations!

{
  "view": {
    "name": "places",
    "first": "/api/dts/annotations/works/places?id=https://betamasaheft.eu/LOC3549Gojjam&page=1",
    "next": "/api/dts/annotations/works/places?id=https://betamasaheft.eu/LOC3549Gojjam&page=2",
    "last": "/api/dts/annotations/works/places?id=https://betamasaheft.eu/LOC3549Gojjam&page=13",
    "previous": null,
    "title": "Index of places",
    "@type": "PartialIndexView",
    "@id": "/api/dts/annotations/works/places?id=https://betamasaheft.eu/LOC3549Gojjam&page=1",
    "totalItems": 127
  },
  "dts:totalChildren": 127,
  "@context": {...},
  "dts:dublincore": {....},
  "title": "Annotations of Goǧǧām in places index.",
  "member": [
    {
      "body": [
        {
          "text": "ጐዣም፡",
          "@lang": "gez",
          "role": "placeName",
          "id": "https://betamasaheft.eu/LOC3549Gojjam"
        }
      ],
      "@context": "http://www.w3.org/ns/anno.jsonld",
      "target": {
        "source": {
          "ref": null,
          "dts:passage": "/api/dts/document?id=https://betamasaheft.eu/LIT3951ChronSusenyos&ref=11",
          "type": "Resource",
          "link": "https://betamasaheft.eu/LIT3951ChronSusenyos.11",
          "id": "https://betamasaheft.eu/LIT3951ChronSusenyos",
          "dts:references": "/api/dts/navigation?id=https://betamasaheft.eu/LIT3951ChronSusenyos&ref=11"
        },
        "selector": {
          "@value": "TEI/text/body/div[2]/div[11]/ab/placeName[1]",
          "type": "XPath"
        }
      },
      "type": "Annotation"
    },
    {
      "body": [
        {
          "text": "ጐዣም።",
          "@lang": "gez",
          "role": "placeName",
          "id": "https://betamasaheft.eu/LOC3549Gojjam"
        }
      ],
      "@context": "http://www.w3.org/ns/anno.jsonld",
      "target": {
        "source": {
          "ref": null,
          "dts:passage": "/api/dts/document?id=https://betamasaheft.eu/LIT3951ChronSusenyos&ref=11",
          "type": "Resource",
          "link": "https://betamasaheft.eu/LIT3951ChronSusenyos.11",
          "id": "https://betamasaheft.eu/LIT3951ChronSusenyos",
          "dts:references": "/api/dts/navigation?id=https://betamasaheft.eu/LIT3951ChronSusenyos&ref=11"
        },
        "selector": {
          "@value": "TEI/text/body/div[2]/div[11]/ab/placeName[3]",
          "type": "XPath"
        }
      },
      "type": "Annotation"
    },
    .... other annotations
  ],
  "@type": "AnnotationCollection",
  "@id": "https://betamasaheft.eu/LOC3549Gojjam",
  "dts:totalParents": 3,
  "totalItems": 127
}

/api/dts/annotations/works/items/LIT3122Galaw

if I go down the /items/ road, I can then have the list of indexes for one work (at this point I am not supporting editions for this, but it would just be yet another level of collections)

{
  "dts:totalChildren": 4,
  "@context": {...},
  "dts:dublincore": {...},
  "title": "Annotations of Chronicle of Galāwdewos in works",
  "member": [
    {
      "dts:totalChildren": 60,
      "title": "Index of persons for Chronicle of Galāwdewos in works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT3122Galaw/persons",
      "dts:totalParents": 1,
      "totalItems": 60
    },
    {
      "dts:totalChildren": 181,
      "title": "Index of places for Chronicle of Galāwdewos in works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT3122Galaw/places",
      "dts:totalParents": 1,
      "totalItems": 181
    },
    {
      "dts:totalChildren": 3,
      "title": "Index of keywords for Chronicle of Galāwdewos in works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT3122Galaw/keywords",
      "dts:totalParents": 1,
      "totalItems": 3
    },
    {
      "dts:totalChildren": 1,
      "title": "Index of works for Chronicle of Galāwdewos in works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT3122Galaw/works",
      "dts:totalParents": 1,
      "totalItems": 1
    }
  ],
  "@type": "AnnotationCollection",
  "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT3122Galaw",
  "dts:totalParents": 3,
  "totalItems": 4
}

/api/dts/annotations/works/items/LIT3122Galaw/persons

and

/api/dts/annotations/works/items/LIT3122Galaw/persons?id=https://betamasaheft.eu/PRS1522Ahmadb

would be then the same, but at a deeper level of context (the most obvious one, the one of the indexes for one edition).

the index of loci is a special beast I think here an example of what I have tried

{
  "dts:totalChildren": 2,
  "@context": {...},
  "dts:dublincore": {...},
  "title": "Index of loci for Zenā mǝssāleyāt za-ṭabib Fisǝʾalgos in works",
  "member": {
    "body": [
      {
        "text": "ተመሰልኩ፡ ጰልቃን፡ ጳልቃን፡ ዘገዳም።",
        "@lang": "gez",
        "role": "ref",
        "id": "betmas:LIT2000Mazmur.101.7"
      }
    ],
    "@context": "http://www.w3.org/ns/anno.jsonld",
    "target": {
      "source": {
        "dts:citeType": "sentence",
        "dts:level": "1",
        "dts:citeDepth": 2,
        "dts:passage": "/api/dts/document?id=https://betamasaheft.eu/LIT4916PhysB&ref=4.2",
        "type": "Resource",
        "link": "https://betamasaheft.eu/LIT4916PhysB.4.2",
        "id": "https://betamasaheft.eu/LIT4916PhysB",
        "dts:ref": "4.2",
        "dts:references": "/api/dts/navigation?id=https://betamasaheft.eu/LIT4916PhysB&ref=4.2"
      },
      "selector": {
        "@value": "TEI/text/body/div[2]/div/div[2]/ab/ref",
        "type": "XPath"
      }
    },
    "type": "Annotation"
  },
  "@type": "AnnotationCollection",
  "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT4916PhysB/loci",
  "dts:totalParents": 4,
  "totalItems": 2
}

in the navigation object I am not doing any start-end, because luckily enough I do not have any of those things to index, only things which would be within a referenceable unit.
I also make up a link property in source and a type property in selector, and certainly some other stuff and I certainly have abused a couple of things here and there. I needed to try it out. At least for my needs this would serve well, and I think it does have what we discussed above, and improves on the fact that the annotations are not different from one level to the other, they will always and consistently be annotation collections at any level.

PonteIneptique · 2020-06-24T16:15:46Z

Regarding your last comment, I'd recommend to check the XPathSelection type : https://www.w3.org/TR/annotation-model/#xpath-selector

Note that I don't know if, for an index, I'd repeat the target multiple time or have separate Annotation (Ie one annotation targeting all different presence of specific places vs. multiple annotation with a single target vs. one annotation with multiple target in the same document).

What you experiment shows though is the important need of a filtering system (and believe me, I don't want to, but I feel we'll have to finally address this...)

balmas · 2020-06-24T18:19:27Z

There is already a Web Annotation API protocol: https://www.w3.org/TR/annotation-protocol/

I think we have to have a compelling reason to implement our own

PietroLiuzzo · 2020-06-24T18:49:58Z

My best reason I think is just: I like DTS better! My real reason would be that I want it tightly linked with the other DTS APIs, to the navigation and document especially, because after all, with an index what I want to do is simply go back to that fragment of text where the indexed term occurs and navigate from there. In my head, and I may be simply failing to abstract enough, an index, whatever its format is, offers a navigation and access feature as much as the structure of passages which I my have and offer in my navigation API, it actually uses that system of reference and has its scope only there. It does not tell me "this is an annotation with this target and this body", it tells me "this thing is present at this passage" or for loci "this passage of text x is cited at this passage of text y". I would love to also have, as there are dts:passage and dts:references a dts:annotations or dts:indexes that would allow me to jump to different set of annotations referring to an id or available at that level of context, i.e. what is indexed here and where is this passage indexed. It would greatly help with cross quotations of ancient texts. As I said, I see how this can be abstracted to fit the same annotation model, but I feel it is much closer and tightly connected to the DTS navigation API than to a tag or a post on a blog. If the same can be used with profit, I am ok with it, I do not have any problem, all on the contrary, however my feeling is that this is actually a different thing. This is probably not compelling, but it is part of my reasons for doing this.
On the filtering I agree, and I could easily add a parameter for a search term which would allow also for facets values to drill further down on the results and narrow down collections this way to have an index of persons in passages in the collection matching the query string and with a specific value for a facet.

PonteIneptique · 2020-06-25T07:47:22Z

There is already a Web Annotation API protocol: https://www.w3.org/TR/annotation-protocol/
I think we have to have a compelling reason to implement our own

I agree with @balmas , I just could not find it again. Looking at it from afar, I think we can reuse the container system with an AnnotationCollection DTS system for discovery, but this would create an hybrid system, and I am not sure I like this idea (like member DTS property + annotation Protocol Items property ? No way it's gonna be easy to parse).

Maybe we should look again at https://iiif.io/api/presentation/3.0/#55-annotation-page : Annotations are members of the canvases and served through the Annotation Protocol with specific pointers.

@balmas I don't think I have seen any discoverability system in the WebAnnotation Protocol. Any pointers ? Because for indexes at least, that's what people are gonna want.

PonteIneptique · 2020-06-25T07:53:15Z

Given that I am developing my though as I write (which is probably a bad idea ?):

Maybe we should look again at https://iiif.io/api/presentation/3.0/#55-annotation-page : Annotations are members of the canvases and served through the Annotation Protocol with specific pointers.

Definitely, I can imagine having a dts:annotations + dts:indexes (which would differentiate both kind of annotations ?) in Collection, Resource and Reference (Navigation Endpoint Objects should have a type #184) object.
If there is no discoverability / curating system for annotations in the WebAnnotation protocol, we should either: propose one to them (Not sure I want to go there, but you know the XKCD comics about creatings standards...) or build onto what DTS does to provide one, which would remove the need for a specific item browser.

balmas · 2020-06-25T13:27:38Z

@balmas I don't think I have seen any discoverability system in the WebAnnotation Protocol. Any pointers ? Because for indexes at least, that's what people are gonna want.

There has been some discussion on the w3 annotation list about this recently but I don't think there is any specific progress on it. See the thread at https://lists.w3.org/Archives/Public/public-openannotation/2020May/0003.html

It might be worth getting in touch with the folks working on Pundit to see if a collaboration is possible?

balmas · 2020-06-25T13:29:23Z

If we don't go the Web Annotation protocol route, then following the IIIF model does make a lot of sense. I get @PietroLiuzzo 's point about keeping the two connected.

PietroLiuzzo mentioned this issue Feb 25, 2020

DTS indexes example implementation BetaMasaheft/Documentation#1328

Closed

PonteIneptique mentioned this issue Oct 9, 2020

Table of Contents Data #112

Closed

PonteIneptique added the Future For after draft 1 label Feb 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ideas for an "indexes" endpoint #167

ideas for an "indexes" endpoint #167

PietroLiuzzo commented Oct 19, 2019

mromanello commented Feb 28, 2020 •

edited

Loading

PietroLiuzzo commented Jun 17, 2020

mromanello commented Jun 18, 2020 •

edited

Loading

PonteIneptique commented Jun 18, 2020

mromanello commented Jun 18, 2020

PonteIneptique commented Jun 18, 2020

PonteIneptique commented Jun 18, 2020

PietroLiuzzo commented Jun 18, 2020

PonteIneptique commented Jun 18, 2020

balmas commented Jun 18, 2020

PonteIneptique commented Jun 19, 2020 •

edited

Loading

PietroLiuzzo commented Jun 19, 2020

PonteIneptique commented Jun 19, 2020 via email

PietroLiuzzo commented Jun 19, 2020

PonteIneptique commented Jun 19, 2020 via email

PietroLiuzzo commented Jun 24, 2020 •

edited

Loading

PonteIneptique commented Jun 24, 2020

balmas commented Jun 24, 2020

PietroLiuzzo commented Jun 24, 2020

PonteIneptique commented Jun 25, 2020 •

edited

Loading

PonteIneptique commented Jun 25, 2020 •

edited

Loading

balmas commented Jun 25, 2020

balmas commented Jun 25, 2020

ideas for an "indexes" endpoint #167

ideas for an "indexes" endpoint #167

Comments

PietroLiuzzo commented Oct 19, 2019

mromanello commented Feb 28, 2020 • edited Loading

PietroLiuzzo commented Jun 17, 2020

mromanello commented Jun 18, 2020 • edited Loading

PonteIneptique commented Jun 18, 2020

mromanello commented Jun 18, 2020

PonteIneptique commented Jun 18, 2020

PonteIneptique commented Jun 18, 2020

PietroLiuzzo commented Jun 18, 2020

PonteIneptique commented Jun 18, 2020

balmas commented Jun 18, 2020

PonteIneptique commented Jun 19, 2020 • edited Loading

PietroLiuzzo commented Jun 19, 2020

PonteIneptique commented Jun 19, 2020 via email

PietroLiuzzo commented Jun 19, 2020

PonteIneptique commented Jun 19, 2020 via email

PietroLiuzzo commented Jun 24, 2020 • edited Loading

PonteIneptique commented Jun 24, 2020

balmas commented Jun 24, 2020

PietroLiuzzo commented Jun 24, 2020

PonteIneptique commented Jun 25, 2020 • edited Loading

PonteIneptique commented Jun 25, 2020 • edited Loading

balmas commented Jun 25, 2020

balmas commented Jun 25, 2020

mromanello commented Feb 28, 2020 •

edited

Loading

mromanello commented Jun 18, 2020 •

edited

Loading

PonteIneptique commented Jun 19, 2020 •

edited

Loading

PietroLiuzzo commented Jun 24, 2020 •

edited

Loading

PonteIneptique commented Jun 25, 2020 •

edited

Loading

PonteIneptique commented Jun 25, 2020 •

edited

Loading