docs update

tf/about/annotateBrowser.html

@@ -51,11 +51,13 @@ <h2 id="start-up">Start up</h2>
 <p>Start a terminal or command prompt.
 You need not go to a specific directory.
 Give the command</p>
-<pre><code class="language-sh">tf HuygensING/suriano --tool=ner --chrome
+<pre><code class="language-sh">tf HuygensING/suriano --tool=ner
 </code></pre>
 <p>The tool opens in a new window in the Chrome browser.
-Currently the tool works best in Chrome, but it is not required.
-If you leave out the <code>--chrome</code> bit, the tool starts in your default browser.</p>
+Currently the tool works best in Chrome, but it is not required.</p>
+<pre><code class="language-sh">tf HuygensING/suriano --tool=ner --chrome
+</code></pre>
+<p>lets the tool starts in Chrome.</p>
 <p>You can restart the tool without navigating away from the web page as follows:</p>
 <pre><code class="language-sh">tf HuygensING/suriano --tool=ner --chrome -noweb
 </code></pre>
@@ -77,6 +79,9 @@ <h2 id="start-up">Start up</h2>
 <p>However, you can duplicate this set, and the copy can be modified.
 So, if the existing set is a good one, requiring only a few tweaks,
 this is a good way to start.</p>
+<p>Some of the sets may have a spreadsheet behind it, which specifies the triggers
+by which entities are looked up. If there are diagnostic messages coming
+from reading and parsing the spreadsheet, you will see them in this column.</p>
 </li>
 <li>
 <p><strong>Entities column</strong>:</p>
@@ -88,16 +93,20 @@ <h2 id="start-up">Start up</h2>
 <p>You can click on each entity. Let's click on <code>Francia</code>:</p>
 <p><img alt="browser" src="../images/Ner/browser-entity.png"></p>
 <p>On the right you see all occurrences of this entity, highlighted.</p>
+<p>If the set has a spreadsheet behind it, you not only see the entities, but also
+their triggers. You can click on a trigger to see its occurrences, and click
+again to unselect that trigger. You can click on an entity to see the occurrences
+of all of its triggers, and click again on the entity to unselect it.</p>
 </li>
 <li>
 <p><strong>Right column</strong>:</p>
 <p>See a list of buckets (i.e. paragraphs in this corpus).</p>
-<p>If you have clicked on an entity, you see the buckets that contain an occurrence
-of that entity.</p>
+<p>If you have clicked on an entity or trigger, you see the buckets that
+contain an occurrence of that entity or trigger.</p>
 <p>If you have not clicked on anything, you see the whole corpus, truncated to the
 first 100 buckets.</p>
-<p>We got here by clicking on an entity. If we click the ❌ next to the entity on top,
-we reset the viewer.</p>
+<p>If we click the ❌ next to the entity on top, we reset the viewer.
+If there is no ❌ then click on the selected entity or trigger to reset the viewer.</p>
 <p>From this point, we can do a succession of steps.</p>
 </li>
 </ul>
@@ -252,22 +261,34 @@ <h2 id="sophistication">Sophistication</h2>
 Whereas the ✅ button fills the <code>eid</code> field with a value derived from the occurrence,
 the ❎ leaves the previous value unchanged. So you can just click <code>add</code> to mark
 the new occurrences with the values of the same entity as before.</p>
-<h1 id="programming">Programming</h1>
+<h1 id="spreadsheets">Spreadsheets</h1>
 <p>The workflow sketched above works fine if you know what to look for, and if there are
 not too many distinct named entities in the corpus.</p>
 <p>Probably you soon discover that there are quite a bit of distinct named entities,
 and each entity has quite a bit of surface forms.
 Then the work load may not feel doable any more.</p>
-<p>For that purpose, there is also a way of annotating-by-programming, where you run the
-annotator tool in a Jupyter Notebook, driven by a spreadsheet with names and
-surface forms.</p>
+<p>For that purpose, there is also a way of annotating-by-spreadsheet, where you run the
+annotator tool, driven by a spreadsheet with names and surface forms.</p>
 <p>See <code><a title="tf.ner.ner" href="../ner/ner.html">tf.ner.ner</a></code>.</p>
 <p>You can then still use this tool after the fact to check the entity assignments
 and make fine-tuning corrections.</p>
+<p>In the tool, you see the annotation sets made by spreadsheets. However,you can not
+edit them, they are read only. What you can do, is to <em>duplicate</em> such a set.
+The duplicate is a proper, editable annotation set, and then you can select new
+occurrences and add them, or delete existing occurrences.</p>
+<p>The resulting set does not have any intrinsic relationship to the original spreadsheet.</p>
+<h2 id="ergonomics-of-annotation">Ergonomics of annotation</h2>
+<p>We try to reduce the work of manual annotations as much as possible.
+It is a balancing act between automating as much as possible, but not so much that
+you miss the fine points in your corpus.</p>
+<p>We need to gather experience in order to arrive at a truly usable tool.</p>
+<p>We are going to mark up the
+<a href="https://gitlab.huc.knaw.nl/suriano/letters">Suriano/letters</a> corpus in this
+way and hope to acquire a lot of experience in the process.</p>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/annotateBrowser.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/annotateBrowser.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/annotateBrowser.md
@@ -349,7 +370,10 @@ <h1>Index</h1>
 <li><a href="#sophistication">Sophistication</a></li>
 </ul>
 </li>
-<li><a href="#programming">Programming</a></li>
+<li><a href="#spreadsheets">Spreadsheets</a><ul>
+<li><a href="#ergonomics-of-annotation">Ergonomics of annotation</a></li>
+</ul>
+</li>
 </ul>
 </div>
 <ul id="index">

tf/about/apps.html

@@ -115,7 +115,7 @@ <h3 id="two-contexts">Two contexts</h3>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/apps.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/apps.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/apps.md

tf/about/background.html

@@ -155,7 +155,7 @@ <h2 id="history">History</h2>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/background.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/background.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/background.md

tf/about/browser.html

@@ -174,7 +174,7 @@ <h3 id="unicode-in-excel-csvs">UNICODE in Excel CSVs</h3>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/browser.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/browser.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/browser.md

tf/about/clientmanual.html

@@ -565,7 +565,7 @@ <h1 id="credits">Credits</h1>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/clientmanual.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/clientmanual.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/clientmanual.md

tf/about/code.html

@@ -32,10 +32,12 @@ <h1 id="code-organization">Code organization</h1>
 <p>The code base of TF can be divided into a few major parts,
 each with their own, identifiable task.</p>
 <p>Some parts of the code are covered by unit tests (<code><a title="tf.about.tests" href="tests.html">tf.about.tests</a></code>).</p>
+<p>There is also a
+<a href="https://annotation.github.io/text-fabric/tf/stats/index.html">count of code lines per module and language</a></p>
 <h2 id="base">Base</h2>
 <p>(<code><a title="tf.core" href="../core/index.html">tf.core</a></code>) The core API is responsible for:</p>
 <dl>
-<dt>Data management</dt>
+<dt>Feature management</dt>
 <dd>TF data consists of <em>feature files</em>.
 TF must be able to load them, save them, import / export from MQL.</dd>
 <dt>Provide an API</dt>
@@ -66,7 +68,7 @@ <h2 id="search">Search</h2>
 <h2 id="advanced">Advanced</h2>
 <p>(<code><a title="tf.advanced" href="../advanced/index.html">tf.advanced</a></code>) TF contains an advanced API geared to auto-downloading
 corpus data and displaying corpus materials in useful ways.</p>
-<h2 id="web-interface">Web interface</h2>
+<h2 id="dynamic-web-interface">Dynamic Web interface</h2>
 <p>(<code><a title="tf.browser" href="../browser/index.html">tf.browser</a></code>) TF contains a browser interface for interacting
 with your corpus without programming.</p>
 <p>The web interface lets you fire queries (search templates) to TF and interact
@@ -78,6 +80,10 @@ <h2 id="web-interface">Web interface</h2>
 </ul>
 <p>This interface be served by a local web server provided with data from a TF app.
 (<code><a title="tf.browser.start" href="../browser/start.html">tf.browser.start</a></code>, <code><a title="tf.browser.kernel" href="../browser/kernel.html">tf.browser.kernel</a></code> and <code><a title="tf.browser.web" href="../browser/web.html">tf.browser.web</a></code>).</p>
+<h2 id="static-web-interface">Static Web interface</h2>
+<p>(<code><a title="tf.client" href="../client/index.html">tf.client</a></code>) There is also a static browser interface: you can build a set of
+static HTML pages with Javascript files out of a corpus, which has a search interface
+of a different kind than <code><a title="tf.search" href="../search/index.html">tf.search</a></code>.</p>
 <h2 id="volumes-and-collections">Volumes and collections</h2>
 <p>(<code><a title="tf.volumes" href="../volumes/index.html">tf.volumes</a></code>) Machinery to support the idea that a TF dataset
 is a work that consists of volumes. Volumes and collections of volumes
@@ -95,10 +101,14 @@ <h2 id="convert">Convert</h2>
 <h2 id="writing">Writing</h2>
 <p>(<code><a title="tf.writing" href="../writing/index.html">tf.writing</a></code>) TF supports several writing systems by means of transliterations
 and conversions between them and UNICODE.</p>
+<h2 id="ner">NER</h2>
+<p>(<code><a title="tf.ner" href="../ner/index.html">tf.ner</a></code> and <code><a title="tf.browser.ner" href="../browser/ner/index.html">tf.browser.ner</a></code>) This is machinery and an interface to do Named
+Entity Recognition, based on patterns you specify. These patterns can be supplied ad
+hoc in a web interface, or systematically in a spreadsheet.</p>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/code.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/code.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/code.md
@@ -173,11 +183,13 @@ <h1>Index</h1>
 <li><a href="#base">Base</a></li>
 <li><a href="#search">Search</a></li>
 <li><a href="#advanced">Advanced</a></li>
-<li><a href="#web-interface">Web interface</a></li>
+<li><a href="#dynamic-web-interface">Dynamic Web interface</a></li>
+<li><a href="#static-web-interface">Static Web interface</a></li>
 <li><a href="#volumes-and-collections">Volumes and collections</a></li>
 <li><a href="#dataset">Dataset</a></li>
 <li><a href="#convert">Convert</a></li>
 <li><a href="#writing">Writing</a></li>
+<li><a href="#ner">NER</a></li>
 </ul>
 </li>
 </ul>

tf/about/corpora.html

@@ -353,7 +353,7 @@ <h3 id="extra-data">Extra data</h3>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/corpora.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/corpora.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/corpora.md

tf/about/datamodel.html

@@ -265,7 +265,7 @@ <h2 id="serializing-and-pre-computing">Serializing and pre-computing</h2>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/datamodel.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/datamodel.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/datamodel.md

tf/about/datasharing.html

@@ -362,7 +362,7 @@ <h3 id="more-modules-at-the-same-time">More modules at the same time</h3>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/datasharing.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/datasharing.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/datasharing.md

tf/about/displaydesign.html

@@ -151,7 +151,7 @@ <h3 id="output">Output</h3>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/displaydesign.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/displaydesign.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/displaydesign.md

tf/about/faq.html

@@ -161,7 +161,7 @@ <h3 id="github-rate-limit-exceeded">GitHub Rate Limit Exceeded!</h3>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/faq.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/faq.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/faq.md

tf/about/fileformats.html

@@ -158,7 +158,7 @@ <h3 id="single-values">Single values</h3>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/fileformats.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/fileformats.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/fileformats.md

tf/about/index.html

@@ -33,7 +33,7 @@ <h1 id="documents">Documents</h1>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/__init__.py#L1-L5" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/__init__.py#L1-L5" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 # Documents

tf/about/install.html

@@ -108,7 +108,7 @@ <h4 id="note-for-linux-users">Note for Linux users</h4>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/install.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/install.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/install.md

tf/about/manual.html

@@ -390,7 +390,7 @@ <h2 id="keyboard-shortcuts">Keyboard shortcuts</h2>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/manual.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/manual.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/manual.md

tf/about/optimizations.html

@@ -187,7 +187,7 @@ <h3 id="edge-features">Edge features</h3>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/optimizations.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/optimizations.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/optimizations.md

tf/about/releases.html

@@ -42,6 +42,25 @@ <h1 id="release-notes">Release notes</h1>
 <pre><code class="language-sh">pip install -e .
 </code></pre>
 <h2 id="12">12</h2>
+<h3 id="126">12.6</h3>
+<h4 id="1260">12.6.0</h4>
+<p>2024-10-08</p>
+<p>Something has changed in the representation of the corpus data: we store something
+extra about sections in the precomputed data: a mapping from section nodes to their
+<em>legal</em> numbering, i.e. each section is assigned a tuple of integers with sequence
+numbers for each level of containing sections and itself. The new data is accessible
+as follows:</p>
+<pre><code class="language-python">C.sections.data[&quot;seqFromSec&quot;]
+C.sections.data[&quot;secFromSeq&quot;]
+</code></pre>
+<p>See also <code><a title="tf.core.prepare.sections" href="../core/prepare.html#tf.core.prepare.sections">sections()</a></code></p>
+<p>This means that older precomputed data can no longer used with this version of
+Text-Fabric. When you run this new version, it will automatically precompute your
+corpus data again upon first usage, without removing the older precomputed data, so you
+can still switch back and forth between TF versions.
+See <code><a title="tf.parameters.PACK_VERSION" href="../parameters.html#tf.parameters.PACK_VERSION">PACK_VERSION</a></code>, which has been bumped from <code>3</code> to <code>4</code> in this version.</p>
+<p>The reason is that the <code><a title="tf.ner" href="../ner/index.html">tf.ner</a></code> facility needs to handle sections in a generic way,
+it needs to work with intervals of sections and intersections of those.</p>
 <h3 id="125">12.5</h3>
 <h4 id="1255">12.5.5</h4>
 <p>2024-10-03</p>
@@ -550,7 +569,7 @@ <h2 id="older-releases">Older releases</h2>
 <details class="source">
 <summary>
 <span>Expand source code</span>
-<a href="https://github.com/annotation/text-fabric/blob/54f8fc70db27df28d334280982b28290fa009c46/tf/about/releases.py#L1-L3" class="git-link">Browse git</a>
+<a href="https://github.com/annotation/text-fabric/blob/94858b4d83d560e89985ab83802ba1c9f2d3d1f6/tf/about/releases.py#L1-L3" class="git-link">Browse git</a>
 </summary>
 <pre><code class="python">&#34;&#34;&#34;
 .. include:: ../docs/about/releases.md
@@ -623,6 +642,10 @@ <h1>Index</h1>
 <ul>
 <li><a href="#release-notes">Release notes</a><ul>
 <li><a href="#12">12</a><ul>
+<li><a href="#126">12.6</a><ul>
+<li><a href="#1260">12.6.0</a></li>
+</ul>
+</li>
 <li><a href="#125">12.5</a><ul>
 <li><a href="#1255">12.5.5</a></li>
 <li><a href="#1254">12.5.4</a></li>