Changelog

--------------------------------
Version 8.4.4 (27 February 2023)
--------------------------------
Fixed: Uninitialized member 'End' in 'rules' when defaulting flexrules to "do nothing".

-------------------------------
Version 8.4.0 (27 January 2023)
-------------------------------
It is now possible to map the same input text tag to multiple dictionary tags
in the tag translation file (option -X). E.g.
PRON_DEMO       PRON_DEMO
PRON_DEMO_S     PRON_DEMO
PRON_DEMO_P     PRON_DEMO
PRON_DEMO_GEN   PRON_DEMO
PRON_DEMO_GEN_S PRON_DEMO
PRON_DEMO_GEN_P PRON_DEMO

-------------------------------
Version 8.3.1 (25 January 2023)
-------------------------------
New predefined 'PROGPRINTDICT' in defines.h to enable serialization of binary
dictionary to text format.

-------------------------------
Version 8.3.0 (23 January 2023)
-------------------------------
CSTlemma assumes UTF-8 encoded input (option -eU).

-------------------------------
Version 8.2.0 (3 December 2021)
-------------------------------
When disambiguating, if multiple lemmas have the same top frequency, only the
first one survives. Previously, this remained undecided and the output was
ambiguous.

-------------------------------
Version 8.1.4 (6 October 2021)
-------------------------------
In text.h, changed type of optionStruct to 'struct'.

-------------------------------
Version 8.1.3 (27 August 2021)
-------------------------------
Added newline before "  $p pattern part of the rule ..."

-------------------------------
Version 8.1.3 (27 August 2021)
-------------------------------
Made sure that
  touch my_empty_rule_file
  ./cstlemma -L -f my_empty_rule_file -i my_text_file.txt
doesn't generate a core dump.

-------------------------------
Version 8.1.2 (18 May 2021)
-------------------------------
Fixed error in decoding X/HTML entities, caused by calling virtual fnction from
inside a constructor.

-------------------------------
Version 8.1.1 (17 April 2021)
-------------------------------
Fixed in applyrules.cpp: recompute string length after 
                         converting word to lowercase.
Example:                 Al-İstişrâḳ 16 bytes 
                         al-istişrâḳ 15 bytes
This bug would sometimes cause memory corruption.

-------------------------------
Version 8.1.0 (8 Mar 2021)
-------------------------------
(1) Fixed bug in dictionary lookup: A dictionary entry with no children (as a
    counterexample: 'hed' with children 'hedde' and 'hede') succesfully matched
    word starting with the dictionary entry, but with extra characters.
    For example, 'Joel' was lemmatized to 'Joe' if 'Joe' didn't have children
    in the dictionary.
(2) Added new output functions $p and $r that can be included in the -W option
    or, if there is no -W option, in the -B option.
    $p evaluates to the pattern part of the rule that constructs the lemma from
       the input word.
    $r evaluates to the replacement part of the rule that constructs the lemma
       from the input word.

-------------------------------
Version 8.0.5 (1 Sep 2020)
-------------------------------
Fixed bug in dictionary lookup for tagged words.

-------------------------------
Version 8.0.4 (13 May 2020)
-------------------------------
Fixed bug that disabled the use of rules if 
(1) the input is tagged and
(2) there is no flexrules file without a tag name extension, e.g.:

    flexrules       <--- this one missing
    flexrules.ADJ
    flexrules.ADV
    ...

(An extensionless flexrules file is used as a fall-back if a token has an
unknown tag.)

-------------------------------
Version 8.0.3 (14 January 2020)
-------------------------------
Option -l+ now works as intended.
Several improvements of C++ source, such as replacing 'typedef enum' with
'enum class' for better scoping.

-------------------------------
Version 8.0.2 (13 January 2020)
-------------------------------
New format option $r that prints the replacement. $p prints the pattern, and
not both pattern and replacement as in version 8.0.1.

-------------------------------
Version 8.0.1 (9 January 2020)
-------------------------------
New format option $p that prints the rule (pattern and replacement)
responsible for how the lemma is constructed from the input word.
Turned on with compiler option -DPRINTRULE. (Work in progress.)

New cstlemma option -l+ that tries to mimick the casing of the word
in the lemma. (This was already promised by -l-, but -l- did in reality
just nothing, whereas -l converted words to lowercase.) (Work in progress.)

-------------------------------
Version 7.38 (28 January 2019)
-------------------------------
Fixed a problem with dictionary look-up. The findword() function in
dictionary.cpp did not check the PoS tag of the word. A sentence initial
preposition 'For' would be found as a noun 'For'. Instead, find() must
see that 'For' does not exist in the dictionary as a preposition, and
attempt to find the lowercased version 'for' in the dictionary (which
does indeed exist). Previously, the error would be detected too late and
the flexrules would be used instead of the dictionary look up. However,
the flexrules would also treat 'For' as a noun.

-------------------------------
Version 7.37 (27 January 2019)
-------------------------------
Fixed a bug in word.cpp that sometimes caused the lemma to be prefixed by the
word. E.g lemma *osvi for the word 'os'. (If the correct lemma is 'vi'.)

-------------------------------
Version 7.36 (2 September 2018)
-------------------------------
Removed superfluous and, in case of option -t-, havoc wreaking test that a tag
must be != '\n'. It is sufficient to test that a word is != '\n'. (The strange
case that a word seems to be '\n' occurs at the end of the output if STREAM is
defined to be 1 in defines.h. This word is an artefact, caused by the '\n' that
is emitted when the stream reaches EOF. The '\n' is necessary to satisfy an
input format like -I '....\n'. See the function getwordI() in flattext.cpp.)

-------------------------------
Version 7.35 (30 August 2018)
-------------------------------
Removed spurious almost empty lines at the end of the output if STREAM == 1.
Also made sure that warnings and error messages are sent to stderr, not stdout.

-------------------------------
Version 7.33 (15 August 2018)
-------------------------------
When STREAM == 1 in defines.h, cstlemma is now able to read lines from standard
input with the -I parameter defined. This did not work previously. The solution
is twofold: (1) a new line character is sent to the current field interpreter
when the end of a line is reached. (Therefore the last field in the -I pattern
must be either \n or \s). (2) If the current field is still expecting input
after the input has come to the end of the line (i.e. if the last field is \s),
then a '\0' byte is sent to the current field. The current field is then able
to set the 'next current field' to null, satisfying the requirement that all
fields must have been matched when the input ends. (The reason why \s behaves
different from \n is that \s reads all white space characters until the first
non-white space character. The \n field only reads exactly one byte. If there
is no non-white space character after the last white space character, the \s
field has to be told that it is done.)

-------------------------------
Version 7.31 (8 December 2017)
-------------------------------
Changed UTF8 to globUTF8 (global variable defined in letterfunc/utf8func)

-------------------------------
Version 7.30 (28 November 2017)
-------------------------------
Improved explanation of -W and -c options. Set -I per default to $w\s if the
input is XML and tags are in an attribute.

-------------------------------
Version 7.29 (10 November 2017)
-------------------------------
Fixed bug: -U option had no effect if version 3 rules were used.

-------------------------------
Version 7.28 (4 October 2017)
-------------------------------
Fixed bug: version of flexrule file was not properly checked when using
PoS-tagged input. Ensure #define TESTING 1 can be used.

-------------------------------
Version 7.27 (14 September 2017)
-------------------------------
Explcitly deallocated spaces used for storing file name and tag name, and
corrected deallocation of string buffer by adding [].

-------------------------------
Version 7.26 (8 September 2017)
-------------------------------
Completely removed resolution of character and entity references from the phase
where XML input is parsed. Resolution now takes places just before words
(including tags, if relevant) are inserted in the hash table that later is
traversed to lemmatise each type in the input text. Before, reference 
resolution only took place for words, and not for tags.

-------------------------------
Version 7.25 (29 August 2017)
-------------------------------
Once more a better explanation of '+' when $i is used.
Replaced tabs by four spaces.
Changed the order of ambiguous lemma when option -H1 applies: The lemma
that would survive pruning (-H0 option) is output first. The << and >>
around pruning candidates are not shown anymore.

-------------------------------
Version 7.24 (23 August 2017)
-------------------------------
Better explanation of the value '+' when $i is used in -c or -W format
strings.

-------------------------------
Version 7.23 (7 August 2017)
-------------------------------
Delegated recognition of character references (&#nnn; or &#xhhh;) to
entities.cpp.

-------------------------------
Version 7.22 (3 August 2017)
-------------------------------
Delegated recognition of HTML entities to new file entities.cpp in the
letterfunc directory, for sharing code with other programs.
Fixed bug that made transformation of HTML entities a mess. 

-------------------------------
Version 7.21 (31 July 2017)
-------------------------------
Updated list of HTML entities. Fixed memory deallocation error in option.cpp.

-------------------------------
Version 7.20 (18 July 2017)
-------------------------------
Fixed recently (June 2017) introduced error that turned off lemmatisation by
rule application. 

-------------------------------
Version 7.19 (26 June 2017)
-------------------------------
Fixed bug that made use of long command lines unreliable.

-------------------------------
Version 7.18 (22 June 2017)
-------------------------------
Added tab character in screen display of -f option.
When lemmatising tagged input, program can now handle flexrule files of
different generations. 

-------------------------------
Version 7.17 (20 March 2017)
-------------------------------
If the same lemma is generated several times (by alternative decision trees),
then only the first instance is kept.

-------------------------------
Version 7.16 (15 March 2017)
-------------------------------
Fixed error in decoding -x option.
Made several explicit type casts.
Moved data members (esp. bit fields) around to minimize padding.

-------------------------------
Version 7.15 (19 October 2016)
-------------------------------
Fixed error in the decoding of XML-entities if the decoded string comprised
more than one byte.

-------------------------------
Version 7.14 (13 October 2016)
-------------------------------
Fixed error in dictionary look-up. Previously, if a word was found in the
dictionary, the lemma form was forced to lower case, even if option -l- was
set. Now lowercasing only happens if -l is set.

-------------------------------
Version 7.13 (19 July 2016)
-------------------------------
Debugged getwordI so it does not have incorrect complains about missing 
input according to the input specification format (-I option).

-------------------------------
Version 7.12 (18 July 2016)
-------------------------------
Typecast streamoff to size-t in XMLtext.cpp

-------------------------------
Version 7.11 (3 May 2016)
-------------------------------
Typecast char to unsigned char when calling isspace and isUpper.

-------------------------------
Version 7.10 (6 February 2016)
-------------------------------
Improved rule based lemmatisation of capitalised words that are at the
beginning of a line.
1) If the same capitalised word occurs elsewhere in the input text in a
non-sentence initial position, then the word is not converted to lowercase
before being lemmatised.
2) Otherwise, the lemma(s) is/are formed in two steps:
a) The word is lemmatised as it is, but any lemmas formed by rules
without a prefix part are discarded. So acceptable rules in this step must be
case sensitive.
b) Finally, the word is lowercased and lemmatised. The lemmas formed in the
second step are appended to those formed in the first step.

Tomake this work, if the input is flat text, each segment should be on a
separate line. A newline character indicates a segment separator. If the input
is an XML-formatted file, use elements like <p> ... </p> or <br /> to indicate
segment breaks. Declare the segment breaking element with the new Xs<element>
option, e.g., -Xsp

-------------------------------
Version 7.01 (24 August 2015)
-------------------------------
Cosmetic changes, allow empty lines in input when making dictionary.

-------------------------------
Version 7.0 (13 August 2015)
-------------------------------
Debugged lemmatisation of ambiguous words. (Previously, not all possible lemmas
were always returned.)
Changed default values for 't', 'u', 'U', 'H' and 'l' options. Without these
options on the command line, the following is assumed: -t- -u- -U- -H2 -l-
That means that it is assumed that the input has no tags, that the output
should not be disambiguated (all candidate lemma's should be produced) and
that the casing of the output should be similar to the input's casing.
(The idea is that you have to turn on options that require extra processing.)
The old algorithm for creating suffix based flexrules is deprecated. Users
who still want to use this algorithm must add an extra 'F': -FF
The prefered way to generate flexrules for cstlemma is the program
'affixtrain'. (https://github.com/kuhumcst/affixtrain)


-------------------------------
Version 6.03 (3 July 2015)
-------------------------------
Renamed 'function'. (This name is already taken.)
Replaced string hashing function in hashmap.cpp. (Please pull new version of https://github.com/kuhumcst/hashmap)
Appended newlines to cpp and h files where desirable.
Deleted unused member variable 'kar' in wordReader class.

-------------------------------
Version 6.02 (21 April 2015)
-------------------------------
Removed unused or unnecessary code. There was a 255 word size limit
that is gone now.

-------------------------------
Version 6.01 (17 February 2015)
-------------------------------
New rules format, V3, which is Recognised by inspecting the start of the file.
The new format allows a greater amount of ambiguity, for instance for  words
with ten different lemmas. Previously the maximum supported ambiguoity was two.

-------------------------------
Version 5.06 (9 August 2014)
-------------------------------
Added call to parseAsXml to parse procesor instructions correctly.

-------------------------------
Version 5.05 (28 February 2014)
-------------------------------
XML input with tagged text in content came out with POS-tags still there,
even if the output format did not specify that.

-------------------------------
Version 5.04 (24 February 2014)
-------------------------------
Introduced default -I format for XML input.

-------------------------------
Version 5.03 (24 February 2014)
-------------------------------
Null terminated strings read with fread.

-------------------------------
Version 5.02 (24 February 2014)
-------------------------------
The -X option, without further option arguments, was not working. Now it does.
If the -I option was used, the program could hang if the input didn't conform
to the -I format.

------------------------------
Version 5.01 (6 January 2014)
------------------------------
No functional changes. Deleted comments that were change notifications and
commented-out code. Made sure that setting PROGMAKEDICT, PROGMAKESUFFIXFLEX or
PROGLEMMATISE has the effect of neutralizing irrelevant code. Only if all three
are defined, all code is compiled, creating a Swiss Army Knive that can 
(1) Create a built-in dictionary
(2) Create an old-style flex rule list 
    (Use the program affixtrain for new style!)
(3) Lemmatise a text.

------------------------------
Version 5.0 (3 January 2014)
------------------------------
When lemmatizing (-L option):
Made #define STREAM 1 work as intended: handle standard input line-wise.
If #define STREAM 0, stdio.h functions are used. In that case, input
MUST come from a named file. (-i option).

stdio.h is not removed everywhere. E.g. makedict functionality (-D option)
still uses fprintf even if STREAM 1 is defined in defines.h.

------------------------------
Version 2.4 (3 March 2005)
------------------------------
Removed (undocumented) requirement that word classes must be capitalised.

------------------------------
Version 2.3 (24 February 2005)
------------------------------
Problem solved with -I option. (input format) 

------------------------------
Version 2.2 (23 February 2005)
------------------------------
Problems with generation of flex rules and dictionary solved.
Addition of $s field (word separator that expands to blank or new line) in -c
format.

------------------------------
Version 2.1 (27 January 2004)
------------------------------
General beautification of code.
Introduction of C++ streams instead of file pointers (optional).
Lemmata can contain uppercase characters.

----------------------------
Version 2.0 (5 January 2004)
----------------------------
Initial version under GPL

-------------------------
Version 1.0 (Autumn 2002)
-------------------------
Initial non-GPL version.