RELEASENOTES

RELEASE NOTES SCORPIO 4.1.15

addresses an issue in the registry subscription manager as stand alone microservice not logging onto the DB properly

RELEASE NOTES SCORPIO 4.1.14

enables by default repairing of migration for all microservices

RELEASE NOTES SCORPIO 4.1.13

The release contains various fixes.

Addresses configuration issues with the distributed setup. 
Make system generated temporal props (createdAt, modifiedAt) to include nanoseconds 


RELEASE NOTES SCORPIO 4.1.12

The release contains various fixes  

RELEASE NOTES SCORPIO 4.1.11

The release contains various fixes  

RELEASE NOTES SCORPIO 4.1.10

The release contains various fixes  

RELEASE NOTES SCORPIO 4.1.9

The release contains various fixes  

RELEASE NOTES SCORPIO 4.1.8

The release addresses an issue which in some cases prevented auto history recording.


RELEASE NOTES SCORPIO 4.1.7

various bugfixes 

RELEASE NOTES SCORPIO 4.1.6

Addressing serialization issues with subscriptions.
Fixing message splitting in different message buses.
 - to give some more information here... each message bus has their own max message size, which sometimes can't be changed. 
   Scorpio will split up messages to fit the size. We are using the default sizes for each message bus. If you want to override this
   set the env var SCORPIO.MESSAGING.MAXSIZE to the byte number your message bus has been to configured to


RELEASE NOTES SCORPIO 4.1.5

With this release we are adding support for various message buses using the Apache Camel extension for the messaging between our 
microservices. For now we have added support for MQTT, Rabbit MQ and AWS SQS.
Since we leveraging Apache Camel you can supply various configuration options via parameters in the connection URI.
You can do this by setting the environment variable MYSETTINGS_MESSAGECONNECTION_OPTIONS. This is the full parameter part of a URI
so you got to start with a leading question mark, e.g. for mqtt MYSETTINGS_MESSAGECONNECTION_OPTIONS=?clientId=MySpecialID&qos=1
Please visit the respective connector websites from Camel for all options
https://camel.apache.org/components/3.20.x/paho-mqtt5-component.html
https://camel.apache.org/components/3.20.x/rabbitmq-component.html
https://camel.apache.org/components/3.20.x/aws2-sqs-component.html  

You can still use the BUSHOST and BUSPORT environment variables to configure the host address. 
You can find new compose examples in the compose-files folder.

Support for Vocab properties added

Various bugfixes

RELEASE NOTES SCORPIO 4.1.4

This release includes various fixes and the ability to directly inline entities from relationships in queries 
please check the doc here for examples 
https://github.com/ScorpioBroker/ScorpioBroker/blob/development-quarkus/docs/en/source/followRelationship.rst

RELEASE NOTES SCORPIO 4.1.3

This release add various bug fixes and a performance improvement on temporal queries. 
These improvements require some restructuring of the temporal database. 
If you have a big temporal database the migration can require a several hours to complete. 
ATTENTION!!!
DO NOT STOP OR RESTART SCORPIO WHILE THE MIGRATION IS RUNNING!
Don't be worried if you see no CPU activity on Scorpio during this phase. The database does most of the work.
If you stop Scorpio during this phase there is a chance that the executing query is still running in the database.
This can pile up and make your DB slow. 

As an alternative you can also run Flyway as standalone container to run the migrations.
This is also a good way to get something repaired.

You can add the following to your existing docker-compose file
flyway:
    image: flyway/flyway
    command: -url=jdbc:postgresql://postgres:5432/ngb -user=ngb -password=ngb -connectRetries=60 -baselineOnMigrate=true migrate -X
    volumes:
      - "./scorpiobroker/AllInOneRunner/src/main/resources/db/migration:/flyway/sql"
    depends_on:
      - postgres
    restart: on-failure

it needs the migration files mounted on /flyway/sql
this assumes you have cloned the git repo into the same folder as your compose file. change the volume mount if needed.
after run
docker-compose -f <your compose file> stop scorpio
to stop scorpio
docker-compose -f <your compose file> start flyway
to start the migration
after finish start scorpio again 
docker-compose -f <your compose file> start scorpio

I would recommend commenting or removing the flyway entry afterwards.   


RELEASE NOTES SCORPIO 4.1.1

This release 

 - adds support for the NGSI-LD replace functionality 
 - fixes an issue with the lastFailure and lastSuccess fields in subscriptions
 
RELEASE NOTES SCORPIO 4.1.0

With this release we are reintroducing native releases for Scorpio.
Scorpio will be release as an 
 - all in one variant, with and without kafka usage (without kafka can't scale but requires less resources) 
 - seperated micro services (individual scaling possible)
with following build variants
 - Java:
   - Release on github
   - Docker images provided on dockerhub and AWS Public ECR
 - Ubuntu x64:
   - Release on github
   - Docker images provided on dockerhub and AWS Public ECR
 - Windows x64:
   - Release on github
 - MacOS x64:
   - Release on github

This release also:

 - adds legacy support for older IoT Agents which are not using the CSourceRegistry operations properly

RELEASE NOTES SCORPIO 4.0.9

In this release 
 - the /q/info endpoint is enabled providing build info
 - /q/health is available since version 3.0.0

RELEASE NOTES SCORPIO 4.0.8

This release addresses a few issues with remote queries and registry loading from the database

RELEASE NOTES SCORPIO 4.0.7

This release 
 - increases the default timeout for transactions
 - addresses an issue in a db migration related to ownership change 

RELEASE NOTES SCORPIO 4.0.6

This release 
 - adds support for merge-patch behaviour
 - adds support previous value in subscriptions
 - removes the last usage of blocking i/o in scorpio making it fully reactive 
 - sets a default idle timeout for database connections. default is 15 seconds and can be changed with the environment variable QUARKUS_DATASOURCE_REACTIVE_IDLE_TIMEOUT to another duration (e.g. 60s)
 - adjusting the default connection pool per tenant to 50 connections. This can be changed with environment variable QUARKUS_DATASOURCE_REACTIVE_MAX_SIZE. The default max connections in postgres is 100. 
   This should be considered when setting up your system. 
 - self hosting of the core context 
	
RELEASE NOTES SCORPIO 4.0.5/FIWARE_8.4

This release is also a re tagging for the FIWARE release to point it to the most up to date version on the FIWARE release.
This release addresses and issue with the regex evaluation in q queries


RELEASE NOTES SCORPIO 4.0.4

This release addresses

 - an issue with attrs queries 
 - an issue with receiverInfo, notifierInfo, notUpdated and contextSourceInfo not being properly expanded and being detected as invalid key   

RELEASE NOTES SCORPIO 4.0.3

This release addresses the following issues

 - A backward compatibility issue with q queries which use the id field. Generally we recommend to use the idPattern field for this
 - An issue with batch upsert where it would sometimes overwrite other entities
 - Proper support for authorization in mqtt notifications

RELEASE NOTES SCORPIO 4.0.2

This release 
 - improves further on the response time of Entity create update append and delete operations under heavy load
 - addresses an issue with the count operator in queries.

RELEASE NOTES SCORPIO 4.0.1

This addresses 
 - an issue with the response time of Entity create update append and delete operations
 - improves the performance of queries which create an entity map

RELEASE NOTES SCORPIO 4.0.0

FRAMEWORK CHANGE

As announced with Relaease 3.0.0, we have moved frameworks, from Spring Boot to Quarkus. After maintaining both versions for a while, from Release 4.0 onwards, we only support the Quarkus framework
and not Spring Boot anymore. 

With this release we are moving to NGSI-LD version 1.6.1. This means there are a few changes in the core context.
As a general warning. There might be query issues with previous created entities with scopes and scope queries. We will soon provide a script to update your current entities up to spec as well as a script/tutorial how to backup your database for Scorpio.

The follow features have been implemented in this release.

 - Multi type support in entities
 - Full federated ops support. You can now create registrations with operations they shall support, e.g. createEntity or retrieveEntity (see spec for full list). All operations are supported. This means you can now create or update entities through a federation broker in a remote broker.
 - Lang property support. Full support for "languageMap" type attributes. Support in q queries updated.
     - In Query and temporal Query 
 - TypeQuery support. Support for new type query language. 
     - In Query and temporal Query
 - Aggr function support in temporal queries (e.g. min, max, avg)
 - A new approach to pagination in queries
   - NGSI-LD will define a new logic for queries in distributed systems. This release already includes the first step for this. 
   The issue with distributed systems is that real time pagination on a full dynamic system is very expensive. 
   Therefore NGSI-LD will introduce two new types of queries. Later a full snapshot logic will be introduced where a snapshot is recorded for a query through which a user can paginate.
   The change to the live query which is implemented in this release already works as follows
    - The federation broker will first collect all entity ids which match a query and create a temporary entityMap. 
    - The id of this map returned on a first query. It is also encoded in the next and prev links.
    - This entityMap is static so entityIds will not change order and a clear pagination is possible.
    - If the entityMap id is provided in the follow up query via the entityMap url parameter the entityMap entries will be retrieve
    - If the entityMap parameter is omitted the query will always retrieve the latest matching entities. This might causes inconsistencies between pages.
 - Support for deletedAt for attributes and entities in temporal recording.
 - at Context hosting/caching. A new endpoint /ngsi-ld/v1/jsonldContexts/ is provided to which users can store at contexts.
 	- Cache support in all operations. Custom at contexts provided in user requests are automatically cached 

  

RELEASE NOTES SCORPIO 3.0.11

Fixed various issues 

Added a default limit for lastN (20) for temporal queries to avoid temporal queries consuming to much memory. 
Will be addressed further with future spec updates. 

There is currently still an open issue with the native versions and tenants. So this is a java version only release.


RELEASE NOTES SCORPIO 3.0.6

In this release we are adressing 

General:

 - changes to the batch handling for notifications. previously a racing condition could have happened.
 - fixed simplified representation on temporal queries
 - an issue with MQTT notifications

RELEASE NOTES SCORPIO 3.0.5

In this release we are adressing

 - fixing of the interval subscription
 - an issue with the auto history recording of patched entities
 - addressing a potential out of resource situation in the subscription handler 
 
Changes in batch handling and subscriptions:
When using entityOperations to create/update/delete entities, they will be bundled into one notification instead of many. If this behaviour is unwanted it can be deactived via setting config/env variable SCORPIO_SUBSCRIPTION_BATCHNOTIFICATIONS to false.

RELEASE NOTES SCORPIO 3.0.4

In this release we are adressing 
 - an issue where kafka messages were not acknowledged in the quarkus branch.
 - adding support for SSL on the kafka connections 
 
RELEASE NOTES SCORPIO 3.0.3

This a minor bugfix release addressing 

 - an issue where the DBUSER and DBPASS settings in the quarkus release where not respected properly
 
RELEASE NOTES SCORPIO 3.0.2

This a minor bugfix release addressing 

 - an issue where the result of a federated query was not merged correctly
 - an issue with the return codes and body of entity operations

RELEASE NOTES SCORPIO 3.0.1 / FIWARE_8.3

This is a minor update for the FIWARE release. 

In this release we adress 
 - an issue where tenants were not properly forwarded in a federation setup. 
 - an issue where scope queries did not contain all results in certain cases
 - in docker images for the Quarkus release, setting the default names for postgres and kafka 
 - in the Spring release addressing an issue where tenant connection pools did not follow the instructed size
 - in the Spring release added support for custom pool sizes for tenants using SCORPIO_TENANTS_DEFAULT_POOLMAX or SCORPIO_TENANTS_TENANTNAME_POOLMAX.
	if SCORPIO_TENANTS_DEFAULT_POOLMAX is set to -1 the default poolsize will be used

RELEASE NOTES SCORPIO 3.0

With the release of Scorpio 3.0 we are happy to announce that quite a few changes. 

FRAMEWORK CHANGE

We have moved frameworks. With this release we are starting to provide a new Scorpio version which is build using the Quarkus framework and not Spring Boot anymore. 
For the time being we will maintain both versions of Scorpio and release them in parallel. However we will eventually fully move development to the Quarkus version.


The move to Quarkus essentially provides less memory consumption and a faster startup time. On openJDK we currently have roughly reduced the startup time and the memory consumption to 50%. 
Moving to Quarkus we have also adapted a reactive approach in the development which allows for a considerable higher number of connections towards the broker which can be handled in parallel.

With the move to Quarkus we are able to provide native executables for Windows, Ubuntu and MacOS. The native executables start in less than a second and reduces the initial memory consumption to less than 100MB.

AVAILABLE VERSIONS

As you might have seen already there are quite a few variants of Scorpio now released. You can either download the Individual Microservices (e.g. EntityManager, QueryManager, etc.) or the All-In-One Variant which starts everything in one VM. Scorpio is provided either as a Java version or natively compiled for Windows(x64), Ubuntu(x64) and MacOS(x64). 
We provide a version with a Eureka integration however since Eureka integration is much better on the Spring side, we strongly recommend for anyone using a Eureka based setup to stay with the Spring version for the time being.
Our All In One variant is now also availble in a version without needing Kafka. This good news for everyone with limit hardware and if you only want to run a single instance of Scorpio. However mind you that scaling of Scorpio instances REQUIRES a Kafka variant.

DOCKER VERSIONS

We will provide docker images on Dockerhub and on the AWS ECR. All Java versions from Quarkus and Spring will be provided and the Ubuntu native version. Going forward from this release we have changed our naming conventions and will provide versions with following naming scheme: scorpiobroker/[<microservices-name>|all-in-one-runner]:<os>-<variant>-<release-number/latest>. Spring versions will have a prefix "Spring" in the tag.