From e5a03687790202a2e27dd412fcb4bfbd0a6560e4 Mon Sep 17 00:00:00 2001 From: henrycotton1 <141598786+henrycotton1@users.noreply.github.com> Date: Fri, 27 Oct 2023 23:03:16 +1100 Subject: [PATCH 01/11] Added my description of the overall architecture --- docs/concepts.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/concepts.md b/docs/concepts.md index 2a976d5..0fd304a 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -15,6 +15,11 @@ SirixDB can help answer questions such as the following: Give me last month's hi Let's turn our focus toward the question of why historical data has not been retained in the past. We postulate that new storage advances in recent years present possibilities to build sophisticated solutions to help answer those questions without the hurdle state-of-the-art systems bring. +## A brief overview of the overall architecture. +Before we go into the specifics of SirixDB, its important to have an understanding of the overall architecture. At the core of SirxDB are databases. These instances store resources, which are generally binary tree encodings of either JSON or XML files - (more on that under the Tree-structure subheading). + +From this database instance, you may either create a new resource or begin a resource session to start as many read-only transactions as you would like or just a single read-write transaction. In essence, you may think of the architecture as one big tree full of prefix trees, where revisions to the tree are always appended. The data of the prefix trees either store the nodes of the JSON or XML trees or they store secondary indexes! + ## Advantages and disadvantages of flash drives, for instance, SSDs As Marc Kramis points out in his paper "Growing Persistent Trees into the 21st Century": From 3f2068df84db8bc3b033d94a143cf102968111c1 Mon Sep 17 00:00:00 2001 From: henrycotton1 <141598786+henrycotton1@users.noreply.github.com> Date: Sat, 28 Oct 2023 13:53:19 +1100 Subject: [PATCH 02/11] Changed "prefix trees" to "tries" --- docs/concepts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts.md b/docs/concepts.md index 0fd304a..cebe7fc 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -18,7 +18,7 @@ Let's turn our focus toward the question of why historical data has not been ret ## A brief overview of the overall architecture. Before we go into the specifics of SirixDB, its important to have an understanding of the overall architecture. At the core of SirxDB are databases. These instances store resources, which are generally binary tree encodings of either JSON or XML files - (more on that under the Tree-structure subheading). -From this database instance, you may either create a new resource or begin a resource session to start as many read-only transactions as you would like or just a single read-write transaction. In essence, you may think of the architecture as one big tree full of prefix trees, where revisions to the tree are always appended. The data of the prefix trees either store the nodes of the JSON or XML trees or they store secondary indexes! +From this database instance, you may either create a new resource or begin a resource session to start as many read-only transactions as you would like or just a single read-write transaction. In essence, you may think of the architecture as one big tree full of tries (the main document index), where revisions to the tree are always appended. The data of the tries either store the nodes of the JSON or XML trees or they store secondary indexes! ## Advantages and disadvantages of flash drives, for instance, SSDs As Marc Kramis points out in his paper "Growing Persistent Trees into the 21st Century": From 8e3b58ad98e292a6a22f947dde0e9e69536e5420 Mon Sep 17 00:00:00 2001 From: henrycotton1 <141598786+henrycotton1@users.noreply.github.com> Date: Sat, 28 Oct 2023 17:50:48 +1100 Subject: [PATCH 03/11] Tutorials and examples documentation --- docs/concepts.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/concepts.md b/docs/concepts.md index cebe7fc..2894e75 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -15,7 +15,7 @@ SirixDB can help answer questions such as the following: Give me last month's hi Let's turn our focus toward the question of why historical data has not been retained in the past. We postulate that new storage advances in recent years present possibilities to build sophisticated solutions to help answer those questions without the hurdle state-of-the-art systems bring. -## A brief overview of the overall architecture. +## A brief overview of the overall architecture Before we go into the specifics of SirixDB, its important to have an understanding of the overall architecture. At the core of SirxDB are databases. These instances store resources, which are generally binary tree encodings of either JSON or XML files - (more on that under the Tree-structure subheading). From this database instance, you may either create a new resource or begin a resource session to start as many read-only transactions as you would like or just a single read-write transaction. In essence, you may think of the architecture as one big tree full of tries (the main document index), where revisions to the tree are always appended. The data of the tries either store the nodes of the JSON or XML trees or they store secondary indexes! @@ -134,3 +134,6 @@ Write peaks occur during incremental versioning due to the requirement of interm Marc Kramis developed a novel sliding snapshot algorithm, which balances read/write performance to circumvent any write-peaks. The algorithm makes use of a sliding window. First, any changed record must be written during a commit. Second, any record older than a predefined length N of the window that has not been changed during these N-revisions must be written, too. Only these N-revisions at max have to be read. Fetching of the page fragments can be done in parallel or linear. In the latter case, the page fragments are read starting with the most recent revision. The algorithm stops once the full page has been reconstructed. You can find the best high-level overview of the algorithm in Marc's Thesis: [Evolutionary Tree-Structured Storage: Concepts, Interfaces, and Applications](http://kops.uni-konstanz.de/handle/123456789/27695) + +## Examples and Tutorials +We know this can be alot to take in so there are some handmade tutorials and examples for you to see all of this theory in action! Within the bundles directory of this repository you will find sirix-examples! This contains both tutorials and examples of how you may directly interact with the code base. From ea3e730a9a3a67fe3950f35a34401ab0acfa516d Mon Sep 17 00:00:00 2001 From: henrycotton1 <141598786+henrycotton1@users.noreply.github.com> Date: Sun, 29 Oct 2023 16:40:34 +1100 Subject: [PATCH 04/11] added a my "How To" for creating a new JsonDataBase. --- docs/concepts.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/concepts.md b/docs/concepts.md index 2894e75..dc58b1a 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -136,4 +136,15 @@ Marc Kramis developed a novel sliding snapshot algorithm, which balances read/wr The algorithm makes use of a sliding window. First, any changed record must be written during a commit. Second, any record older than a predefined length N of the window that has not been changed during these N-revisions must be written, too. Only these N-revisions at max have to be read. Fetching of the page fragments can be done in parallel or linear. In the latter case, the page fragments are read starting with the most recent revision. The algorithm stops once the full page has been reconstructed. You can find the best high-level overview of the algorithm in Marc's Thesis: [Evolutionary Tree-Structured Storage: Concepts, Interfaces, and Applications](http://kops.uni-konstanz.de/handle/123456789/27695) ## Examples and Tutorials -We know this can be alot to take in so there are some handmade tutorials and examples for you to see all of this theory in action! Within the bundles directory of this repository you will find sirix-examples! This contains both tutorials and examples of how you may directly interact with the code base. +We know this can be alot to take in so there are some handmade tutorials and examples for you to see all of this theory in action! Within the bundles directory of the sirix repository you will find sirix-examples! This contains both tutorials and examples of how you may directly interact with the code base. You may use these in conjunction with the following how to's in order to get started with sirix! + +## How to create your first Json database +The following may prove useful in helping you to understand CreateJsonDatabase.java within the tutorials directory of sirix. +1 - Create a Path variable that points to the directory containing the Json files you would like to store. +2 - Create a new variable storing the specific Json file you would like to store using the path variable from the previous step. +3 - Create a new variable to store the database file using "Constants.SIRIX_DATA_LOCATION.resolve("nameOfDataBase");" +4 - Check if the database already exists and if so, delete it. +5 - Create a new variable to store a new database configuration and use this variable to create a new Json database instance. +6 - Try to create a new variable, storing the database +7 - If able to create this variable, create a new Resource to store in the database +8 - Try to begin a new Resource Session and if able, Insert the specified Json file into the subtree. From 09c2ddd31dae2d204e52ecb361e41eb878dc6505 Mon Sep 17 00:00:00 2001 From: henrycotton1 <141598786+henrycotton1@users.noreply.github.com> Date: Sun, 29 Oct 2023 16:41:17 +1100 Subject: [PATCH 05/11] Update concepts.md --- docs/concepts.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/concepts.md b/docs/concepts.md index dc58b1a..8d6c115 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -140,8 +140,8 @@ We know this can be alot to take in so there are some handmade tutorials and exa ## How to create your first Json database The following may prove useful in helping you to understand CreateJsonDatabase.java within the tutorials directory of sirix. -1 - Create a Path variable that points to the directory containing the Json files you would like to store. -2 - Create a new variable storing the specific Json file you would like to store using the path variable from the previous step. +* 1 - Create a Path variable that points to the directory containing the Json files you would like to store. +* 2 - Create a new variable storing the specific Json file you would like to store using the path variable from the previous step. 3 - Create a new variable to store the database file using "Constants.SIRIX_DATA_LOCATION.resolve("nameOfDataBase");" 4 - Check if the database already exists and if so, delete it. 5 - Create a new variable to store a new database configuration and use this variable to create a new Json database instance. From 986df1b033e2526f834c56d795530230e2b76ff2 Mon Sep 17 00:00:00 2001 From: henrycotton1 <141598786+henrycotton1@users.noreply.github.com> Date: Sun, 29 Oct 2023 16:41:52 +1100 Subject: [PATCH 06/11] Fixed formatting for my How To. --- docs/concepts.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/concepts.md b/docs/concepts.md index 8d6c115..7025e5b 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -142,9 +142,9 @@ We know this can be alot to take in so there are some handmade tutorials and exa The following may prove useful in helping you to understand CreateJsonDatabase.java within the tutorials directory of sirix. * 1 - Create a Path variable that points to the directory containing the Json files you would like to store. * 2 - Create a new variable storing the specific Json file you would like to store using the path variable from the previous step. -3 - Create a new variable to store the database file using "Constants.SIRIX_DATA_LOCATION.resolve("nameOfDataBase");" -4 - Check if the database already exists and if so, delete it. -5 - Create a new variable to store a new database configuration and use this variable to create a new Json database instance. -6 - Try to create a new variable, storing the database -7 - If able to create this variable, create a new Resource to store in the database -8 - Try to begin a new Resource Session and if able, Insert the specified Json file into the subtree. +* 3 - Create a new variable to store the database file using "Constants.SIRIX_DATA_LOCATION.resolve("nameOfDataBase");" +* 4 - Check if the database already exists and if so, delete it. +* 5 - Create a new variable to store a new database configuration and use this variable to create a new Json database instance. +* 6 - Try to create a new variable, storing the database +* 7 - If able to create this variable, create a new Resource to store in the database +* 8 - Try to begin a new Resource Session and if able, Insert the specified Json file into the subtree. From dbf29d094491068092448b0b16e4ee36e636d9a7 Mon Sep 17 00:00:00 2001 From: henrycotton1 <141598786+henrycotton1@users.noreply.github.com> Date: Sun, 29 Oct 2023 16:46:45 +1100 Subject: [PATCH 07/11] Made my How To Generic to include XML databases --- docs/concepts.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/concepts.md b/docs/concepts.md index 7025e5b..904e55c 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -138,13 +138,13 @@ The algorithm makes use of a sliding window. First, any changed record must be w ## Examples and Tutorials We know this can be alot to take in so there are some handmade tutorials and examples for you to see all of this theory in action! Within the bundles directory of the sirix repository you will find sirix-examples! This contains both tutorials and examples of how you may directly interact with the code base. You may use these in conjunction with the following how to's in order to get started with sirix! -## How to create your first Json database -The following may prove useful in helping you to understand CreateJsonDatabase.java within the tutorials directory of sirix. -* 1 - Create a Path variable that points to the directory containing the Json files you would like to store. -* 2 - Create a new variable storing the specific Json file you would like to store using the path variable from the previous step. +## How to create your first database +The following may prove useful in helping you to understand CreateJsonDatabase.java or CreateXmlDatabase.java within the tutorials directory of sirix. +* 1 - Create a Path variable that points to the directory containing the files you would like to store. +* 2 - Create a new variable storing the specific file you would like to store using the path variable from the previous step. * 3 - Create a new variable to store the database file using "Constants.SIRIX_DATA_LOCATION.resolve("nameOfDataBase");" * 4 - Check if the database already exists and if so, delete it. -* 5 - Create a new variable to store a new database configuration and use this variable to create a new Json database instance. +* 5 - Create a new variable to store a new database configuration and use this variable to create a new Json/Xml database instance. * 6 - Try to create a new variable, storing the database * 7 - If able to create this variable, create a new Resource to store in the database -* 8 - Try to begin a new Resource Session and if able, Insert the specified Json file into the subtree. +* 8 - Try to begin a new Resource Session and if able, Insert the specified file into the subtree. From f238783ff43809ed081b019a284863a1c3acecf1 Mon Sep 17 00:00:00 2001 From: henrycotton1 <141598786+henrycotton1@users.noreply.github.com> Date: Sun, 29 Oct 2023 16:58:10 +1100 Subject: [PATCH 08/11] Added my "How To" for creating a new versioned resource --- docs/concepts.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/concepts.md b/docs/concepts.md index 904e55c..5ebe0fd 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -148,3 +148,16 @@ The following may prove useful in helping you to understand CreateJsonDatabase.j * 6 - Try to create a new variable, storing the database * 7 - If able to create this variable, create a new Resource to store in the database * 8 - Try to begin a new Resource Session and if able, Insert the specified file into the subtree. + +## How to create a versioned Resource +The following may prove useful in helping you to understand CreateJsonDatabase.java or CreateXmlDatabase.java within the tutorials directory of sirix. + +Please note that if you have already created a database, you may skip steps 1 - 3 +* 1 - Create a new variable to store the database file using "Constants.SIRIX_DATA_LOCATION.resolve("nameOfDataBase");" +* 2 - Check if the database already exists and if so, delete it. +* 3 - Create a new variable to store a new database configuration and use this variable to create a new Json/Xml database instance. +* 6 - Try to create a new variable, storing the database +* 7 - If able to create this variable, create a new Resource to store in the database +* 8 - Try to begin a new Resource Session and if able, begin a Node transaction +* 9 - use the JsonDocumentCreator to create a new document, giving the Node transaction as input. +* 10 - From here you may add as many changes as you like using various methods upon the Node Transaction. Once you are happy with your changes use .commit(); to confirm them. From 48f354c38bfa070cf78677acbd00df9d9f5852d5 Mon Sep 17 00:00:00 2001 From: henrycotton1 <141598786+henrycotton1@users.noreply.github.com> Date: Sun, 29 Oct 2023 16:59:26 +1100 Subject: [PATCH 09/11] Fixed a typo --- docs/concepts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts.md b/docs/concepts.md index 5ebe0fd..c7e3df5 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -150,7 +150,7 @@ The following may prove useful in helping you to understand CreateJsonDatabase.j * 8 - Try to begin a new Resource Session and if able, Insert the specified file into the subtree. ## How to create a versioned Resource -The following may prove useful in helping you to understand CreateJsonDatabase.java or CreateXmlDatabase.java within the tutorials directory of sirix. +The following may prove useful in helping you to understand CreateVersionedJsonResource.java or CreateVersionedXmlResource.java within the tutorials directory of sirix. Please note that if you have already created a database, you may skip steps 1 - 3 * 1 - Create a new variable to store the database file using "Constants.SIRIX_DATA_LOCATION.resolve("nameOfDataBase");" From bc2fda1cf8df39d26345d56cb16d2551461e9a51 Mon Sep 17 00:00:00 2001 From: henrycotton1 <141598786+henrycotton1@users.noreply.github.com> Date: Sun, 29 Oct 2023 17:03:19 +1100 Subject: [PATCH 10/11] Further edits --- docs/concepts.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/concepts.md b/docs/concepts.md index c7e3df5..b4c89c5 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -156,8 +156,8 @@ Please note that if you have already created a database, you may skip steps 1 - * 1 - Create a new variable to store the database file using "Constants.SIRIX_DATA_LOCATION.resolve("nameOfDataBase");" * 2 - Check if the database already exists and if so, delete it. * 3 - Create a new variable to store a new database configuration and use this variable to create a new Json/Xml database instance. -* 6 - Try to create a new variable, storing the database -* 7 - If able to create this variable, create a new Resource to store in the database -* 8 - Try to begin a new Resource Session and if able, begin a Node transaction -* 9 - use the JsonDocumentCreator to create a new document, giving the Node transaction as input. -* 10 - From here you may add as many changes as you like using various methods upon the Node Transaction. Once you are happy with your changes use .commit(); to confirm them. +* 4 - Try to create a new variable, storing the database +* 5 - If able to create this variable, create a new Resource to store in the database +* 6 - Try to begin a new Resource Session and if able, begin a Node transaction +* 7 - Use the JsonDocumentCreator to create a new document, giving the Node transaction as input. +* 8 - From here you may add as many changes as you like using various methods upon the Node Transaction. Once you are happy with your changes use .commit(); to confirm them. From 268a0852e22820f8124a911752f814481c23038f Mon Sep 17 00:00:00 2001 From: henrycotton1 <141598786+henrycotton1@users.noreply.github.com> Date: Sun, 29 Oct 2023 17:12:44 +1100 Subject: [PATCH 11/11] Provided some further documentation Added a brief overview of the overall architecture of Sirix, mentioned the examples bundle and its utility to a newcomer and wrote some "How To's". All of these changes are located in concepts.md --- docs/concepts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts.md b/docs/concepts.md index b4c89c5..fb4c3f9 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -16,7 +16,7 @@ SirixDB can help answer questions such as the following: Give me last month's hi Let's turn our focus toward the question of why historical data has not been retained in the past. We postulate that new storage advances in recent years present possibilities to build sophisticated solutions to help answer those questions without the hurdle state-of-the-art systems bring. ## A brief overview of the overall architecture -Before we go into the specifics of SirixDB, its important to have an understanding of the overall architecture. At the core of SirxDB are databases. These instances store resources, which are generally binary tree encodings of either JSON or XML files - (more on that under the Tree-structure subheading). +Before we go into the specifics of SirixDB, its important to have an understanding of the overall architecture. At the core of SirxDB are databases. These database instances store resources, which are generally binary tree encodings of either JSON or XML files - (more on that under the Tree-structure subheading). From this database instance, you may either create a new resource or begin a resource session to start as many read-only transactions as you would like or just a single read-write transaction. In essence, you may think of the architecture as one big tree full of tries (the main document index), where revisions to the tree are always appended. The data of the tries either store the nodes of the JSON or XML trees or they store secondary indexes!