Handle the cases were a field is both indexed and not indexed within a time range

[cawaring]

Fields can be added or removed from the index over time. We do not always handle cases where a field spans a time range where it's both indexed and not index. We need to handle this case appropriately. This would include using the field as an indexed field for the portion of time it is indexed and avoid using it to prune queries and ranges when it's not indexed.

[ivakegg]

Suggest marking the time ranges in the datawave metadata table, and then using that info in the DefaultQueryPlanner appropriately. The list of indexed fields sent to the QueryIterator will need to be adjusted for each range as appropriate.

[ivakegg]

The Data Dictionary should also show this information.

[ivakegg]

FYI, the PushdownMissingIndexRangeNodesVisitor is sort of an initial implementation of this capability. However instead of pushing those nodes down, perhaps range stream index lookup can return a stream of shards for the holes, and also modify the query for those ranges to push the terms down to the evaluation phase. The comment I put into another pull request may help one understand this: #925 (comment).

[ivakegg]

BTW having the ingest automatically fill in the metadata for when a field is indexed would be great as well. This should probably be encoded in the same way we are condensing the F column here: #828. For existing systems we would need to manually update (a tool would be nice) the metadata for data already ingested.

[jzgithub1]

BTW having the ingest automatically fill in the metadata for when a field is indexed would be great as well. This should probably be encoded in the same way we are condensing the F column here: #828. For existing systems we would need to manually update (a tool would be nice) the metadata for data already ingested.

This is done with the new IndexColumnIterator class.

[jzgithub1]

BTW having the ingest automatically fill in the metadata for when a field is indexed would be great as well. This should probably be encoded in the same way we are condensing the F column here: #828. For existing systems we would need to manually update (a tool would be nice) the metadata for data already ingested.

Already implemented in the pull request for this ticket.

[jzgithub1]

FYI, the PushdownMissingIndexRangeNodesVisitor is sort of an initial implementation of this capability. However instead of pushing those nodes down, perhaps range stream index lookup can return a stream of shards for the holes, and also modify the query for those ranges to push the terms down to the evaluation phase. The comment I put into another pull request may help one understand this: #925 (comment).

The code inside of processTree in the DefaultQueryPlanner looks like it should be able to take care to index holes. The problem is the index holes are never set inside of the ShardQueryConfiguration object prior to this line on line 1082:
queryTree = PushdownMissingIndexRangeNodesVisitor.pushdownPredicates(queryTree, config, metadataHelper);

I am going to investigate your idea more thoroughly though since this is your area of expertise. I will engage @lbschanno about this issue.

[ivakegg]

As I believe you have discovered, the existing "index hole" mechanism is value based (i.e. values are missing in the index from 'a' to 'b' for a specified date range). This ticket will result in a different kind of index hole which is field based meaning that a field will be missing all entries in the index for a specified date range or date ranges. I suggest we use the terminology of ValueIndexHole and FieldIndexHole. Once we have the FieldIndexHoles encoded in the metadata table and provide a mechanism to get those out, we should put those in the ShardQueryConfiguration along side of the ValueIndexHoles.

Currently we have an IndexHoleMarkerJexlNode that is used to mark fields in the query that need to be delayed until evaluation time. When this is added to the query plan, this denotes that we cannot use that term for index lookup across the entire query date range. This avoids missing any results but avoids being able to use the index for those portions of the date range that might still be available to us. It would be desirable for both ValueIndexHoles and FieldIndexHoles that we create separate plans for portions of the query range to be able to take full advantage of the index where possible. I have two possibilities that come to mind on how to achieve this. We should evaluate these and any other ideas before continuing with the implementation.

1. We could create a query planner that uses a separate DefaultQueryPlanner instance per portion of the query date range which is consistent in terms of index holes. This would result is multiple range streams which could be funneled into one range stream. This was the existing DefaultQueryPlanner would not really change except for adding a visitor (or extending the existing index hole visitor) to mark fields that holes for the specified portion of the query date range.
2. We could modify the global index lookups to return exhaustive sets of shard ranges for those portions of the index that are not available for the field being queried. So the index lookup would return the UIDs (document ranges) and shards (fi ranges) when scanning the index and when it hits the date range that is deemed unavailable/missing, it would simply generate artificial shards (fi ranges). The same thing would need to be done for the field index scans, including the ivarators (DatawaveFieldIndexCachingIterator…).
3. The existing DefaultQueryPlanner would generate multiple plans concurrently for each consistent portion of the date range relative to index holes. This is basically like option 1, but the management of the plans would be pushed into the existing DefaultQueryPlanner.

[lbschanno]

Created PR datawave-metadata-utils/pull/29 to add functionality for retrieving field index holes from the metadata table.