added uncertainty scaling based on the validation set #33
Conversation
| @@ -1102,7 +1104,20 @@ def estimate_uncertainty(self,rel_node_dof_tolerance=1e-5): | |||
| node.rule.uncertainty = node.parent.rule.uncertainty | |||
| elif node.rule.num_data == 0: | |||
| node.rule.uncertainty = 0.0 #if n=0 the LASSO should drive node.rule.value to zero so there should be approximately no variance contribution | |||
|
|
|||
|
|
|||
| if self.validation_set and len(node_uncertainties) > 0: | |||
There was a problem hiding this comment.
Enable assignment of the validation_set in this function in case someone wants to tune it after generating the tree easily.
| return confidence_levels, proportion_correct | ||
|
|
||
| def objective_function(scaling_factor, errs, uncs, n = 500): |
There was a problem hiding this comment.
Can we make "objective_function" either more specific or embed in another function
|
|
||
|
|
||
| if self.validation_set and len(node_uncertainties) > 0: | ||
| val_predictions_uncertainties = [self.evaluate(d.mol, estimate_uncertainty=True) for d in self.validation_set] |
There was a problem hiding this comment.
We probably only want to run this once perhaps at the end of generation so either we put this is in a separate function "scale_uncertainties" or add a flag to this function to not do this so we can not scale them everytime we call this during tree generation except one final time at the end.
| def get_bounded_fraction(errs, uncs, confidence_level): | ||
| t = scipy.stats.norm.ppf((1 + confidence_level) / 2) | ||
| return np.sum(uncs * t >= np.abs(errs)) / len(errs) |
There was a problem hiding this comment.
I think this formula is wrong...the uncs you're pulling I believe are variances not standard deviations...also separately I think the len(errs) needs a sqrt?
No description provided.