Add a scanner option for ignoring comment lines

[jwodder]

E.g., as in robots.txt

If lines matching r"^\s+#" are counted as comments, there will need to be some way to discern whether such lines are non-comment continuation lines instead.