"last_seen" date on publisher gets updated even when rss contents are missing

[roncanepa]

@mielliott noticed that the last_seen date for a publisher seemed to get updated even when it shouldn't. Did some digging and was able to confirm. The code handles any http return code >400 properly, but one particular publisher's rss URL was returning their custom "error" page with a 200 OK. Thus, it doesn't get handled or bubbled up properly, nor is deeper inspection of the string contents occurring.

This field is mostly just for human use but should still be corrected.

line 164 of update_db_from_rss in /idigbio_ingestion/update_publisher_recordset.py , which then passes to _do_rss in the same file

[danstoner]

Which is the particular publisher?

[roncanepa]

@danstoner I've sent you a link that has additional info and logging

[danstoner]

We probably want to add a last_feed_parse_success date or something similar if we need to track the results of the feed parse event.

[danstoner]

We are using the feedparser lib. We can probably leverage bozo detection.

https://pythonhosted.org/feedparser/bozo.html#advanced-bozo

For example, on content that is a web page instead of an xml RSS feed:

>>> feed.bozo
1
>>> feed.bozo_exception
SAXParseException('mismatched tag',)

Note to self, also check if we should be using len(feed.entries) instead of len(feed).

[nrejac]

See: #72 to find where to correct this.