Federated Wiki Feeds Logic

The background wiki checking logic is that every wiki in the all feeds list gets checked once an hour and any wikis flagged as active are checked once a minute.

When a wikis sitemap is pulled, if I find that it's updated I update my cached copy and ping the rssCloud server of an update.

I also fetch a copy of wikis from search.fed.wiki.org and every minute take five wikis from that list and see if I can get a list of peers for it. If it doesn't have the "present" plugin enabled, it returns one item.

Any wiki in the list of peers that I haven't seen before gets added to the all feeds list as active, although if it isn't active it will get flagged as inactive after the first pull of its sitemap.

This way, any wiki that is updated regularly should have updates show up in the river within a minute (or so) while stale wikis that don't get updated regularly don't gum up the system.

Rosters needed special logic. Most rosters in the fediverse won't need to be fetched on a regular basis because nobody is viewing their river.

However, fetching a roster the first time can be slow. Especially if it links out to other rosters or lists of references.

So if a roster list or river is fetched, it will be added to a list of active rosters. These rosters will be checked at a rate of one roster a minute in a loop. If the list or river hasn't been fetched in over two weeks it's dropped from the list of active rosters and won't be kept fresh in the cache.