April 11, 2018
---------------------------------------------------------------
I've been working on rewriting Contrition. As a major part of
this, the indexer is undergoing a lot of changes.

In the current version of Contrition, things are very closely
tied to my server configuration and require a lot of manual
steps to add servers, run the indexer, and update the database.
I'm hoping to resolve this.

While conducting some test runs, it randomly selected a few
servers from the current data set. Due to bugs on my part, it
did not load robots.txt (for the servers that have this),
which may have caused it to place undue load on the affected
servers.

Specifically, in Contrition's old indexer, I had a manual list
of exclusions hard coded. This wasn't copied over into the new
one, and a parsing bug in the code to load the list from a
file caused the exclusion list to be empty.

I apologize for this.

The servers that were scanned by this were:

  dusted.dk
  forthworks.com
  fritterware.org
  gopher.black
  pain.ka10.de
  tellus.stangled.net
  vernunftzentrum.de
  zelibertinegamer.me

I have fixed this by now implementing a proper robots.txt
handler and have stopped all futher scans until after I get a
chance to test this much more extensively.

---------------------------------------------------------------