If you’ve configured Arena to email you every exception that occurs in Arena, you may get tired of seeing the exceptions that occur when a search bot crawls your site. I certainly do. Fortunately, you can configure Arena so that it ignores exceptions based on the value of the HTTP_USER_AGENT http header.
When an exception occurs, Arena will evaluate the current HTTP_USER_AGENT value, and if it contains any of the values you’ve defined in the “ExceptionUserAgentIgnore” organization setting, then the exception email will not be sent.
Here’s the current value of our “ExceptionUserAgentIgnore” setting…
msnbot;Slurp;CCV Search;Googlebot;gsa-crawler;ia_archiver;BusyBot;Gigabot;MJ12bot;PycURL;ScanAlert;exabot;singingfish;becomebot;converacrawler;twiceler;crawler;WebCopier
Since there always seems to be a new search bot created, you may need to periodically add new values to this org setting. You’ll now when you begin to get a flurry of new exceptions reported.
