The scraping

Things are now back to normal after the holiday break, which I hope everyone enjoyed. I duly caught the traditional lousy cold, which I wouldn’t recommend to anyone, and was laid up for several days over the New Year. Given the reports in Great Britain of a violent stomach virus, which has closed many hospital wards, I probably came off quite well compared to others.

A day or so ago came the sad news from Harry Fougiaxis of the death of Greek Grandmaster composer Byron Zappas (1927-2008). I will be posting some of his wonderful compositions on the website in the near future.

Some of you will remember that last year Anders Thulin warned me about ‘scraping’. Well, it has happened.

Over a period of 2 to 3 hours, in the very early morning of 19th December 2007 some 700 MB of data was requested and delivered by my website. This was 47% of the monthly total for December. Analysis in the server logs shows that the requesting machine (server?) had an IP address of 70.174.136.105. This machine is apparently domiciled in Fairfax, Virginia in the USA. Given the amount of data shifted in such a short time, this was clearly a robot, not a real user. The robot hit 75,765 pages and the logs show that it looped around problem id numbers, apparently using several threads. These records imply that it went through the whole of the Meson database. From evidence in the logs, it seems likely that this robot was written in the Python programming language. So somebody, presumably in the USA, has a copy of the Meson database, and I suppose that it is likely that it will appear elsewhere on the Internet at some point in the future.

This makes me angry for a couple of reasons. First, although (1) I made the decision to make Meson freely available, (2) I don’t own any of the problems and (3) no security has been broken, I nonetheless feel that the problems I have spent years collecting have been stolen. Second, my bandwidth allowance from my service provider might have been exceeded (it wasn’t, as it happens) and I could have ending up paying for the robot-wielder to acquire the data.

There is no point in locking the door after the horse has bolted, so I shall be taking no immediate steps to stop this happening again. However, I shall not be adding any new problems or making any amendments until I have devised and implemented a successful anti-robot strategy. This may take some while, as I have other tasks of an arguably higher priority. Part of the strategy is likely to include registration for users. Comments welcome.