Absolutely, but it turns out to be a really common misconfiguration found in default configs and tutorials. Definitely something to be aware of when configuring FastCGI.
My thoughts exactly. Turning a living creature into a plaything and masquerading it as educational is just sick. It degrades the creature, and it degrades the science.
If you want to learn about GP, the Genetic Programming Field Guide - http://www.gp-field-guide.org.uk/ - is an awesome book, it taught me a lot. In fact, I liked it so much, I bought a hard copy. Highly recommended!
For this particular project I needed access to more than just the page itself - I needed to interact with scripts and have access to the page resources on a level which matched that of a browser. What you're seeing is only the tip of a much larger iceberg, for which efficiency wasn't the main objective.
You're correct that you can write more efficient spiders if you don't want to do anything fancy (indeed, I've written simple scrapers and spiders in various languages), but this project leant itself better to a full browser environment.
Thank goodness for that. Countless hours have been lost debating how best to implement this pointless law, and the amount of business lost due to unsightly and confusing consent banners must have been huge.
I think it might have been Derek Sivers who wrote a great essay about how in business the best policy is often just to ignore silly rules and regulations (except the really serious ones) until someone pulls you up on it. I took that to heart, and that was ultimately the best policy with this.
The ICO had dropped enough hints that they'd be lenient and go after the big boys and most evil violators first that 99% of sites were wasting their time panicking about implementing this stuff.. yet panic they did.
yes given that my employer a FTSE 100 publisher must have spent a huge amount time and money on this stupid law - can we claim this back against our tax bill.
Well, yes. Generally and imprecisely speaking, expenses are deducted from revenues and the net is what's taxable. Your employer will end up paying a little less corporation tax because of it. Whether it's a net loss for the government is another matter, as what isn't paid in corporation tax might be paid in national insurance and individual income taxes.
Could your employer sue the government for their compliance costs? Almost certainly not.
