This is my final draft of a letter that was published in April 2007 in The Psychologist Volume 20, Number 4.

PLEASE FEED THE SPIDERS.

Do you provide your students with links to online resources? Are your materials in a secure system such as blackboard? If so, did you realise you are helping to kill off the very resources you are recommending?

Search engines build their indexes using programs known as ‘spiders’. Spiders automatically crawl around the web from link to link, recording both the content of the web and how it is all linked together. When you search for a particular term, search engines return a list of pages ordered by their relative values, and these value scores are calculated using the linking information gathered by the spiders.

Just as the impact factor of a printed article is assessed by citations, if a certain web page has lots of links to it from other websites, this suggests the authors of those other websites think the page has useful content. These links act as votes for that page and the value of a page, from a search engine’s point of view, is (approximately) a weighted sum of the values of all the other web-pages that link to it.

This system provides an effective and reasonable way to indirectly assess the likely utility of web pages. However the system is dependent on the idea that the entire internet is available for public access. Search engine spiders cannot look inside secure systems so they cannot see any links that are stored there. As departments increasingly move content (including links to third-party resources) into secure areas, their ‘votes’ for these third-party resources are lost.

As an example, three years ago my own website on attachment theory had about 150 visits a day and a first page listing in Google for the search term ‘attachment theory’. Today I have about 450 visits a day yet my site is now three pages deep in a Google search for the same term. Since I have software that tracks where my visitors come from, I know that the number of links to my site has steadily increased but new links and many old links are now hidden inside secure systems. As far as the search engines are concerned, my site has lost rather than gained votes.

I am not alone in experiencing this. As universities increasingly move their internet-based materials into secure areas, resources linked to by those who are (arguably) in the best position to judge their value are losing ground to pages serving commercial interests. Companies often pay to get incoming links and people searching for academic content on the internet will increasingly find the search engine listings are dominated by booksellers, full-text-for-$30-journal-sites, marketing for college courses, and ‘example’ essay writing services.

While there may be other factors involved, our increasing use of secure systems for course materials is undoubtedly reducing the ‘impact factor’ of the content that we leave in the public domain. Fortunately the solution is simple: whenever you include a link to an internet resource in your secured teaching materials, make sure you also provide a link from a public access page such as a departmental “useful links” page or a personal homepage. If you want to keep open access psychology resources accessible then make sure your links are accessible as well: please feed the spiders.

Richard Atkins, London School of Hygiene and Tropical Medicine