Observed Web Robot Behavior on Decaying Web Subsites

Authors

Joan A. Smith, Old Dominion UniversityFollow
Frank McCown Ph.D., Harding UniversityFollow
Michael L. Nelson, Old Dominion University

Document Type

Article

Publication Title

D-Lib Magazine

Publication Date

2-2006

Volume

12

Issue

2

Abstract

We describe the observed crawling patterns of various search engines (including Google, Yahoo and MSN) as they traverse a series of web subsites whose contents decay at predetermined rates. We plot the progress of the crawlers through the subsites, and their behaviors regarding the various file types included in the web subsites. We chose decaying subsites because we were originally interested in tracking the implication of using search engine caches for digital preservation. However, some of the crawling behaviors themselves proved to be interesting and have implications on using a search engine as an interface to a digital library.

Creative Commons Usage License

This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.

Copyright held by

Author

Recommended Citation

Smith, J. A., McCown, F., & Nelson, M. L. (2006). Observed Web Robot Behavior on Decaying Web Subsites. D-Lib Magazine, 12 (2). http://dx.doi.org/10.1045/february2006-smith