Web crawling

The process through which Internet bots load and browse a website in particular and the World Wide Web in general. These robots are known as crawlers, spiders or bots.

Search engines find and index pages online by crawling the web. The process of setting up your website to facilitate crawling and indexing by search engines is known as technical SEO.