Web crawler

A web crawler is a bot that navigates the World Wide Web, most often for the purpose of discovering and indexing pages for search engines. Sites that have built their own web indexes will use their own crawlers.

Making sure your website can be found, accessed and crawled by is the primary purpose of technical SEO.