Crawlers (or bots) are used to assemble information available on the internet. By utilizing website navigation menus, and studying internal and external hyperlinks, the bots start to understand the context of a web page. Of course, the words, images, and different knowledge on pages additionally hel