3W24 is an index of web pages. If you want your site to appear in the search results not to block our crawler.
The 3w24 bot supports the use robots.txt file (The Robots Exclusion Protocol) since version 1.1, the 1.0 version is considered an alpha version and not follow this protocol. If the file is not found it is considered that the access to the web is unrestricted.
If the file exists and can be downloaded 3w24 robot parses its contents. If a string "User-agent: 3w24bot" or "User-agent: *" is detected, it then checks the directive "Disallow:".
If the directive has no arguments, it is considered that the access to the web is unrestricted.