void ixOutputCompactRobotsTxt( RobotsTxtParserT Parser, char *RobotsTxt)
Parser: A robots.txt parser created with ixCreateRobotsTxtParser() which has parsed a robots.txt file.
RobotsTxt: A pointer to buffer large enough to hold the compact robots.txt.
robots.txt allows web administrators to control which robots access their web sites and how the web site is to be crawled. ixOutputCompactRobotsTxt() outputs a shorted form of robots.txt to a buffer which applies only to your robot. This shorted form does not contain instructions for other robots or any comments found within the original robots.txt file and is suitable for storing in a database to reduce network usage during the web crawling. The buffer RobotsTxt should be at least ixRobotsTxtLength() long and preferably a bit longer. The compact form of robots.txt will be placed in the RobotsTxt buffer and will be NULL (0x00) terminated.
Robots.txt, Robots Spec
ixCreateRobotsTxtParser, ixDeleteRobotsTxtParser, ixSetRobotName, ixParseRobotsTxt, ixRobotsPermissionGranted, ixRobotsPermissionGrantedFullURL, ixRobotsTxtLength