- API
 |
Function List | |
|
Topical List | |
Main Index
|
Introduction | |
|
Calling Sequences | |
|
Query Processing | |
|
Relevancy Ranking | |
|
Data Types | |
|
Error Handling | |
|
Support | |
- Lextek Products
|
Onix Text Retrieval Engine | |
|
Lextek Document Profiler & Categorizer | |
|
Brevity Document Summarizer | |
|
RouteX Routing Engine | |
|
Language Identifier | |
ixOutputCompactRobotsTxt
Name
ixOutputCompactRobotsTxt
Synopsis
void ixOutputCompactRobotsTxt( RobotsTxtParserT Parser, char *RobotsTxt)
Arguments
Parser: A robots.txt parser created with ixCreateRobotsTxtParser() which has parsed a robots.txt file.
RobotsTxt: A pointer to buffer large enough to hold the compact robots.txt.
Returns
Nothing.
Description
robots.txt allows web administrators to control which robots access their web sites and how the web site is to be crawled. ixOutputCompactRobotsTxt() outputs a shorted form of robots.txt to a buffer which applies only to your robot. This shorted form does not contain instructions for other robots or any comments found within the original robots.txt file and is suitable for storing in a database to reduce network usage during the web crawling. The buffer RobotsTxt should be at least ixRobotsTxtLength() long and preferably a bit longer. The compact form of robots.txt will be placed in the RobotsTxt buffer and will be NULL (0x00) terminated.
See Also
Robots.txt, Robots Spec
ixCreateRobotsTxtParser, ixDeleteRobotsTxtParser, ixSetRobotName, ixParseRobotsTxt, ixRobotsPermissionGranted, ixRobotsPermissionGrantedFullURL, ixRobotsTxtLength