- API
 |
Function List | |
|
Topical List | |
Main Index
|
Introduction | |
|
Calling Sequences | |
|
Query Processing | |
|
Relevancy Ranking | |
|
Data Types | |
|
Error Handling | |
|
Support | |
- Lextek Products
|
Onix Text Retrieval Engine | |
|
Lextek Document Profiler & Categorizer | |
|
Brevity Document Summarizer | |
|
RouteX Routing Engine | |
|
Language Identifier | |
ixRobotsTxtLength
Name
ixRobotsTxtLength
Synopsis
size_t ixRobotsTxtLength(RobotsTxtParserT Parser)
Arguments
Parser: A robots.txt parser which has a robots.txt file read into it.
Returns
The approximate number of bytes a robots.txt file would be when output.
Description
ixOutputCompactRobotsTxt allows you to output a "compact" robots.txt which only has instructions which apply to your web crawler and without any of the comments which occurred in the original robots.txt file. Use ixRobotsTxtLength to find out how large your memory buffer needs to be to hold this compact robots.txt. This function only returns an approximate length and so it is a good idea to add a few bytes to the value returned from ixRobotsTxtLength before allocating and using any memory for ixOutputCompactRobotsTxt.
See Also
Robots.txt, Robots Spec
ixOutputCompactRobotsTxt, ixCreateRobotsTxtParser, ixDeleteRobotsTxtParser, ixSetRobotName, ixParseRobotsTxt, ixRobotsPermissionGranted, ixRobotsPermissionGrantedFullURL