- API
 |
Function List | |
|
Topical List | |
Main Index
|
Introduction | |
|
Calling Sequences | |
|
Query Processing | |
|
Relevancy Ranking | |
|
Data Types | |
|
Error Handling | |
|
Support | |
- Lextek Products
|
Onix Text Retrieval Engine | |
|
Lextek Document Profiler & Categorizer | |
|
Brevity Document Summarizer | |
|
RouteX Routing Engine | |
|
Language Identifier | |
ixParseRobotsTxt
Name
ixParseRobotsTxt
Synopsis
void ixParseRobotsTxt(RobotsTxtParserT Parser, char *RobotsTxt)
Arguments
Parser: A robots.txt parser which was created by a call to ixCreateRobotsTxtParser.
RobotsTxt: A pointer to a buffer containing a robots.txt file. The robots.txt must be NULL terminated.
Returns
Nothing.
Description
ixParseRobotsTxt tells the parser to parse the NULL terminated robots.txt contained in the buffer RobotsTxt. It is important that you set your robot's name with a call to ixSetRobotName before you parse the robots.txt file so the parser will know which instructions pertain to your crawler. Note that a robots.txt parser is good for only 1 robots.txt file. Do not try to use a parser on multiple robots.txt files. (It is better to create a new one and dispose of the old ones as you do your web crawling.)
See Also
Robots.txt, Robots Spec
ixCreateRobotsTxtParser, ixDeleteRobotsTxtParser, ixSetRobotName, ixOutputCompactRobotsTxt, ixRobotsPermissionGranted, ixRobotsPermissionGrantedFullURL, ixRobotsTxtLength