- API
 |
Function List | |
|
Topical List | |
Main Index
|
Introduction | |
|
Calling Sequences | |
|
Query Processing | |
|
Relevancy Ranking | |
|
Data Types | |
|
Error Handling | |
|
Support | |
- Lextek Products
|
Onix Text Retrieval Engine | |
|
Lextek Document Profiler & Categorizer | |
|
Brevity Document Summarizer | |
|
RouteX Routing Engine | |
|
Language Identifier | |
ixSetRobotName
Name
ixSetRobotName
Synopsis
void ixSetRobotName( RobotsTxtParserT Parser, char *Name, size_t CompareLength)
Arguments
Parser: A Robots.txt parser which was created by a call to ixCreateRobotsTxtParser.
Name: A pointer to a NULL terminated string with the name of your webcrawler.
CompareLength: The number of characters in your webcrawler's name which should be used in comparisons with robots.txt instructions.
Returns
Nothing.
Description
Robots.txt is capable of giving specific instructions to different webcrawlers. This is handled by each webcrawler having a name and that name being specified in robots.txt. ixSetRobotName lets you set the name of your webcrawler's robot so it may follow the instructions given in robots.txt. ixSetRobotName also lets you set a compare length. This is useful so that you may include version information in your robot's name and still have it match (or not match) properly with robots.txt. For example, if your web crawler is called "ICrawl v. 2.06", you would want to have a compare length of 6 which would allow your robot to follow directions for all versions named "ICrawl". The name for the robot is specified as a zero terminated string and may be up to 100 characters in length.
See Also
Robots.txt, Robots Spec
ixCreateRobotsTxtParser,ixDeleteRobotsTxtParser, ixParseRobotsTxt, ixOutputCompactRobotsTxt, ixRobotsPermissionGranted, ixRobotsPermissionGrantedFullURL, ixRobotsTxtLength