Onix Text Retrieval Toolkit
API Reference

API
Function List
Topical List

Main Index

Introduction
Calling Sequences
Query Processing
Relevancy Ranking
Data Types
Error Handling
   
Support
   
Lextek Products
Onix Text Retrieval Engine
Lextek Document Profiler & Categorizer
Brevity Document Summarizer
RouteX Routing Engine
Language Identifier

ixSetRobotName

Name

ixSetRobotName

Synopsis

void ixSetRobotName( RobotsTxtParserT Parser, char *Name, size_t CompareLength)

Arguments

Parser: A Robots.txt parser which was created by a call to ixCreateRobotsTxtParser.

Name: A pointer to a NULL terminated string with the name of your webcrawler.

CompareLength: The number of characters in your webcrawler's name which should be used in comparisons with robots.txt instructions.

Returns

Nothing.

Description

Robots.txt is capable of giving specific instructions to different webcrawlers.  This is handled by each webcrawler having a name and that name being specified in robots.txt.  ixSetRobotName lets you set the name of your webcrawler's robot so it may follow the instructions given in robots.txt.  ixSetRobotName also lets you set a compare length.  This is useful so that you may include version information in your robot's name and still have it match (or not match) properly with robots.txt.  For example, if your web crawler is called "ICrawl v. 2.06", you would want to have a compare length of 6 which would allow your robot to follow directions for all versions named "ICrawl".  The name for the robot is specified as a zero terminated string and may be up to 100 characters in length.

See Also

Robots.txt, Robots Spec
ixCreateRobotsTxtParser,ixDeleteRobotsTxtParser, ixParseRobotsTxt, ixOutputCompactRobotsTxt, ixRobotsPermissionGranted, ixRobotsPermissionGrantedFullURL, ixRobotsTxtLength