Onix Text Retrieval Toolkit
API Reference

API
Function List
Topical List

Main Index

Introduction
Calling Sequences
Query Processing
Relevancy Ranking
Data Types
Error Handling
   
Support
   
Lextek Products
Onix Text Retrieval Engine
Lextek Document Profiler & Categorizer
Brevity Document Summarizer
RouteX Routing Engine
Language Identifier

ixParseRobotsTxt

Name

ixParseRobotsTxt

Synopsis

void ixParseRobotsTxt(RobotsTxtParserT Parser, char *RobotsTxt)

Arguments

Parser: A robots.txt parser which was created by a call to ixCreateRobotsTxtParser.

RobotsTxt: A pointer to a buffer containing a robots.txt file. The robots.txt must be NULL terminated.

Returns

Nothing.

Description

ixParseRobotsTxt tells the parser to parse the NULL terminated robots.txt contained in the buffer RobotsTxt.  It is important that you set your robot's name with a call to ixSetRobotName before you parse the robots.txt file so the parser will know which instructions pertain to your crawler.  Note that a robots.txt parser is good for only 1 robots.txt file.  Do not try to use a parser on multiple robots.txt files.  (It is better to create a new one and dispose of the old ones as you do your web crawling.)

See Also

Robots.txt, Robots Spec
ixCreateRobotsTxtParser, ixDeleteRobotsTxtParser, ixSetRobotName, ixOutputCompactRobotsTxt, ixRobotsPermissionGranted, ixRobotsPermissionGrantedFullURL, ixRobotsTxtLength