Onix Text Retrieval Toolkit
API Reference

API
Function List
Topical List

Main Index

Introduction
Calling Sequences
Query Processing
Relevancy Ranking
Data Types
Error Handling
   
Support
   
Lextek Products
Onix Text Retrieval Engine
Lextek Document Profiler & Categorizer
Brevity Document Summarizer
RouteX Routing Engine
Language Identifier

ixOutputCompactRobotsTxt

Name

ixOutputCompactRobotsTxt

Synopsis

void ixOutputCompactRobotsTxt( RobotsTxtParserT Parser, char *RobotsTxt)

Arguments

Parser: A robots.txt parser created with ixCreateRobotsTxtParser() which has parsed a robots.txt file.

RobotsTxt: A pointer to buffer large enough to hold the compact robots.txt.

Returns

Nothing.

Description

robots.txt allows web administrators to control which robots access their web sites and how the web site is to be crawled.  ixOutputCompactRobotsTxt() outputs a shorted form of robots.txt to a buffer which applies only to your robot.  This shorted form does not contain instructions for other robots or any comments found within the original robots.txt file and is suitable for storing in a database to reduce network usage during the web crawling.  The buffer RobotsTxt should be at least ixRobotsTxtLength() long and preferably a bit longer.  The compact form of robots.txt will be placed in the RobotsTxt buffer and will be NULL (0x00)  terminated.

See Also

Robots.txt, Robots Spec
ixCreateRobotsTxtParser, ixDeleteRobotsTxtParser, ixSetRobotName, ixParseRobotsTxt, ixRobotsPermissionGranted, ixRobotsPermissionGrantedFullURL, ixRobotsTxtLength