Onix Text Retrieval Toolkit
API Reference

API
Function List
Topical List

Main Index

Introduction
Calling Sequences
Query Processing
Relevancy Ranking
Data Types
Error Handling
   
Support
   
Lextek Products
Onix Text Retrieval Engine
Lextek Document Profiler & Categorizer
Brevity Document Summarizer
RouteX Routing Engine
Language Identifier

ixRobotsTxtLength

Name

ixRobotsTxtLength

Synopsis

size_t ixRobotsTxtLength(RobotsTxtParserT Parser)

Arguments

Parser: A robots.txt parser which has a robots.txt file read into it.

Returns

The approximate number of bytes a robots.txt file would be when output.

Description

ixOutputCompactRobotsTxt allows you to output a "compact" robots.txt which only has instructions which apply to your web crawler and without any of the comments which occurred in the original robots.txt file. Use ixRobotsTxtLength to find out how large your memory buffer needs to be to hold this compact robots.txt.  This function only returns an approximate length and so it is a good idea to add a few bytes to the value returned from ixRobotsTxtLength before allocating and using any memory for ixOutputCompactRobotsTxt.

See Also

Robots.txt, Robots Spec
ixOutputCompactRobotsTxt, ixCreateRobotsTxtParser, ixDeleteRobotsTxtParser, ixSetRobotName, ixParseRobotsTxt, ixRobotsPermissionGranted, ixRobotsPermissionGrantedFullURL