Package crawlercommons.urlfrontier

Class URLFrontierGrpc.URLFrontierBlockingStub

java.lang.Object

io.grpc.stub.AbstractStub<URLFrontierGrpc.URLFrontierBlockingStub>

io.grpc.stub.AbstractBlockingStub<URLFrontierGrpc.URLFrontierBlockingStub>

crawlercommons.urlfrontier.URLFrontierGrpc.URLFrontierBlockingStub

Enclosing class:

URLFrontierGrpc

public static final class URLFrontierGrpc.URLFrontierBlockingStub
extends io.grpc.stub.AbstractBlockingStub<URLFrontierGrpc.URLFrontierBlockingStub>

A stub to allow clients to do limited synchronous rpc calls to service URLFrontier.

Nested Class Summary

Nested classes/interfaces inherited from class io.grpc.stub.AbstractStub

io.grpc.stub.AbstractStub.StubFactory<T extends io.grpc.stub.AbstractStub<T>>

Method Summary

Modifier and Type	Method	Description
Urlfrontier.Empty	blockQueueUntil(Urlfrontier.BlockQueueParams request)	Block a queue from sending URLs; the argument is the number of seconds of UTC time since Unix epoch 1970-01-01T00:00:00Z.
protected URLFrontierGrpc.URLFrontierBlockingStub	build(io.grpc.Channel channel, io.grpc.CallOptions callOptions)
Urlfrontier.Long	countURLs(Urlfrontier.CountUrlParams request)	Count URLs currently in the frontier *
Urlfrontier.Long	deleteCrawl(Urlfrontier.DeleteCrawlMessage request)	Delete an entire crawl, returns the number of URLs removed this way *
Urlfrontier.Long	deleteQueue(Urlfrontier.QueueWithinCrawlParams request)	Delete the queue based on the key in parameter, returns the number of URLs removed this way *
Urlfrontier.Boolean	getActive(Urlfrontier.Local request)	Returns true if the crawl is active, false if it has been deactivated with SetActive(Boolean) *
Urlfrontier.Stats	getStats(Urlfrontier.QueueWithinCrawlParams request)	Return stats for a specific queue or an entire crawl.
Iterator<Urlfrontier.URLInfo>	getURLs(Urlfrontier.GetParams request)	Stream URLs due for fetching from M queues with up to N items per queue *
Urlfrontier.URLItem	getURLStatus(Urlfrontier.URLStatusRequest request)	Get status of a particular URL This does not take into account URL scheduling.
Urlfrontier.StringList	listCrawls(Urlfrontier.Local request)	Return the list of crawls handled by the frontier(s) *
Urlfrontier.StringList	listNodes(Urlfrontier.Empty request)	Return the list of nodes forming the cluster the current node belongs to *
Urlfrontier.QueueList	listQueues(Urlfrontier.Pagination request)	Return a list of queues for a specific crawl.
Iterator<Urlfrontier.URLItem>	listURLs(Urlfrontier.ListUrlParams request)	List all URLs currently in the frontier This does not take into account URL scheduling.
Urlfrontier.Empty	setActive(Urlfrontier.Active request)	De/activate the crawl.
Urlfrontier.Empty	setCrawlLimit(Urlfrontier.CrawlLimitParams request)	Sets crawl limit for domain *
Urlfrontier.Empty	setDelay(Urlfrontier.QueueDelayParams request)	Set a delay from a given queue.
Urlfrontier.Empty	setLogLevel(Urlfrontier.LogLevelParams request)	Overrides the log level for a given package *

Methods inherited from class io.grpc.stub.AbstractBlockingStub

newStub, newStub

Methods inherited from class io.grpc.stub.AbstractStub

getCallOptions, getChannel, withCallCredentials, withChannel, withCompression, withDeadline, withDeadlineAfter, withDeadlineAfter, withExecutor, withInterceptors, withMaxInboundMessageSize, withMaxOutboundMessageSize, withOnReadyThreshold, withOption, withWaitForReady

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

build

protected URLFrontierGrpc.URLFrontierBlockingStub build(io.grpc.Channel channel,
 io.grpc.CallOptions callOptions)

Specified by:

build in class io.grpc.stub.AbstractStub<URLFrontierGrpc.URLFrontierBlockingStub>

listNodes

public Urlfrontier.StringList listNodes(Urlfrontier.Empty request)

 Return the list of nodes forming the cluster the current node belongs to *
 

listCrawls

public Urlfrontier.StringList listCrawls(Urlfrontier.Local request)

 Return the list of crawls handled by the frontier(s) *
 

deleteCrawl

public Urlfrontier.Long deleteCrawl(Urlfrontier.DeleteCrawlMessage request)

 Delete an entire crawl, returns the number of URLs removed this way *
 

listQueues

public Urlfrontier.QueueList listQueues(Urlfrontier.Pagination request)

 Return a list of queues for a specific crawl. Can chose whether to include inactive queues (a queue is active if it has URLs due for fetching);
by default the service will return up to 100 results from offset 0 and exclude inactive queues.*
 

getURLs

public Iterator<Urlfrontier.URLInfo> getURLs(Urlfrontier.GetParams request)

 Stream URLs due for fetching from M queues with up to N items per queue *
 

getStats

public Urlfrontier.Stats getStats(Urlfrontier.QueueWithinCrawlParams request)

 Return stats for a specific queue or an entire crawl. Does not aggregate the stats across different crawlids. *
 

deleteQueue

public Urlfrontier.Long deleteQueue(Urlfrontier.QueueWithinCrawlParams request)

 Delete the queue based on the key in parameter, returns the number of URLs removed this way *
 

blockQueueUntil

public Urlfrontier.Empty blockQueueUntil(Urlfrontier.BlockQueueParams request)

 Block a queue from sending URLs; the argument is the number of seconds of UTC time since Unix epoch
1970-01-01T00:00:00Z. The default value of 0 will unblock the queue. The block will get removed once the time
indicated in argument is reached. This is useful for cases where a server returns a Retry-After for instance. 
 

setActive

public Urlfrontier.Empty setActive(Urlfrontier.Active request)

 De/activate the crawl. GetURLs will not return anything until SetActive is set to true. PutURLs will still take incoming data. *
 

getActive

public Urlfrontier.Boolean getActive(Urlfrontier.Local request)

 Returns true if the crawl is active, false if it has been deactivated with SetActive(Boolean) *
 

setDelay

public Urlfrontier.Empty setDelay(Urlfrontier.QueueDelayParams request)

 Set a delay from a given queue.
No URLs will be obtained via GetURLs for this queue until the number of seconds specified has 
elapsed since the last time URLs were retrieved.
Usually informed by the delay setting of robots.txt.
 

setLogLevel

public Urlfrontier.Empty setLogLevel(Urlfrontier.LogLevelParams request)

 Overrides the log level for a given package *
 

setCrawlLimit

public Urlfrontier.Empty setCrawlLimit(Urlfrontier.CrawlLimitParams request)

 Sets crawl limit for domain *
 

getURLStatus

public Urlfrontier.URLItem getURLStatus(Urlfrontier.URLStatusRequest request)

 Get status of a particular URL 
This does not take into account URL scheduling.
Used to check current status of an URL within the frontier
 

listURLs

public Iterator<Urlfrontier.URLItem> listURLs(Urlfrontier.ListUrlParams request)

 List all URLs currently in the frontier 
This does not take into account URL scheduling.
Used to check current status of all URLs within the frontier
 

countURLs

public Urlfrontier.Long countURLs(Urlfrontier.CountUrlParams request)

 Count URLs currently in the frontier *