crawlercommons.urlfrontier

Class URLFrontierGrpc.URLFrontierBlockingStub

java.lang.Object

io.grpc.stub.AbstractStub<S>

io.grpc.stub.AbstractBlockingStub<URLFrontierGrpc.URLFrontierBlockingStub>

crawlercommons.urlfrontier.URLFrontierGrpc.URLFrontierBlockingStub

Enclosing class:

URLFrontierGrpc

public static final class URLFrontierGrpc.URLFrontierBlockingStub
extends io.grpc.stub.AbstractBlockingStub<URLFrontierGrpc.URLFrontierBlockingStub>

Nested Class Summary

Nested classes/interfaces inherited from class io.grpc.stub.AbstractStub

io.grpc.stub.AbstractStub.StubFactory<T extends io.grpc.stub.AbstractStub<T>>

Method Summary

Modifier and Type	Method and Description
Urlfrontier.Empty	blockQueueUntil(Urlfrontier.BlockQueueParams request) Block a queue from sending URLs; the argument is the number of seconds of UTC time since Unix epoch 1970-01-01T00:00:00Z.
protected URLFrontierGrpc.URLFrontierBlockingStub	build(io.grpc.Channel channel, io.grpc.CallOptions callOptions)
Urlfrontier.Integer	deleteQueue(Urlfrontier.String request) Delete the queue based on the key in parameter, returns the number of URLs removed this way *
Urlfrontier.Boolean	getActive(Urlfrontier.Empty request) Returns true if the crawl is active, false if it has been deactivated with SetActive(Boolean) *
Urlfrontier.Stats	getStats(Urlfrontier.String request) Return stats for a specific queue or the whole crawl if the value if empty or null *
java.util.Iterator<Urlfrontier.URLInfo>	getURLs(Urlfrontier.GetParams request) Stream URLs due for fetching from M queues with up to N items per queue *
Urlfrontier.QueueList	listQueues(Urlfrontier.Pagination request) Return the names of up to N active queues a queue is active if it has URLs due for fetching; by default the service will return up to 100 results from offset 0 *
Urlfrontier.Empty	setActive(Urlfrontier.Boolean request) De/activate the crawl.
Urlfrontier.Empty	setDelay(Urlfrontier.QueueDelayParams request) Set a delay from a given queue.

Methods inherited from class io.grpc.stub.AbstractBlockingStub

newStub, newStub

Methods inherited from class io.grpc.stub.AbstractStub

getCallOptions, getChannel, withCallCredentials, withChannel, withCompression, withDeadline, withDeadlineAfter, withExecutor, withInterceptors, withMaxInboundMessageSize, withMaxOutboundMessageSize, withOption, withWaitForReady

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

build

protected URLFrontierGrpc.URLFrontierBlockingStub build(io.grpc.Channel channel,
                                                        io.grpc.CallOptions callOptions)

Specified by:

build in class io.grpc.stub.AbstractStub<URLFrontierGrpc.URLFrontierBlockingStub>

listQueues

public Urlfrontier.QueueList listQueues(Urlfrontier.Pagination request)

 Return the names of up to N active queues
a queue is active if it has URLs due for fetching; by default the service will return up to 100 results from offset 0 *
 

getURLs

public java.util.Iterator<Urlfrontier.URLInfo> getURLs(Urlfrontier.GetParams request)

 Stream URLs due for fetching from M queues with up to N items per queue *
 

getStats

public Urlfrontier.Stats getStats(Urlfrontier.String request)

 Return stats for a specific queue or the whole crawl if the value if empty or null *
 

deleteQueue

public Urlfrontier.Integer deleteQueue(Urlfrontier.String request)

 Delete the queue based on the key in parameter, returns the number of URLs removed this way *
 

blockQueueUntil

public Urlfrontier.Empty blockQueueUntil(Urlfrontier.BlockQueueParams request)

 Block a queue from sending URLs; the argument is the number of seconds of UTC time since Unix epoch
1970-01-01T00:00:00Z. The default value of 0 will unblock the queue. The block will get removed once the time
indicated in argument is reached. This is useful for cases where a server returns a Retry-After for instance. 
 

setActive

public Urlfrontier.Empty setActive(Urlfrontier.Boolean request)

 De/activate the crawl. GetURLs will not return anything until SetActive is set to true. PutURLs will still take incoming data. *
 

getActive

public Urlfrontier.Boolean getActive(Urlfrontier.Empty request)

 Returns true if the crawl is active, false if it has been deactivated with SetActive(Boolean) *
 

setDelay

public Urlfrontier.Empty setDelay(Urlfrontier.QueueDelayParams request)

 Set a delay from a given queue.
No URLs will be obtained via GetURLs for this queue until the number of seconds specified has 
elapsed since the last time URLs were retrieved.
Usually informed by the delay setting of robots.txt.