crawlercommons.urlfrontier

Class URLFrontierGrpc.URLFrontierImplBase

java.lang.Object

crawlercommons.urlfrontier.URLFrontierGrpc.URLFrontierImplBase

All Implemented Interfaces:

io.grpc.BindableService

Enclosing class:

URLFrontierGrpc

public abstract static class URLFrontierGrpc.URLFrontierImplBase
extends java.lang.Object
implements io.grpc.BindableService

Constructor Summary

Constructors

Constructor and Description
URLFrontierImplBase()

Method Summary

Modifier and Type	Method and Description
io.grpc.ServerServiceDefinition	bindService()
void	blockQueueUntil(Urlfrontier.BlockQueueParams request, io.grpc.stub.StreamObserver<Urlfrontier.Empty> responseObserver) Block a queue from sending URLs; the argument is the number of seconds of UTC time since Unix epoch 1970-01-01T00:00:00Z.
void	deleteQueue(Urlfrontier.String request, io.grpc.stub.StreamObserver<Urlfrontier.Integer> responseObserver) Delete the queue based on the key in parameter, returns the number of URLs removed this way *
void	getStats(Urlfrontier.String request, io.grpc.stub.StreamObserver<Urlfrontier.Stats> responseObserver) Return stats for a specific queue or the whole crawl if the value if empty or null *
void	getURLs(Urlfrontier.GetParams request, io.grpc.stub.StreamObserver<Urlfrontier.URLInfo> responseObserver) Stream URLs due for fetching from M queues with up to N items per queue *
void	listQueues(Urlfrontier.Integer request, io.grpc.stub.StreamObserver<Urlfrontier.StringList> responseObserver) Return the names of up to N active queues a queue is active if it has URLs due for fetching; the default value of 0 sets no limit to the number of queue names to return *
io.grpc.stub.StreamObserver<Urlfrontier.URLItem>	putURLs(io.grpc.stub.StreamObserver<Urlfrontier.String> responseObserver) Push URL items to the server; they get created (if they don't already exist) in case of DiscoveredURLItems or updated if KnownURLItems *
void	setActive(Urlfrontier.Boolean request, io.grpc.stub.StreamObserver<Urlfrontier.Empty> responseObserver) De/activate the crawl.
void	setDelay(Urlfrontier.QueueDelayParams request, io.grpc.stub.StreamObserver<Urlfrontier.Empty> responseObserver) Set a delay from a given queue.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

URLFrontierImplBase

public URLFrontierImplBase()

Method Detail

listQueues

public void listQueues(Urlfrontier.Integer request,
                       io.grpc.stub.StreamObserver<Urlfrontier.StringList> responseObserver)

 Return the names of up to N active queues
a queue is active if it has URLs due for fetching; the default value of 0 sets no limit to the number of queue names to return *
 

getURLs

public void getURLs(Urlfrontier.GetParams request,
                    io.grpc.stub.StreamObserver<Urlfrontier.URLInfo> responseObserver)

 Stream URLs due for fetching from M queues with up to N items per queue *
 

putURLs

public io.grpc.stub.StreamObserver<Urlfrontier.URLItem> putURLs(io.grpc.stub.StreamObserver<Urlfrontier.String> responseObserver)

 Push URL items to the server; they get created (if they don't already exist) in case of DiscoveredURLItems or updated if KnownURLItems *
 

getStats

public void getStats(Urlfrontier.String request,
                     io.grpc.stub.StreamObserver<Urlfrontier.Stats> responseObserver)

 Return stats for a specific queue or the whole crawl if the value if empty or null *
 

deleteQueue

public void deleteQueue(Urlfrontier.String request,
                        io.grpc.stub.StreamObserver<Urlfrontier.Integer> responseObserver)

 Delete the queue based on the key in parameter, returns the number of URLs removed this way *
 

blockQueueUntil

public void blockQueueUntil(Urlfrontier.BlockQueueParams request,
                            io.grpc.stub.StreamObserver<Urlfrontier.Empty> responseObserver)

 Block a queue from sending URLs; the argument is the number of seconds of UTC time since Unix epoch
1970-01-01T00:00:00Z. The default value of 0 will unblock the queue. The block will get removed once the time
indicated in argument is reached. This is useful for cases where a server returns a Retry-After for instance. 
 

setActive

public void setActive(Urlfrontier.Boolean request,
                      io.grpc.stub.StreamObserver<Urlfrontier.Empty> responseObserver)

 De/activate the crawl. GetURLs will not return anything until SetActive is set to true. PutURLs will still take incoming data. *
 

setDelay

public void setDelay(Urlfrontier.QueueDelayParams request,
                     io.grpc.stub.StreamObserver<Urlfrontier.Empty> responseObserver)

 Set a delay from a given queue.
No URLs will be obtained via GetURLs for this queue until the number of seconds specified has 
elapsed since the last time URLs were retrieved.
Usually informed by the delay setting of robots.txt.
 

bindService

public final io.grpc.ServerServiceDefinition bindService()

Specified by:

bindService in interface io.grpc.BindableService