Package crawlercommons.urlfrontier
Class URLFrontierGrpc.URLFrontierBlockingStub
- java.lang.Object
-
- io.grpc.stub.AbstractStub<S>
-
- io.grpc.stub.AbstractBlockingStub<URLFrontierGrpc.URLFrontierBlockingStub>
-
- crawlercommons.urlfrontier.URLFrontierGrpc.URLFrontierBlockingStub
-
- Enclosing class:
- URLFrontierGrpc
public static final class URLFrontierGrpc.URLFrontierBlockingStub extends io.grpc.stub.AbstractBlockingStub<URLFrontierGrpc.URLFrontierBlockingStub>
-
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description Urlfrontier.EmptyblockQueueUntil(Urlfrontier.BlockQueueParams request)* Block a queue from sending URLs; the argument is the number of seconds of UTC time since Unix epoch 1970-01-01T00:00:00Z.protected URLFrontierGrpc.URLFrontierBlockingStubbuild(io.grpc.Channel channel, io.grpc.CallOptions callOptions)Urlfrontier.LongdeleteCrawl(Urlfrontier.DeleteCrawlMessage request)* Delete an entire crawl, returns the number of URLs removed this way *Urlfrontier.LongdeleteQueue(Urlfrontier.QueueWithinCrawlParams request)* Delete the queue based on the key in parameter, returns the number of URLs removed this way *Urlfrontier.BooleangetActive(Urlfrontier.Local request)* Returns true if the crawl is active, false if it has been deactivated with SetActive(Boolean) *Urlfrontier.StatsgetStats(Urlfrontier.QueueWithinCrawlParams request)* Return stats for a specific queue or an entire crawl.Iterator<Urlfrontier.URLInfo>getURLs(Urlfrontier.GetParams request)* Stream URLs due for fetching from M queues with up to N items per queue *Urlfrontier.StringListlistCrawls(Urlfrontier.Local request)* Return the list of crawls handled by the frontier(s) *Urlfrontier.StringListlistNodes(Urlfrontier.Empty request)* Return the list of nodes forming the cluster the current node belongs to *Urlfrontier.QueueListlistQueues(Urlfrontier.Pagination request)* Return a list of queues for a specific crawl.Urlfrontier.EmptysetActive(Urlfrontier.Active request)* De/activate the crawl.Urlfrontier.EmptysetDelay(Urlfrontier.QueueDelayParams request)* Set a delay from a given queue.Urlfrontier.EmptysetLogLevel(Urlfrontier.LogLevelParams request)* Overrides the log level for a given package *
-
-
-
Method Detail
-
build
protected URLFrontierGrpc.URLFrontierBlockingStub build(io.grpc.Channel channel, io.grpc.CallOptions callOptions)
- Specified by:
buildin classio.grpc.stub.AbstractStub<URLFrontierGrpc.URLFrontierBlockingStub>
-
listNodes
public Urlfrontier.StringList listNodes(Urlfrontier.Empty request)
* Return the list of nodes forming the cluster the current node belongs to *
-
listCrawls
public Urlfrontier.StringList listCrawls(Urlfrontier.Local request)
* Return the list of crawls handled by the frontier(s) *
-
deleteCrawl
public Urlfrontier.Long deleteCrawl(Urlfrontier.DeleteCrawlMessage request)
* Delete an entire crawl, returns the number of URLs removed this way *
-
listQueues
public Urlfrontier.QueueList listQueues(Urlfrontier.Pagination request)
* Return a list of queues for a specific crawl. Can chose whether to include inactive queues (a queue is active if it has URLs due for fetching); by default the service will return up to 100 results from offset 0 and exclude inactive queues.*
-
getURLs
public Iterator<Urlfrontier.URLInfo> getURLs(Urlfrontier.GetParams request)
* Stream URLs due for fetching from M queues with up to N items per queue *
-
getStats
public Urlfrontier.Stats getStats(Urlfrontier.QueueWithinCrawlParams request)
* Return stats for a specific queue or an entire crawl. Does not aggregate the stats across different crawlids. *
-
deleteQueue
public Urlfrontier.Long deleteQueue(Urlfrontier.QueueWithinCrawlParams request)
* Delete the queue based on the key in parameter, returns the number of URLs removed this way *
-
blockQueueUntil
public Urlfrontier.Empty blockQueueUntil(Urlfrontier.BlockQueueParams request)
* Block a queue from sending URLs; the argument is the number of seconds of UTC time since Unix epoch 1970-01-01T00:00:00Z. The default value of 0 will unblock the queue. The block will get removed once the time indicated in argument is reached. This is useful for cases where a server returns a Retry-After for instance.
-
setActive
public Urlfrontier.Empty setActive(Urlfrontier.Active request)
* De/activate the crawl. GetURLs will not return anything until SetActive is set to true. PutURLs will still take incoming data. *
-
getActive
public Urlfrontier.Boolean getActive(Urlfrontier.Local request)
* Returns true if the crawl is active, false if it has been deactivated with SetActive(Boolean) *
-
setDelay
public Urlfrontier.Empty setDelay(Urlfrontier.QueueDelayParams request)
* Set a delay from a given queue. No URLs will be obtained via GetURLs for this queue until the number of seconds specified has elapsed since the last time URLs were retrieved. Usually informed by the delay setting of robots.txt.
-
setLogLevel
public Urlfrontier.Empty setLogLevel(Urlfrontier.LogLevelParams request)
* Overrides the log level for a given package *
-
-