Package crawlercommons.urlfrontier
Class URLFrontierGrpc.URLFrontierBlockingStub
java.lang.Object
io.grpc.stub.AbstractStub<S>
io.grpc.stub.AbstractBlockingStub<URLFrontierGrpc.URLFrontierBlockingStub>
crawlercommons.urlfrontier.URLFrontierGrpc.URLFrontierBlockingStub
- Enclosing class:
- URLFrontierGrpc
public static final class URLFrontierGrpc.URLFrontierBlockingStub
extends io.grpc.stub.AbstractBlockingStub<URLFrontierGrpc.URLFrontierBlockingStub>
A stub to allow clients to do synchronous rpc calls to service URLFrontier.
-
Nested Class Summary
Nested classes/interfaces inherited from class io.grpc.stub.AbstractStub
io.grpc.stub.AbstractStub.StubFactory<T extends io.grpc.stub.AbstractStub<T>> -
Method Summary
Modifier and TypeMethodDescription* Block a queue from sending URLs; the argument is the number of seconds of UTC time since Unix epoch 1970-01-01T00:00:00Z.protected URLFrontierGrpc.URLFrontierBlockingStubbuild(io.grpc.Channel channel, io.grpc.CallOptions callOptions) * Delete an entire crawl, returns the number of URLs removed this way ** Delete the queue based on the key in parameter, returns the number of URLs removed this way *getActive(Urlfrontier.Local request) * Returns true if the crawl is active, false if it has been deactivated with SetActive(Boolean) ** Return stats for a specific queue or an entire crawl.getURLs(Urlfrontier.GetParams request) * Stream URLs due for fetching from M queues with up to N items per queue *getURLStatus(Urlfrontier.URLStatusRequest request) * Get status of a particular URL This does not take into account URL scheduling.listCrawls(Urlfrontier.Local request) * Return the list of crawls handled by the frontier(s) *listNodes(Urlfrontier.Empty request) * Return the list of nodes forming the cluster the current node belongs to *listQueues(Urlfrontier.Pagination request) * Return a list of queues for a specific crawl.listURLs(Urlfrontier.ListUrlParams request) * List all URLs currently in the frontier This does not take into account URL scheduling.setActive(Urlfrontier.Active request) * De/activate the crawl.* Sets crawl limit for domain *setDelay(Urlfrontier.QueueDelayParams request) * Set a delay from a given queue.setLogLevel(Urlfrontier.LogLevelParams request) * Overrides the log level for a given package *Methods inherited from class io.grpc.stub.AbstractBlockingStub
newStub, newStubMethods inherited from class io.grpc.stub.AbstractStub
getCallOptions, getChannel, withCallCredentials, withChannel, withCompression, withDeadline, withDeadlineAfter, withExecutor, withInterceptors, withMaxInboundMessageSize, withMaxOutboundMessageSize, withOnReadyThreshold, withOption, withWaitForReady
-
Method Details
-
build
protected URLFrontierGrpc.URLFrontierBlockingStub build(io.grpc.Channel channel, io.grpc.CallOptions callOptions) - Specified by:
buildin classio.grpc.stub.AbstractStub<URLFrontierGrpc.URLFrontierBlockingStub>
-
listNodes
* Return the list of nodes forming the cluster the current node belongs to *
-
listCrawls
* Return the list of crawls handled by the frontier(s) *
-
deleteCrawl
* Delete an entire crawl, returns the number of URLs removed this way *
-
listQueues
* Return a list of queues for a specific crawl. Can chose whether to include inactive queues (a queue is active if it has URLs due for fetching); by default the service will return up to 100 results from offset 0 and exclude inactive queues.*
-
getURLs
* Stream URLs due for fetching from M queues with up to N items per queue *
-
getStats
* Return stats for a specific queue or an entire crawl. Does not aggregate the stats across different crawlids. *
-
deleteQueue
* Delete the queue based on the key in parameter, returns the number of URLs removed this way *
-
blockQueueUntil
* Block a queue from sending URLs; the argument is the number of seconds of UTC time since Unix epoch 1970-01-01T00:00:00Z. The default value of 0 will unblock the queue. The block will get removed once the time indicated in argument is reached. This is useful for cases where a server returns a Retry-After for instance.
-
setActive
* De/activate the crawl. GetURLs will not return anything until SetActive is set to true. PutURLs will still take incoming data. *
-
getActive
* Returns true if the crawl is active, false if it has been deactivated with SetActive(Boolean) *
-
setDelay
* Set a delay from a given queue. No URLs will be obtained via GetURLs for this queue until the number of seconds specified has elapsed since the last time URLs were retrieved. Usually informed by the delay setting of robots.txt.
-
setLogLevel
* Overrides the log level for a given package *
-
setCrawlLimit
* Sets crawl limit for domain *
-
getURLStatus
* Get status of a particular URL This does not take into account URL scheduling. Used to check current status of an URL within the frontier
-
listURLs
* List all URLs currently in the frontier This does not take into account URL scheduling. Used to check current status of all URLs within the frontier
-