Package crawlercommons.urlfrontier

Class URLFrontierGrpc.URLFrontierFutureStub

java.lang.Object
io.grpc.stub.AbstractStub<S>
io.grpc.stub.AbstractFutureStub<URLFrontierGrpc.URLFrontierFutureStub>
crawlercommons.urlfrontier.URLFrontierGrpc.URLFrontierFutureStub
Enclosing class:
URLFrontierGrpc
public static final class URLFrontierGrpc.URLFrontierFutureStub extends io.grpc.stub.AbstractFutureStub<URLFrontierGrpc.URLFrontierFutureStub>
A stub to allow clients to do ListenableFuture-style rpc calls to service URLFrontier.

  • Method Details

    • build

      protected URLFrontierGrpc.URLFrontierFutureStub build(io.grpc.Channel channel, io.grpc.CallOptions callOptions)
      Specified by:
      build in class io.grpc.stub.AbstractStub<URLFrontierGrpc.URLFrontierFutureStub>

    • listNodes

      public com.google.common.util.concurrent.ListenableFuture<Urlfrontier.StringList> listNodes(Urlfrontier.Empty request)
       * Return the list of nodes forming the cluster the current node belongs to *

    • listCrawls

      public com.google.common.util.concurrent.ListenableFuture<Urlfrontier.StringList> listCrawls(Urlfrontier.Local request)
       * Return the list of crawls handled by the frontier(s) *

    • deleteCrawl

      public com.google.common.util.concurrent.ListenableFuture<Urlfrontier.Long> deleteCrawl(Urlfrontier.DeleteCrawlMessage request)
       * Delete an entire crawl, returns the number of URLs removed this way *

    • listQueues

      public com.google.common.util.concurrent.ListenableFuture<Urlfrontier.QueueList> listQueues(Urlfrontier.Pagination request)
       * Return a list of queues for a specific crawl. Can chose whether to include inactive queues (a queue is active if it has URLs due for fetching);
 by default the service will return up to 100 results from offset 0 and exclude inactive queues.*

    • getStats

      public com.google.common.util.concurrent.ListenableFuture<Urlfrontier.Stats> getStats(Urlfrontier.QueueWithinCrawlParams request)
       * Return stats for a specific queue or an entire crawl. Does not aggregate the stats across different crawlids. *

    • deleteQueue

      public com.google.common.util.concurrent.ListenableFuture<Urlfrontier.Long> deleteQueue(Urlfrontier.QueueWithinCrawlParams request)
       * Delete the queue based on the key in parameter, returns the number of URLs removed this way *

    • blockQueueUntil

      public com.google.common.util.concurrent.ListenableFuture<Urlfrontier.Empty> blockQueueUntil(Urlfrontier.BlockQueueParams request)
       * Block a queue from sending URLs; the argument is the number of seconds of UTC time since Unix epoch
 1970-01-01T00:00:00Z. The default value of 0 will unblock the queue. The block will get removed once the time
 indicated in argument is reached. This is useful for cases where a server returns a Retry-After for instance.

    • setActive

      public com.google.common.util.concurrent.ListenableFuture<Urlfrontier.Empty> setActive(Urlfrontier.Active request)
       * De/activate the crawl. GetURLs will not return anything until SetActive is set to true. PutURLs will still take incoming data. *

    • getActive

      public com.google.common.util.concurrent.ListenableFuture<Urlfrontier.Boolean> getActive(Urlfrontier.Local request)
       * Returns true if the crawl is active, false if it has been deactivated with SetActive(Boolean) *

    • setDelay

      public com.google.common.util.concurrent.ListenableFuture<Urlfrontier.Empty> setDelay(Urlfrontier.QueueDelayParams request)
       * Set a delay from a given queue.
 No URLs will be obtained via GetURLs for this queue until the number of seconds specified has
 elapsed since the last time URLs were retrieved.
 Usually informed by the delay setting of robots.txt.

    • setLogLevel

      public com.google.common.util.concurrent.ListenableFuture<Urlfrontier.Empty> setLogLevel(Urlfrontier.LogLevelParams request)
       * Overrides the log level for a given package *

    • setCrawlLimit

      public com.google.common.util.concurrent.ListenableFuture<Urlfrontier.Empty> setCrawlLimit(Urlfrontier.CrawlLimitParams request)
       * Sets crawl limit for domain *

    • getURLStatus

      public com.google.common.util.concurrent.ListenableFuture<Urlfrontier.URLItem> getURLStatus(Urlfrontier.URLStatusRequest request)
       * Get status of a particular URL
 This does not take into account URL scheduling.
 Used to check current status of an URL within the frontier