Package crawlercommons.urlfrontier

Interface Urlfrontier.URLInfoOrBuilder

  • All Superinterfaces:
    com.google.protobuf.MessageLiteOrBuilder, com.google.protobuf.MessageOrBuilder
    All Known Implementing Classes:
    Urlfrontier.URLInfo, Urlfrontier.URLInfo.Builder
    Enclosing class:
    Urlfrontier
    public static interface Urlfrontier.URLInfoOrBuilder
extends com.google.protobuf.MessageOrBuilder

    • Method Detail

      • getUrl

        String getUrl()
         * URL *
        string url = 1;
        Returns:
        The url.

      • getUrlBytes

        com.google.protobuf.ByteString getUrlBytes()
         * URL *
        string url = 1;
        Returns:
        The bytes for url.

      • getKey

        String getKey()
         * The key is used to put the URLs into queues, the value can be anything set by the client but would typically be the hostname,
 domain name or IP or the URL. If not set, the service will use a sensible default like hostname.
        string key = 2;
        Returns:
        The key.

      • getKeyBytes

        com.google.protobuf.ByteString getKeyBytes()
         * The key is used to put the URLs into queues, the value can be anything set by the client but would typically be the hostname,
 domain name or IP or the URL. If not set, the service will use a sensible default like hostname.
        string key = 2;
        Returns:
        The bytes for key.

      • getMetadataCount

        int getMetadataCount()
         *
 Arbitrary key / values stored alongside the URL. Can be anything needed by the crawler like http status, source URL etc...
        map<string, .urlfrontier.StringList> metadata = 3;

      • containsMetadata

        boolean containsMetadata​(String key)
         *
 Arbitrary key / values stored alongside the URL. Can be anything needed by the crawler like http status, source URL etc...
        map<string, .urlfrontier.StringList> metadata = 3;

      • getMetadataMap

        Map<String,​Urlfrontier.StringList> getMetadataMap()
         *
 Arbitrary key / values stored alongside the URL. Can be anything needed by the crawler like http status, source URL etc...
        map<string, .urlfrontier.StringList> metadata = 3;

      • getMetadataOrDefault

        Urlfrontier.StringList getMetadataOrDefault​(String key,
                                            Urlfrontier.StringList defaultValue)
         *
 Arbitrary key / values stored alongside the URL. Can be anything needed by the crawler like http status, source URL etc...
        map<string, .urlfrontier.StringList> metadata = 3;

      • getMetadataOrThrow

        Urlfrontier.StringList getMetadataOrThrow​(String key)
         *
 Arbitrary key / values stored alongside the URL. Can be anything needed by the crawler like http status, source URL etc...
        map<string, .urlfrontier.StringList> metadata = 3;

      • getCrawlID

        String getCrawlID()
         * crawl ID *
        string crawlID = 4;
        Returns:
        The crawlID.

      • getCrawlIDBytes

        com.google.protobuf.ByteString getCrawlIDBytes()
         * crawl ID *
        string crawlID = 4;
        Returns:
        The bytes for crawlID.