NAME

       waymore - Tool to discover extensive data from online archives

SYNOPSIS

          waymore [-h] [-i INPUT] [-n] [-mode {U,R,B}] [-oU OUTPUT_URLS] [-oR OUTPUT_RESPONSES] [-f] [-fc FC] [-mc MC] [-l <signed integer>] [-from <yyyyMMddhhmmss>] [-to <yyyyMMddhhmmss>]
                  [-ci {h,d,m,none}] [-ra REGEX_AFTER] [-url-filename] [-xwm] [-xcc] [-xav] [-xus] [-xvt] [-lcc LCC] [-lcy LCY] [-t <seconds>] [-p <integer>] [-r RETRIES] [-m <integer>]
                  [-ko [KEYWORDS_ONLY]] [-lr LIMIT_REQUESTS] [-ow] [-nlf] [-c CONFIG] [-wrlr WAYBACK_RATE_LIMIT_RETRY] [-urlr URLSCAN_RATE_LIMIT_RETRY] [-co] [-nd] [-v] [--version]

DESCRIPTION

       waymore  is a versatile tool designed to extract comprehensive information from various sources including
       the Wayback Machine, Common Crawl, Alien Vault OTX, URLScan, and VirusTotal. Whether you're searching for
       historical web data or analyzing security threats,  waymore  provides  a  seamless  experience  with  its
       intuitive interface and extensive features.

OPTIONS

       -h, --help:
              Display  command  usage  and options. Provides quick access to comprehensive assistance, including
              detailed explanations of available options.

       -i INPUT, --input INPUT:
              The target domain (or file of domains) to find links for. This can be a domain only, or  a  domain
              with a specific path.  If it is a domain only to get everything for that domain, don't prefix with
              "www."

       -n, --no-subs:
              Don't  include subdomains of the target domain (only used if input is not a domain with a specific
              path).

       -mode {U,R,B}:
              The mode to run: U (retrieve URLs only), R (download Responses only) or B (Both).

       -oU OUTPUT_URLS, --output-urls OUTPUT_URLS:
              The file to save the Links output to, including path if necessary. If the "-oR"  argument  is  not
              passed,  a "results" directory will be created in the path specified by the DEFAULT_OUTPUT_DIR key
              in config.yml file (typically defaults to "~/.config/waymore/").  Within that, a directory will be
              created with target domain (or domain with path) passed with "-i" (or for  each  line  of  a  file
              passed with "-i").

       -oR OUTPUT_RESPONSES, --output-responses OUTPUT_RESPONSES:
              The  directory  to save the response output files to, including path if necessary. If the argument
              is  not  passed,  a  "results"  directory  will  be  created  in  the  path   specified   by   the
              DEFAULT_OUTPUT_DIR  key  in  config.yml  file (typically defaults to "~/.config/waymore/"). Within
              that, a directory will be created with target domain (or domain with path) passed  with  "-i"  (or
              for each line of a file passed with "-i").

       -f, --filter-responses-only:
              The  initial  links  from Wayback Machine will not be filtered (MIME Type and Response Code), only
              the responses that are downloaded, e.g. it maybe useful to still see all available paths from  the
              links even if you don't want to check the content.

       -fc FC:
              Filter HTTP status codes for retrieved URLs and responses. Comma separated list of codes (default:
              the  FILTER_CODE  values  from  config.yml).   Passing  this argument will override the value from
              config.yml

       -mc MC:
              Only Match HTTP status codes for retrieved URLs and responses.  Comma  separated  list  of  codes.
              Passing this argument overrides the config FILTER_CODE and -fc.

       -l <signed integer>, --limit <signed integer>:
              How  many  responses  will  be  saved  (if -mode is R or B). A positive value will get the first N
              results, a negative value will will get the last N results.  A value of 0 will get  ALL  responses
              (default: 5000)

       -from <yyyyMMddhhmmss>, --from-date <yyyyMMddhhmmss>:
              What  date to get responses from. If not specified it will get from the earliest possible results.
              A partial value can be passed, e.g. 2016, 201805, etc.

       -to <yyyyMMddhhmmss>, --to-date <yyyyMMddhhmmss>:
              What date to get responses to. If not specified it will get to  the  latest  possible  results.  A
              partial value can be passed, e.g. 2016, 201805, etc.

       -ci {h,d,m,none}, --capture-interval {h,d,m,none}:
              Filters  the  search  on Wayback Machine (archive.org) to only get at most 1 capture per hour (h),
              day (d) or month (m). This filter is used for responses only.  The default is 'd' but can also  be
              set to 'none' to not filter anything and get all responses.

       -ra REGEX_AFTER, --regex-after REGEX_AFTER:
              RegEx  for  filtering  purposes  against links found all sources of URLs AND responses downloaded.
              Only positive matches will be output.

       -url-filename:
              Set the file name of downloaded responses to the URL that generated  the  response,  otherwise  it
              will  be  set  to  the  hash value of the response.  Using the hash value means multiple URLs that
              generated the same response will only result in one file being saved for that response.

       -xwm:  Exclude checks for links from Wayback Machine (archive.org)

       -xcc:  Exclude checks for links from commoncrawl.org

       -xav:  Exclude checks for links from alienvault.com

       -xus:  Exclude checks for links from urlscan.io

       -xvt:  Exclude checks for links from virustotal.com

       -lcc LCC:
              Limit the number of Common Crawl index collections searched, e.g. '-lcc 10' will just  search  the
              latest  10 collections (default: 3).  As of July 2023 there are currently 95 collections.  Setting
              to 0 (default) will search ALL collections. If you don't want to search Common Crawl at  all,  use
              the -xcc option.

       -lcy LCY:
              Limit  the  number  of  Common Crawl index collections searched by the year of the index data. The
              earliest index has data from 2008.  Setting to 0 (default) will search  collections  or  any  year
              (but  in  conjunction  with  -lcc).  For example, if you are only interested in data from 2015 and
              after, pass -lcy 2015. If you don't want to search Common Crawl at all, use the -xcc option.

       -t <seconds>, --timeout <seconds>:
              This is for archived responses only! How many seconds to wait for the server to send  data  before
              giving up (default: 30 seconds)

       -p <integer>, --processes <integer>:
              Basic  multithreading  is  done when getting requests for a file of URLs. This argument determines
              the number of processes (threads) used (default: 1)

       -r RETRIES, --retries RETRIES:
              The number of retries for requests that get connection error or rate limited (default: 1).

       -m <integer>, --memory-threshold <integer>:
              The memory threshold percentage. If the machines memory goes above the threshold, the program will
              be stopped and ended gracefully before running out of memory (default: 95)

       -ko [KEYWORDS_ONLY], --keywords-only [KEYWORDS_ONLY]:
              Only return links and responses that contain keywords that you are interested in. This can  reduce
              the  time it takes to get results.  If you provide the flag with no value, Keywords are taken from
              the comma separated list in the "config.yml" file with the "FILTER_KEYWORDS"  key,  otherwise  you
              can  pass  an  specific Regex value to use, e.g. -ko "admin" to only get links containing the word
              admin, or -ko ".js(?|$)" to only get JS files. The Regex check is NOT case sensitive.

       -lr LIMIT_REQUESTS, --limit-requests LIMIT_REQUESTS:
              Limit the number of requests that will be made when getting links  from  a  source  (this  doesn't
              apply  to  Common  Crawl).  Some targets can return a huge amount of requests needed that are just
              not feasible to get, so this can be used to manage that situation. This defaults to 0 (Zero) which
              means there is no limit.

       -ow, --output-overwrite:
              If the URL output file (default waymore.txt) already exists, it will  be  overwritten  instead  of
              being appended to.

       -nlf, --new-links-file:
              If  this  argument  is  passed,  a  .new file will also be written that will contain links for the
              latest run.

       -c CONFIG, --config CONFIG:
              Path to the YML config file. If not passed, it looks for file 'config.yml' in the  same  directory
              as runtime file 'waymore.py'.

       -wrlr WAYBACK_RATE_LIMIT_RETRY, --wayback-rate-limit-retry WAYBACK_RATE_LIMIT_RETRY:
              The  number  of  minutes  the  user  wants  to  wait  for  a  rate  limit pause on Watback Machine
              (archive.org) instead of stopping with a 429 error (default: 3).

       -urlr URLSCAN_RATE_LIMIT_RETRY, --urlscan-rate-limit-retry URLSCAN_RATE_LIMIT_RETRY:
              The number of minutes the user wants to wait for a rate  limit  pause  on  URLScan.io  instead  of
              stopping with a 429 error (default: 1).

       -co, --check-only:
              This will make a few minimal requests to show you how many requests, and roughly how long it could
              take, to get URLs from the sources and downloaded responses from Wayback Machine.

       -nd, --notify-discord:
              Whether  to  send a notification to Discord when waymore completes. It requires WEBHOOK_DISCORD to
              be provided in the config.yml file.

       -v, --verbose:
              Verbose output

       --version:
              Show version number

EXAMPLES

       Common usage:

       • Example 1:

       Just get the URLs from all sources for redbull.com (-mode U  is  just  for  URLs,  so  no  responses  are
       downloaded):

          $ waymore -i redbull.com -mode U

       The   URLs   are   saved   in   the   same   path   as  config.yml  (typically  ~/.config/waymore)  under
       results/redbull.com/waymore.txt

       • Example 2:

       Get ALL the URLs from Wayback for redbull.com (no filters are applied in mode U with -f, and no URLs  are
       retrieved  from  Common  Crawl,  Alien Vault, URLScan and Virus Total, because -xcc, -xav, -xus, -xvt are
       passed respectively). Save the FIRST 200 responses that are found starting from 2022 (-l 200 -from 2022):

          $ waymore -i redbull.com -f -xcc -xav -xus -xvt -l 200 -from 2022

       • Example 3:

       You can pipe waymore to other tools. Any errors are sent to stderr  and  any  links  found  are  sent  to
       stdout.  The  output  file  is  still  created  in addition to the links being piped to the next program.
       However, archived responses are not piped to the next program, but they are still written to  files.  For
       example:

          $ waymore -i redbull.com -mode U | unfurl keys | sort -u

       You can also pass the input through stdin instead of -i:

          $ cat redbull_subs.txt | waymore

       • Example 4:

       Sometimes you may just want to check how many requests, and how long waymore is likely to take if you ran
       it  for  a  particular  domain.   You  can  do  a quick check by using the -co/--check-only argument. For
       example:

          $ waymore -i redbull.com --check-only

AUTHOR

       Aquila Macedo <aquilamacedo@riseup.net>

COPYRIGHT

       Expat

                                                   2024-03-22                                         WAYMORE(1)