Enabling `--output-module=json` Returns Approx. Double the Number of IP's Responding

[JustinTimperio]

Describe the bug
It is unclear to me if this is a bug or some very weird async behavior but enabling the JSON module always returns approximately double the number of IPs compared to the normal output. I have tested this on multiple systems across multiple ports and shards with the exact same behavior existing in each case.

CLI Arguments

sudo zmap -p 80 --shards=250 --shard=0 --seed=1337 -o shard_0.csv
cat shard_0.csv | wc -l 
243204

VS

sudo zmap -p 80 --shards=250 --shard=0 --seed=1337 -o shard_0.json --output-module=json
cat shard_0.json | wc -l 
514767

Expected behavior
I think most users would expect that enabling the JSON module would only change the output type. I certainly did not expect enabling it would yield 2x the number of responses.

Environment:

• OS: Arch Linux 5.13.12-arch1-1
• Version: zmap 2.1.1

[dadrian]

This is almost definitely because ZMap is returning packets that are duplicates and/or RSTs in the JSON module (check the relevant fields in the output). Can you try again with latest main or https://github.com/zmap/zmap/releases/tag/v3.0.0-beta1?

@zakird, what behavior did we decide on here?

[JustinTimperio]

@dadrian
So I am testing now on zmap Development Build. Commit e625873 and things seem even more broke here. In zmap 2.1.1 a single shard out of 250 only takes about 2 min, this new one takes 25mins. That does not seem normal or even mildly acceptable speed-wise. I'll let it run for curiosity sake

[dadrian]

things seem even more broke here

Have you looked at your output yet to see if it includes duplicates and/or resets?

only takes about 2 min, this new one takes 25mins

Newer versions of ZMap default to 10K pps, set -r 0 to have unlimited rate, or knock it up to around 100K.

That does not seem normal or even mildly acceptable speed-wise.

What is an "acceptable" speed for this software that I have provided to you for free and am spending my morning helping you debug?

[JustinTimperio]

@dadrian

What is an "acceptable" speed for this software that I have provided to you for free and am spending my morning helping you debug?

Please don't mistake this for hostility. I simply meant this was not even closely comparable to the old version. It was not my intention to be bitting. I did not see anything in the docs about the new default rate limit so a 25x drop in speed was certainly surprising in a bad way.

Have you looked at your output yet to see if it includes duplicates and/or resets?

I have checked for duplicates but as far as I have seen, there have been none. My guess is that something in the background is causing a miss attribution of packets on the backend.

things seem even more broke here

sudo zmap -p 80 --shards=250 --shard=0 --seed=1337 -o shard_0.json --output-module=json
cat shard_0.json | wc -l 
721554

VS

sudo zmap -p 80 --shards=250 --shard=0 --seed=1337 -o shard_0.csv
cat shard_0.csv | wc -l 
246304

[JustinTimperio]

I would also note that it seems to be exclusive to the --output-module=json flag. The CSV output seems to be consistent across versions

[dadrian]

Just to confirm, you checked something along the lines of cat shard_0.json | jq -cr '.saddr' | sort | uniq -d | wc -l?

Also, can you post the output of info-level logs or higher (-v) from startup, prior to sending the first packet?

[JustinTimperio]

@dadrian So when I discovered the bug, the entries were in a database so I just checked for duplicate ips which only came in about 3%(pretty average for what we scan). I'll do a quick double-check in bash just to be safe though.

[JustinTimperio]

Okay, so I just double-checked this, and here are the results.

cat shard_0.csv | sort | uniq -d | wc -l
0

VS

cat shard_0.json | jq -cr '.saddr' | sort | uniq -d | wc -
4473

Looks like some dups but certainly not enough to cause the discrepancy being seen here.

Logs show nothing suspect but I'll pass them on just in case.
JSON_LOG.txt
CSV_LOG.txt

[dadrian]

CSV

Aug 31 00:01:10.766 [DEBUG] recv: duplicate responses will be excluded from output
Aug 31 00:01:10.767 [DEBUG] recv: unsuccessful responses will be excluded from output

JSON

Aug 31 00:04:36.089 [DEBUG] recv: duplicate responses will be included in output
Aug 31 00:04:36.090 [DEBUG] recv: unsuccessful responses will be included in output

If you don't specify an output module, exclude duplicates and unsuccessful responses (RSTs), skip the header, and only output saddr, so that ZMap output can be easily piped into another program.

If you do specify an output module, whether it's JSON or CSV, we don't automatically do any of that. We should probably increase the verbosity of the log messages explaining this.

You're getting "extra" results in the JSON case because it includes RSTs and duplicates, not just unique SYN ACKs