A few months ago, Sarah Jamie Lewis released the wonderful OnionScan; a tool for enumerating (and resolving) potential security issues arising from poorly configured Tor Hidden Services. It’s kind of a big deal for people who are interested in that sort of thing.
As cool as OnionScan is, scanning Hidden Services one at a time tends to become rather tedious. Fortunately, Justin Seitz wrote up a nice tutorial on automating OnionScan through a Python wrapper, and being one of those people who are interested in that sort of thing, I set it all up on a dedicated server and left it to run for a few days.
Using Justin’s initial list of 8592 Hidden Services as a starting point, I ended up with 10010 completed scans (which was good) and 10010 distinct JSON files containing the results (which was not so good). “There’s bound to be something interesting in there”, I thought. I could get a rough idea of the state of things by grepping the results files for JSON entries, and even tried throwing EyeWitness at the web and VNC services, but still didn’t really get anywhere. What I really needed was some kind of database.
Introducing onionscan2db…
OnionScan can write its results out to machine-readable JSON files, so parsing them is fairly straightforward. I used Python for no other reason than I like it, and SQLite3 because it’s simple and Python supports it without the need for any additional modules.
The code is probably best described as “functional”. It’s not particularly pretty and there’s definitely room for speeding up the database writes, but it’ll take a take more than 10000 JSON formatted OnionScan results and build an SQLite database that can then be used to do something useful.
The tool is available from GitHub and can be run from the following command
python onionscan2db -d <onionscan-results-directory> -o <output-database>
I’ll try to keep it updated in line with OnionScan, but the code is relatively modular and it shouldn’t be too difficult for anyone else to improve the database structure and import functions as necessary.