Netdelta is a tool for monitoring networks and flagging alerts upon changes in advertised services. Now – I like Python and especially Django, and around 2014 or so i was asked to setup a facility for monitoring for changes in that organisation’s perimeter. After some considerable digging, i found nada, as in nothing, apart from a few half-baked student projects. So i went off and coded Netdelta, and the world has never been the same since.
I guess when i started with Netdelta i didn’t see it as a solution that would be widely popular because i was under the impression you could just do some basic shell scripting with nmap, and ndiff is specifically designed for delta flagging. However what became apparent at an early stage was that timeouts are a problem. A delta will be flagged when a host or service times out – and this happens a lot, even on a gigabit LAN, and it happens even more in public clouds. I built some analytics into Netdelta that looks back over the scan history and data and makes a call on the likelihood of a false positive (red, amber green).
Most organisations i worked with would benefit from this. One classic example i can think of – a trading house that had been on an aggressive M&A spree, maybe it was Black Friday or…? Anyway – they fired some network engineers and hired some new and cheaper ones, exacerbating what was already a poorly managed perimeter scenario. CISO wanted to know what was going off with these Internet facing subnets – enter Netdelta. Unauthorised changes are a problem! I am directly aware of no fewer than 6 incidents that occurred as a result of exposed SSH, SMB (Wannacry), and more recently RDP, and indirectly aware of many more.
Anyway without further waffle, here’s how you get Netdelta up and running. Warning – there are a few moving parts, but if someone wants it in Docker, let me know.
I always go with Ubuntu. The differences between Linux distros are like the differences between mueslis. My build was on 18.04 but its highly likely 19 variants will be just fine.
apt-get update apt-get install curl nmap apache2 python3 python3-pip python3-venv rabbitmq-server mysql-server libapache2-mod-wsgi-py3 git apt-get -y install software-properties-common add-apt-repository ppa:certbot/certbot apt-get install -y python-certbot-apache
Clone the repository from github into your <netdelta root>
git clone https://github.com/SevenStones/netdelta.git
Filesystem
Create the user that will own <netdelta root>
useradd -s /bin/bash -d /home/<user> -m <user>
Create the directory that will host the Netdelta Django project if necessary
Add the user to a suitable group, and strip world permissions from netdelta directories
groupadd <group> usermod -G <group> <user> usermod -G <group> www-data chown -R www-data:<group> /var/www chown -R <user>:<group> <netdelta root>
Make the logs dir, e.g. /logs, and you will need to modify /nd/netdelta_logger.py to point to this location. Note the celery monitor logs go to /var/log/celery/celery-monitor.log …which of course you can change.
Strip world permissions from all netdelta and apache root dirs:
chmod -Rv o-rwx <web root>
chmod -Rv o-rwx <netdelta root>
Virtualenv
The required packages are in requirements.txt, in the root of the git repo. Your virtualenv build with Python 3 goes approximately like this …
python3 -m venv /path/to/new/virtual/environment
You activate thusly: source </path/to/new/virtual/environment/>/bin/activate
Then suck in the requirements as root, remembering to fix permissions after you do this.
pip3 install wheel pip3 install -r <netdelta root>/requirements.txt
You can use whatever supported database you like. MySQL is assumed here.
The Python framework mysqlclient was used with earlier versions of Django and MySQL. but with Python 3 and later Django versions, the word on the street is PyMySQL is the way to go. With this though, it took some trickery to get the Django project up and running; in the form of init.py for the project (<netdelta root>/netdelta/.init.py) and adding a few lines …
import pymysql pymysql.install_as_MySQLdb()
While in virtualenv, and under your netdelta root, add a superuser for the DF
Patch libnmap
Two main mods to the libnmap in usage with Netdelta were necessary. First, with later versions of Celery (>3.1), there was a security issue with “deamonic processes are not allowed to have children”, for which an alternative fork of libnmap fixed the problem. Then we needed to return to Netdelta the process id of the running nmap port scanner process.
cd /opt git clone https://github.com/pyoner/python-libnmap.git cp ./python-libnmap/libnmap/process.py <virtualenv root>/lib/python3.x/site-packages/libnmap/
then patch libnmap to allow Netdelta to kill scanning processes
<netdelta_root>/scripts/fix-libnmap.bash
Change the environment variables to match your install and use the virtualenv name as a parameter
Database Setup
Create a database called netdetla and use whichever encoding snd collation you like.
CREATE DATABASE netdelta CHARACTER SET utf8 COLLATE utf8_general_ci;
Then from <netdelta root> with virtualenv engaged:
python manage.py makemigrations nd
python manage.py migrate
Web Server
I am assuming all you good security pros don’t want to use the development server? Well as you’re only dealing with port scan data then….your call. I’m assuming Apache as a production web server.
You will need to give Apache a stub web root and enable the wsgi module. For the latter i added this to apache2.conf – this gives you some control over the exact version of Python loaded.
LoadModule wsgi_module "/usr/lib/python3.7/site-packages/mod_wsgi/server/mod_wsgi-py37.cpython-37m-i386-linux-gnu.so"
WSGIPythonHome "/usr"
Under the DocumentRoot line in your apache config file, give the pointers for WSGI.
WSGIDaemonProcess <site> python-home=<virtualenv root> python-path=<netdelta root> WSGIProcessGroup <site> WSGIScriptAlias / <netdelta root>/netdelta/wsgi.py Alias /static/ <netdelta root>/netdelta/
Note also you will need to adjust your wsgi.py under <netdelta root>/netdelta/ –
# Add the site-packages of the chosen virtualenv to work with site.addsitedir('<virtualenv root>/lib/python3.7/site-packages')
Celery
From the current shell
in …<virtualenv> ….under <netdelta root>
celery worker -E -A nd -n default -Q default --loglevel=info -B --logfile=<netdelta root>/logs/celery.log
Under systemd (you will almost certainly want to do this) with the root user. The script pointed to by systemd for
systemctl start celery
can have all the environment checking (this isn’t intended to be a tutorial in BASH scripting), but the core of it…
cd <netdelta root>
nohup $VIRTUALENV_DIR/bin/celery worker -E -A nd -n ${SITE} -Q ${SITE} --loglevel=info -B --logfile=${SITE_LOGS}/celery.log >/dev/null 2>&1 &
And then you can put together your own scripts for status, stop, restart.