Author Topic: RF Noise Monitoring system on a Raspberry Pi  (Read 2745 times)

Anton Janovsky (ZR6AIC)

  • Administrator
  • Jr. Member
  • *****
  • Karma: +0/-0
RF Noise Monitoring system on a Raspberry Pi
« on: January 30, 2018, 11:36:00 am »
 RF Noise Monitoring system on a Raspberry Pi
(This system will record RF signals and upload the results to the Server where trending graphs are created)
This system is now also available on Ubuntu live memory stick.
Link to Videohttps://youtu.be/ehpsF19SaA8
Here link to live system operating at http://zr6aic.giga.co.za/hf_noise/graph/1Mhz_Power.php and http://zr6aic.giga.co.za/vhf_heatmap/images/

This system software is available pre installed on a 16GB SD card from Giga Technology

Raspberry Pi OS with HF monitoring software pre installed

What do I get on the software?

I system that will record 1Mhz band with samples every 2 minutes generating Graphs and Heat-map of the hole 1 to 30Mhz band.

What hardware do I need?

The  100khz to 1.7Ghz HF dongle as in picture below Screenshot of Graph monitoring.

url of link on system http://localhost/hf_noise/graph/1Mhz_Power.php

Link to live system is here http://zr6aic.giga.co.za/hf_noise/graph/1Mhz_Power.php

Screen shot of the HF Monitoring system
How does this system work?

Block Diagram of the HF Noise Monitoring system.



How dose the HF Noise Monitoring system work

    The HF noise monitoring system takes a 1Mhz bandwidth samples every 2 minutes using the rtl_power utility and save the measurements in a CVS fie.
    The 2 minute scheduling is done with a Crontab calling a script in the hf_noise directory.
    The RF Samples is taken at a 1Mhz bandwidth from 1M to 30Mhz therefore 29 of  the CVS files is created and gets appended as the measurements is made.
    A Perl script utility is the executed from a cron scedular  to read the 29 CSV files and import there data into a RRD database.
    The RRD database then gets interrogated by another Perl script scheduled by a cronjob to generate the graph Images and and html files and get saved in the Apache web server graph directory witch is the exposed on http://localhost/hf_noise/graph/1Mhz_Power.php
    Link to Live system http://zr6aic.giga.co.za/hf_noise/graph/1Mhz_Power.php
    The RRD database will then create the Hourly, Daily,Weekly,Monthly and Yearly averages and   then generates the graphs every 2 minutes according to the parameters passed to Database.
    The generated graphs is then displayed in a php page via a Apache web server with an php plugin on the following url http://localhost/hf_noise/graph/1Mhz_Power.php.
    Trends can then be seen on a graphs over time.
    The heat map is generated by a python script from different CSV files and is saved in the /var/www/hf_noise/images directory and can be accessed on the local web server on the following url http://localhost/hf_noise/images
    Link to live system http://zr6aic.giga.co.za/vhf_heatmap/images/
    The systems has a static IP by default on Ethernet 192.168.10.200 and 192.168.10.205 on wifi if you have wifi dongle installed. You can change it to your ip

Here is a list of RRD database management tips

    How to backup my RRD db data?
    use rrdtool dump hf_noise.rrd > filename.xml to export the data to XML format.
    How to Restore my RRD db data?
    Transfer the XML dump to the target system. Run rrdtool restore [--range-check|-r] [--force-overwrite|-f] filename.xml filename.rrd to create a new RRD from the XML dump. See rrdrestore for details.
    How to change some of the the RRD db data?
    Use rrdtool dump to export RRD files to XML.
    Open the XML file, find and edit the bad data.
    Restore the RRD file using rrdtool restore.

 How to generate a brand new RRD db?


There is an script in the hf_noise directory called create_rrd_db.sh

You can just run this script in the hf_directory sudo ./create_rrd_db.sh


Here is the details of the scrip.

    rrdtool create  hf_noise.rrd --start now-2d --step 120 DS:1Mhz_Power:GAUGE:120:-50:10 DS:2Mhz_Power:GAUGE:120:-40:10 DS:3Mhz_Power:GAUGE:120:-50:10 DS:4Mhz_Power:GAUGE:120:-40:10 DS:5Mhz_Power:GAUGE:120:-50:10 DS:6Mhz_Power:GAUGE:120:-40:10 DS:7Mhz_Power:GAUGE:120:-50:10 DS:8Mhz_Power:GAUGE:120:-40:10 DS:9Mhz_Power:GAUGE:120:-50:10 DS:10Mhz_Power:GAUGE:120:-40:10 DS:11Mhz_Power:GAUGE:120:-50:10 DS:12Mhz_Power:GAUGE:120:-40:10 DS:13Mhz_Power:GAUGE:120:-50:10 DS:14Mhz_Power:GAUGE:120:-40:10 DS:15Mhz_Power:GAUGE:120:-50:10 DS:16Mhz_Power:GAUGE:120:-40:10 DS:17Mhz_Power:GAUGE:120:-50:10 DS:18Mhz_Power:GAUGE:120:-40:10 DS:19Mhz_Power:GAUGE:120:-50:10 DS:20Mhz_Power:GAUGE:120:-40:10 DS:21Mhz_Power:GAUGE:120:-50:10 DS:22Mhz_Power:GAUGE:120:-40:10 DS:23Mhz_Power:GAUGE:120:-50:10 DS:24Mhz_Power:GAUGE:120:-40:10 DS:25Mhz_Power:GAUGE:120:-50:10 DS:26Mhz_Power:GAUGE:120:-40:10 DS:27Mhz_Power:GAUGE:120:-50:10 DS:28Mhz_Power:GAUGE:120:-40:10 DS:29Mhz_Power:GAUGE:120:-50:10 DS:30Mhz_Power:GAUGE:120:-40:10 RRA:AVERAGE:0.5:1:864000 RRA:AVERAGE:0.5:60:129600 RRA:AVERAGE:0.5:3600:13392 RRA:AVERAGE:0.5:86400:3660

I have still got my csv files and want to reload it into my rrd database


HF Heat map monitoring System.


Link on local network http://localhost/hf_noise/images


link to live system http://zr6aic.giga.co.za/hf_noise/images/

Here is  a screenshot of the monitoring page and the green button

 can be used to select the frequency band of interest



Single view

Here is a view of the HF spectrum from 1Mhz to 30Mhz (activity can now be seen for the day)

Here is a view of the heat map that was selected

If you click on the image the the image will zoomed to maximum
Enlarged View

Here is a screenshot of the enlarged view.



So here is an example of the utility that generate the CSV file
Noise Recording

rtl_power -f 24M:1.7G:1M -g 50 -i 20m -1 noise-unmodded.csv

Here is the instillation instruction


hf_noise


Source code avalibe from https://github.com/antonjan/hf_noise


Alt text
Instillation Instructions


#Create user hfnoise

sudo adduser hfnoise

sudo usermod -aG sudo hfnoise


#change to hfnoise user with the following command. (dont try to install with other users)

su - hfnoise

#enter your new password

sudo apt-get update

#If you get this error below do the following

#** (appstreamcli:2761): CRITICAL **: Error while moving old database out of the way. AppStream cache update failed.

#then run this command and try again sudo chmod -R a+rX,u+w /var/cache/app-info/xapian/default

#now lets upgrade and get some coffie :-)


sudo upgrade

#We now need to install git to download the noise monitoring application

sudo apt-get install git

sudo apt install cmake

sudo apt-get install libusb-1.0-0-dev

sudo apt install python-setuptools

sudo easy_install pip

sudo pip install image

cd

#for ubuntu 17.10 do the following.

wget http://archive.ubuntu.com/ubuntu/pool/universe/i/imageinfo/imageinfo_0.04-0ubuntu11_amd64.deb

sudo dpkg -i ./imageinfo_0.04-0ubuntu11_amd64.deb

# for Ubuntu 16.04 do the following

 wget http://archive.ubuntu.com/ubuntu/pool/universe/i/imageinfo/imageinfo_0.04-0ubuntu10_amd64.deb

sudo dpkg -i ./imageinfo_0.04-0ubuntu10_amd64.deb

cd

#git clone git://git.osmocom.org/rtl-sdr.git

#Please make sure you use this repository as the osmond one dont support direct conversion mode

git clone https://github.com/keenerd/rtl-sdr.git

cd rtl-sdr/

mkdir build

cd build

cmake ../

make

sudo make install

sudo ldconfig

ls

sudo vi /etc/modprobe.d/no-rtl.conf

#add the following blacklist dvb_usb_rtl28xxu

blacklist rtl2832

blacklist rtl2830

#reboot

#we now need to install an application that will allow us to reset the USB port where the rtl dongle is installed if it locks up for some reason (this has append when there is lightning in the aria.So what i do is motor if the csv file get updated if not I reset the usb port with this utility.

cd

git clone https://github.com/jkulesza/usbreset.git

cd usbreset

#lets compile the application

cc usbreset.c -o usbreset

#we now need to establish witch usb port is your rtl dongle installed.

sudo lsusb

#we now need to edit the shell script that monitor the csv file update with your rtl usb port details.

#now look for the device with this name "Realtek Semiconductor Corp. RTL2838 DVB-T" in my case it was "Bus 003 Device 004" to be like E.g sudo /home/hfnoise/usbreset/usbreset /dev/bus/usb/003/004

cd

git clone  https://github.com/antonjan/hf_noise.git

cd hf_noise

cd sh

vi or use your editor and check_if_file_was_updated.sh

change the line that looks like this "vi /home/anton/Downloads/usbreset/usbreset /dev/bus/usb/"

002/005" with your detail

#pull the noise monitoring system from github

git clone https://github.com/antonjan/hf_noise.git

#install Apache2 server

sudo apt-get install apache2

sudo systemctl start apache2.service

sudo systemctl enable apache2.service

sudo apt install php7.1 libapache2-mod-php7.1

sudo a2enmod php7.1

sudo systemctl restart apache2

cd /var/www/html

sudo mkdir hf_noise

sudo mkdir hf_noise/graph

sudo mkdir hf_noise/images

cd hf_noise/graph

sudo cp -r /home/hfnoise/hf_noise/*.php ./
sudo cp -r /home/hfnoise/hf_noise/*.html ./
sudo cp -r /home/hfnoise/hf_noise/*.png ./
cd /var/www/html/hf_noise/images
sudo cp -r /home/hfnoise/hf_noise/image/*.php ./

sudo cp -r /home/hfnoise/hf_noise/image/*.html ./

sudo cp -r /home/hfnoise/hf_noise/image/*.jpg ./

sudo cp -r  /home/hfnoise/hf_noise/image/thumb ./

#now test if hfnoise webserver work.

firefox http://localhost/hf_noise

#you should see the index page with no pictures.

#We now need to install the heatmap application.

#lets go to home directory and install heatmap

cd

#got hf_noise directory and install heatmap by git from github

cd hf_noise

#Get the heatmap python progrm from github

git clone https://github.com/keenerd/rtl-sdr-misc.git

cd /home/hfnoise/hf_noise/rtl-sdr-misc/heatmap

#give the application execution writes

chmod 776 *.py

#Now we need to install the perl librareries

#CPAN, the Comprehensive Perl Archive Network, is the primary source for #publishing and fetching the latest modules

cd

sudo apt-get install perl build-essential curl

sudo apt-get install rrdtool

sudo apt-get install librrds-perl

cd /usr/local/bin

sudo wget https://raw.github.com/miyagawa/cpanminus/master/cpanm

sudo chmod +x cpanm

cd

sudo cpanm --self-upgrade --sudo

#sudo apt-get install cpanminus

sudo cpanm Text::Trim

sudo cpanm Text::CSV

sudo cpanm Date::Manip

sudo cpanm Text::Trim

# mite be only trim

sudo cpanm RRD::Simple

cd

cd hf_noise

cd sh

#run then script to generate rrd db

chmod 776 create_rrd_db.sh

./create_rrd_db.sh

#Ok now connect your HF dongel to your laptop usb port and connect to an antenna.

#Ok now everything should be ready.

#we need to enable the scripts in the crontab (scheduler)

#run the script to setup the scheduler

sudo python ./create_crontab.py


#If you want to see how the crontab was setup you can check with this command

sudo crontab -e

#Below is the what you should see

0,5,10,15,20,25,30,35,40,45,50,55 * * * * /home/hfnoise/hf_noise/sh/create_heatmap.sh 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58 * * * * /home/hfnoise/hf_noise/sh/run_hf_noise_monitor_and_graph.sh 1,6,11,16,21,26,31,36,41,46,51,56 * * * * /home/hfnoise/hf_noise


#save file and now and we are now done.

#The images and history file should now be automatically being created every 5 minutes.

#Make sure you have along wire antenna connected to your HF rtl dongle to hf noise recordings.

#********************end******************

#How can I check that data is collected.

#run the command

ls -lrt /home/hfnoise/hf_noise

#you should see somthing lyk this

-rw-r--r-- 1 root    root          0 Nov  8 19:06 hf_monitoring_08-11-2017.cvs

-rw-r--r-- 1 root    root          0 Nov  8 19:06 hf_monitoring_graph_08-11-2017.cvs

You should also see graph images in you browser.

firefox http://localhost/hf_noise/graph/1Mhz_Power.php


File upload to Master Analytics site will be available when it goes alive.








« Last Edit: January 30, 2018, 11:58:52 am by Anton Janovsky (ZR6AIC) »