Raspberry Pi Home Datalogger

[Static]

Lazy coding.
I had some iterative issues. I was repeatedly scanning the log. It probably added up to a dozen times. I got the time to generate a chart to under 18 seconds.
That's still really insane. Changing the number of records examined used in the chart doesn't really change the time required to generate the chart.

I'm wondering if I change the pickling mode. I think I can store it in binary.

Nuts.

[Static]

I need to figure out what is the faster way to do this with Python3. I may need to switch to the Marshal Module. I'm going to have to do a ton more coding, I think. It helps that my current screw-up can be measured with a stop-watch.

[Static]

Code: Select all

#!/usr/bin/python
import time
from numpy import *
import matplotlib as mpl
mpl.use('Agg')
import matplotlib.pyplot as plt

import cPickle as pickle


def log_line(level,loginfo, logfile = 'SensorLog2.log'):
    protocol = pickle.HIGHEST_PROTOCOL
    log = open(logfile, 'ab+')
    epochtime = time.time()
    realtime = time.asctime()
    logtime = {'Date':realtime}
    loginfo.update(logtime)
    timemark = {'Time': epochtime}
    loginfo.update(timemark)
    loglevel = {'Level': level}
    loginfo.update(loglevel)
    pickle.dump(loginfo, log, protocol)
    log.close()


def log_read(logfile = 'SensorLog2.log'):
    log = open(logfile, 'rb')
    testval = True
    loglist = []
    x = 0
    while testval == True:
        try:
            x = x + 1
            b = pickle.load(log)
            loglist.append(b)
            b = {}
        except:
            testval == False
            break
    log.close()
    return loglist, x-1

#def log_length(logfile = 'SensorLog.log'):
#    log = open(logfile, 'r')
#    testval = True
#    loglist = []
#    x = 0
#    while testval == True:
#        try:
#            x = x + 1
#            b = pickle.load(log)
#            loglist.append(b)
#            b = {}
#        except:
#            testval == False
#            break
#    return x-1              #it appears that the first record in the pickle isn't log data

def log_param_list(parameter, filler = 'NULL', logfile = 'SensorLog2.log'):
    loglist, length = log_read(logfile)
#    length = log_length(logfile)
    parameterlist=[]
    x=0
    for i in range(length):
        #print(i)
        c = loglist[i]
        if parameter in c:
            pulledvalue = c[parameter]
            parameterlist.append(pulledvalue)
            #print('Found', parameter, ' at ', i)
        else:
            #print('Not found')
            if filler != 'NULL':
                parameterlist.append(filler)
                #print('Filler ', filler, ' appended')
            #else:
                #print('Nothing appended')
           

    return parameterlist

def entry_level(logentry):
    temp1limits = {'critlow' : 0, 'warnlow' : 12, 'warnhigh' : 25, 'crithigh' : 30}
    temp2limits = {'critlow' : 0, 'warnlow' : 12, 'warnhigh' : 25, 'crithigh' : 30}
    info = 'Info'
    if 'temp1' in logentry:
        temp1 = logentry['temp1']
        if temp1 < temp1limits['critlow'] or temp1 > temp1limits['crithigh']:
            info = 'Critical'
           
        elif temp1 < temp1limits['warnlow'] or temp1 > temp1limits['warnhigh']:
            info = 'Warning'
           
    if 'temp2' in logentry and info != 'Critical':
        temp2 = logentry['temp2']
        if temp2 < temp2limits['critlow'] or temp2 > temp2limits['crithigh']:
            info = 'Critical'
           
        elif temp2 < temp2limits['warnlow'] or temp2 > temp2limits['warnhigh']:
            info = 'Warning'
           
    if info == 'Critical':
        info = 'Critical'     
    elif info == 'Warning':
        info = 'Warning'       
    else:
        info = 'Info'
       
    return info

def genchart(parameter, length = 1440, filler = 'NULL', logfile = 'SensorLog2.log', chartfile = 'Chart.png'):
    #parameterlength = log_length(logfile)
    parameterlist = log_param_list(parameter, filler, logfile)
    parameterlength = len(parameterlist)
    if length > parameterlength:
        length = parameterlength
    start = parameterlength - length
    end = parameterlength
    chartlist = parameterlist[start:end]
    plt.plot(chartlist, label=parameter)
    plt.legend()
    plt.savefig(chartfile)
    plt.close()



I'm currently using this code, and trying to figure out why it takes 14 seconds to parse 1200 records generated by this code.

Code: Select all

#!/usr/bin/python

from Logger import *
from Adafruit_BMP085 import BMP085
from Ty_TMP102 import TMP102
from Ty_TSL2561 import TSL2561
import time


def poll():
    bmp = BMP085(0x77)
    tmp = TMP102(0x48)
    tsl = TSL2561(0x39)
    logentry = {}
    temp1 = {'temp1' :bmp.readTemperature()}
    logentry.update(temp1)
    temp2 = {'temp2' :tmp.readTemperature()}
    logentry.update(temp2)
    pressure1 = {'pressure1' :bmp.readPressure()}
    logentry.update(pressure1)
    light1 = {'light1' :tsl.readLux()}
    logentry.update(light1)


    info = entry_level(logentry)
   
    log_line(info, logentry)
    print(logentry['Date'],logentry['temp1'],logentry['temp2'])

x = 1
y=0
while x == 1:
    y=y+1
    poll()
    print(y)
    time.sleep(1)


[Static]

And I've realized my complete and utter insanity.

I was trying to compare apples to... well.... nothing, really.

The code seems to take a little bit to "start-up". The way I was testing the code before (having it generate charts and websites once a minute) didn't let me see the amount of time that the chart generator took. When I was testing the new code, I was just testing the chart generator.

It's parsing close to 6000 records, and doing it in the same amount of time. I'll check with 20,000 records in the morning, but I think they're going to be around the same amount of time. Then I just need to re-write the scheduling software, and webpage creation code.

Oh yeah, and I still haven't figured out how to make the program start at boot-up. Grrrr.

[Static]

Yeah, I still don't understand how computers work.

OK, I just did a speed test on my program asking it to generate two temperature plots on the same graph. It went through 25,000 lines of log, pulled two temps for 20,000 lines of the log, and plotted it. The entire process took about 35 seconds. During this time, the data collection portion of the program (running in a separate SSL window) didn't even twitch.

So generating the plots takes some time, but it doesn't affect the other things that are going on. I would have thought that the computer would have just stopped and chewed through one process before jumping on the other. I thought I would have to program in some sort of smart process to have it bounce back and forth (I was really worried about this). Nope. Looks like the silicon is planning better than the meat.

WOOHOO

I think I see a place to tweak my code a little bit. Right now, the graph generator is running through the logs twice, once for each dataset that it is pulling. I'm going to re-code it so it pulls both sets of data on the one run through the code.

[Static]

The new code basis is really working well. I've got a couple of tweaks to do, but the foundation is very sound.
The core of the system logs sensor values to sensor names in a cPickle'd dictionary. Each line of the log is appended with a readable time/date, an elapsed epoch time, and a "log level" label.
The log is then easily searchable and graphable. I built a couple of different functions for graphs that will allow for individual sensors to be graphed, multiple sensors to be compared (useful for temps), and for multiple sensors to be compared across a fixed proportion (useful for wildly differing values pulled from sensors).

Today I ferreted out how to enable the second I2C bus from the RaspberryPi.org boards:
http://www.raspberrypi.org/phpBB3/viewtopic.php?f=44&t=33092

The code is pretty simple. Just run it, and the second I2C bus seems to be enabled.

I also soldered a header to the P5 location on the board so that I could easily attach the second I2C bus. I need to put pictures of that up.

[Static]

OK, it was a long time in coming, but the road was particularly bumpy.

First, I've got some pictures and some links to code for two I2C buses on one Raspberry Pi:
http://medicforlife.blogspot.com/2013/02/two-i2c-buses-on-raspberry-pi.html

I ran into an issue with the Adafruit_I2C.py code. I had to kit-bash two versions of the code together in order to get my BMP085 temperature/pressure sensors to work. My version of the code seems to be working great for me, but I haven't seen any outside verification. That code is here: (http://forums.adafruit.com/viewtopic.php?f=8&t=38255&p=190464#p190464

Right now I'm running two I2C sensor lines. Both are using a BMP085 for temperature and pressure, and a TSL2561 light sensor (code here: http://forums.adafruit.com/viewtopic.php?f=8&t=34922). The light sensor is still really quirky in bright conditions. I'm starting to wonder if one of my sensors has some damage or something.

I'm going to try to get the website code back up and running over the next couple of days. This will involve writing the code to automate the creation of graph files and putting them in the right spot. Then I'll need to automate the generation of the webpage. Finally I'll need to re-write the scheduling code to do everything at regular intervals and set up the Apache server on the Pi correctly. I did all of this once before, but it was not efficient or pretty.