Yesterday I was annoyed at Xively after spending hours trying to get their library to work with the limited documentation, well if I had written this earlier today, I would have been totally livid. I overcame my problem with updating more than one datastream (Xively term) at a time, but it certainly wasnt because their documentation was clear. Far from it, I found an example of updating a couple of items in a download of their library, BUT IT HAD COMPILE ERRORS.
We've all seen this. Examples that need libraries they don't mention, code fragments that don't make sense, full blown examples that only illustrate a trivial case, and the ultimate insult, examples that don't compile. I chased one of those for about an hour before I found the solution.
I did get a python script to work updating more than one datastream to a brand new feed I created on Xively. Yes, you have to learn a new set of terms to use this stuff. Armed with this tiny bit of success, I put in the code to gather XBee packets and update some global variables, then push them up to Xively and ran it for an hour or so to watch what happened. It worked pretty well. I don't have code in it to gather all the data I want, or a way to store it such that my new web server can present it, but I've got a nice start.
Let's talk about some of the things I've discovered. The python scheduler works really well and does all the stuff I want. I can set a routine to run any time I want, even to the point of six months from now at noon. It's not quite as good at tiny periods, but I don't need that right now. The Xively library is actually pretty extensive, but the documentation is terrible. They don't even have a list of classes that are available, they rely on samples that suck instead. The XBee library doesn't work the way a person coming from an Arduino experience expects it to, it forces you to use threads which makes passing data around harder than expected. To make up for this, there is a cool library that can queue things up for you so another thread can grab them. This little queue is really nice and could be used in a lot of different ways.
The HUGE advantage is the raspberry's handling of the internet. It just works. No long delays while the ethernet chip makes up its mind to work, you have tons of connections to play with, processing the returned data is a snap since python has an enormous string handling library. There's so much that can be done there it's amazing.
Here's what I have so far:
This gives the following output on the raspberry:
What I do is create a thread to catch XBee packets and queue them up to be handled. In the main thread, I grab the packets off the queue and take them apart, saving a couple of important items in global variables. I have an event scheduled to print the value of the global variables every 15 seconds and another event scheduled to run every minute and send updates to Xively. Yes, this is an odd way of doing it, but it's what I already do on my current house controller. I've found that scheduling things to happen is a much simpler way of handling tasks than anything else I've tried.
There's no internet handling in this module at all. I will do that next. As I mentioned before, I have two thermostats that are hooked to my local lan that can take commands and respond; I'll put the code in to query them every so often and save the results. Since the internet handling in python is so robust, that shouldn't be a problem at all.
The big problem is deciding how to store the house data in such a way that the web server I have running on the Pi can get at it. Everyone uses a database, but I'm not sure a big hunk of code like that is reasonable for a task like this. Gotta think about it and experiment a bit before I go that route.
Perseverance, or maybe bull-headedness, has gotten me this far and I truly hope other people that are thinking about doing something like this stumble across this site. It just might save them some of the headaches I've had.
We've all seen this. Examples that need libraries they don't mention, code fragments that don't make sense, full blown examples that only illustrate a trivial case, and the ultimate insult, examples that don't compile. I chased one of those for about an hour before I found the solution.
I did get a python script to work updating more than one datastream to a brand new feed I created on Xively. Yes, you have to learn a new set of terms to use this stuff. Armed with this tiny bit of success, I put in the code to gather XBee packets and update some global variables, then push them up to Xively and ran it for an hour or so to watch what happened. It worked pretty well. I don't have code in it to gather all the data I want, or a way to store it such that my new web server can present it, but I've got a nice start.
Let's talk about some of the things I've discovered. The python scheduler works really well and does all the stuff I want. I can set a routine to run any time I want, even to the point of six months from now at noon. It's not quite as good at tiny periods, but I don't need that right now. The Xively library is actually pretty extensive, but the documentation is terrible. They don't even have a list of classes that are available, they rely on samples that suck instead. The XBee library doesn't work the way a person coming from an Arduino experience expects it to, it forces you to use threads which makes passing data around harder than expected. To make up for this, there is a cool library that can queue things up for you so another thread can grab them. This little queue is really nice and could be used in a lot of different ways.
The HUGE advantage is the raspberry's handling of the internet. It just works. No long delays while the ethernet chip makes up its mind to work, you have tons of connections to play with, processing the returned data is a snap since python has an enormous string handling library. There's so much that can be done there it's amazing.
Here's what I have so far:
The Python Script
#! /usr/bin/python
# This is an example of asyncronous receive
# What it actually does is fork off a new process
# to do the XBee receive. This way, the main
# code can go do somthing else and hand waiting
# for the XBee messages to come in to another
# process.
from xbee import ZigBee
from apscheduler.scheduler import Scheduler
import logging
import datetime
import time
import serial
import Queue
import xively
#-------------------------------------------------
# on the Raspberry Pi the serial port is ttyAMA0
XBEEPORT = '/dev/ttyAMA0'
XBEEBAUD_RATE = 9600
# The XBee addresses I'm dealing with
BROADCAST = '\x00\x00\x00\x00\x00\x00\xff\xff'
UNKNOWN = '\xff\xfe' # This is the 'I don't know' 16 bit address
# The Xively feed id and API key that is needed
FEED_ID = 'putsomethinghere'
API_KEY = 'and here to'
# Global items that I want to keep track of
CurrentPower = 0
DayMaxPower = 0
DayMinPower = 50000
CurrentOutTemp = 0
DayOutMaxTemp = -50
DayOutMinTemp = 200
#-------------------------------------------------
logging.basicConfig()
#------------ XBee Stuff ------------------------
packets = Queue.Queue() # When I get a packet, I put it on here
# Open serial port for use by the XBee
ser = serial.Serial(XBEEPORT, XBEEBAUD_RATE)
# this is a call back function. When a message
# comes in this function will get the data
def message_received(data):
packets.put(data, block=False)
#print 'gotta packet'
def sendPacket(where, what):
# I'm only going to send the absolute minimum.
zb.send('tx',
dest_addr_long = where,
# I always use the 'unknown' value for this
# it's too much trouble to keep track of two
# addresses for the device
dest_addr = UNKNOWN,
data = what)
# In my house network sending a '?\r' (question mark, carriage
# return) causes the controller to send a packet with some status
# information in it as a broadcast. As a test, I'll send it and
# the receive above should catch the response.
def sendQueryPacket():
# I'm broadcasting this message only
# because it makes it easier for a monitoring
# XBee to see the packet. This is a test
# module, remember?
#print 'sending query packet'
sendPacket(BROADCAST, '?\r')
# OK, another thread has caught the packet from the XBee network,
# put it on a queue, this process has taken it off the queue and
# passed it to this routine, now we can take it apart and see
# what is going on ... whew!
def handlePacket(data):
global CurrentPower, DayMaxPower, DayMinPower
global CurrentOutTemp, DayOutMaxTemp, DayOutMinTemp
#print data # for debugging so you can see things
# this packet is returned every time you do a transmit
# (can be configure out), to tell you that the XBee
# actually send the darn thing
if data['id'] == 'tx_status':
if ord(data['deliver_status']) != 0:
print 'Transmit error = ',
print data['deliver_status'].encode('hex')
# The receive packet is the workhorse, all the good stuff
# happens with this packet.
elif data['id'] == 'rx':
rxList = data['rf_data'].split(',')
if rxList[0] == 'Status':
# remember, it's sent as a string by the XBees
tmp = int(rxList[1]) # index 1 is current power
if tmp > 0: # Things can happen to cause this
# and I don't want to record a zero
CurrentPower = tmp
DayMaxPower = max(DayMaxPower,tmp)
DayMinPower = min(DayMinPower,tmp)
tmp = int(rxList[3]) # index 3 is outside temp
CurrentOutTemp = tmp
DayOutMaxTemp = max(DayOutMaxTemp, tmp)
DayOutMinTemp = min(DayOutMinTemp, tmp)
else:
print 'Unimplemented XBee frame type'
#-------------------------------------------------
# This little status routine gets run by scheduler
# every 15 seconds
def printHouseData():
print('Power Data: Current %s, Min %s, Max %s'
%(CurrentPower, DayMinPower, DayMaxPower))
print('Outside Temp: Current %s, Min %s, Max %s'
%(CurrentOutTemp, DayOutMinTemp, DayOutMaxTemp))
print
# This is where the update to Xively happens
def updateXively():
print("Updating Xively with value: %s and %s"%(CurrentPower, CurrentOutT
emp))
print
now = datetime.datetime.utcnow()
feed.datastreams = [
xively.Datastream(id='outside_temp', current_value=CurrentOutTem
p, at=now),
xively.Datastream(id='power_usage', current_value=CurrentPower,
at=now)
]
feed.update()
#------------------Stuff I schedule to happen -----
sendsched = Scheduler()
sendsched.start()
# every 30 seconds send a house query packet to the XBee network
sendsched.add_interval_job(sendQueryPacket, seconds=30)
# every 15 seconds print the most current power info
sendsched.add_interval_job(printHouseData, seconds=15)
# every minute update the data store on Xively
sendsched.add_interval_job(updateXively, seconds=60)
# Create XBee library API object, which spawns a new thread
zb = ZigBee(ser, callback=message_received)
# Initialize api client
api = xively.XivelyAPIClient(API_KEY)
# and get my feed
feed = api.feeds.get(FEED_ID)
#Do other stuff in the main thread
while True:
try:
time.sleep(0.1)
if packets.qsize() > 0:
# got a packet from recv thread
# See, the receive thread gets them
# puts them on a queue and here is
# where I pick them off to use
newPacket = packets.get_nowait()
# now go dismantle the packet
# and use it.
handlePacket(newPacket)
except KeyboardInterrupt:
break
# halt() must be called before closing the serial
# port in order to ensure proper thread shutdown
zb.halt()
ser.close()
# This is an example of asyncronous receive
# What it actually does is fork off a new process
# to do the XBee receive. This way, the main
# code can go do somthing else and hand waiting
# for the XBee messages to come in to another
# process.
from xbee import ZigBee
from apscheduler.scheduler import Scheduler
import logging
import datetime
import time
import serial
import Queue
import xively
#-------------------------------------------------
# on the Raspberry Pi the serial port is ttyAMA0
XBEEPORT = '/dev/ttyAMA0'
XBEEBAUD_RATE = 9600
# The XBee addresses I'm dealing with
BROADCAST = '\x00\x00\x00\x00\x00\x00\xff\xff'
UNKNOWN = '\xff\xfe' # This is the 'I don't know' 16 bit address
# The Xively feed id and API key that is needed
FEED_ID = 'putsomethinghere'
API_KEY = 'and here to'
# Global items that I want to keep track of
CurrentPower = 0
DayMaxPower = 0
DayMinPower = 50000
CurrentOutTemp = 0
DayOutMaxTemp = -50
DayOutMinTemp = 200
#-------------------------------------------------
logging.basicConfig()
#------------ XBee Stuff ------------------------
packets = Queue.Queue() # When I get a packet, I put it on here
# Open serial port for use by the XBee
ser = serial.Serial(XBEEPORT, XBEEBAUD_RATE)
# this is a call back function. When a message
# comes in this function will get the data
def message_received(data):
packets.put(data, block=False)
#print 'gotta packet'
def sendPacket(where, what):
# I'm only going to send the absolute minimum.
zb.send('tx',
dest_addr_long = where,
# I always use the 'unknown' value for this
# it's too much trouble to keep track of two
# addresses for the device
dest_addr = UNKNOWN,
data = what)
# In my house network sending a '?\r' (question mark, carriage
# return) causes the controller to send a packet with some status
# information in it as a broadcast. As a test, I'll send it and
# the receive above should catch the response.
def sendQueryPacket():
# I'm broadcasting this message only
# because it makes it easier for a monitoring
# XBee to see the packet. This is a test
# module, remember?
#print 'sending query packet'
sendPacket(BROADCAST, '?\r')
# OK, another thread has caught the packet from the XBee network,
# put it on a queue, this process has taken it off the queue and
# passed it to this routine, now we can take it apart and see
# what is going on ... whew!
def handlePacket(data):
global CurrentPower, DayMaxPower, DayMinPower
global CurrentOutTemp, DayOutMaxTemp, DayOutMinTemp
#print data # for debugging so you can see things
# this packet is returned every time you do a transmit
# (can be configure out), to tell you that the XBee
# actually send the darn thing
if data['id'] == 'tx_status':
if ord(data['deliver_status']) != 0:
print 'Transmit error = ',
print data['deliver_status'].encode('hex')
# The receive packet is the workhorse, all the good stuff
# happens with this packet.
elif data['id'] == 'rx':
rxList = data['rf_data'].split(',')
if rxList[0] == 'Status':
# remember, it's sent as a string by the XBees
tmp = int(rxList[1]) # index 1 is current power
if tmp > 0: # Things can happen to cause this
# and I don't want to record a zero
CurrentPower = tmp
DayMaxPower = max(DayMaxPower,tmp)
DayMinPower = min(DayMinPower,tmp)
tmp = int(rxList[3]) # index 3 is outside temp
CurrentOutTemp = tmp
DayOutMaxTemp = max(DayOutMaxTemp, tmp)
DayOutMinTemp = min(DayOutMinTemp, tmp)
else:
print 'Unimplemented XBee frame type'
#-------------------------------------------------
# This little status routine gets run by scheduler
# every 15 seconds
def printHouseData():
print('Power Data: Current %s, Min %s, Max %s'
%(CurrentPower, DayMinPower, DayMaxPower))
print('Outside Temp: Current %s, Min %s, Max %s'
%(CurrentOutTemp, DayOutMinTemp, DayOutMaxTemp))
# This is where the update to Xively happens
def updateXively():
print("Updating Xively with value: %s and %s"%(CurrentPower, CurrentOutT
emp))
now = datetime.datetime.utcnow()
feed.datastreams = [
xively.Datastream(id='outside_temp', current_value=CurrentOutTem
p, at=now),
xively.Datastream(id='power_usage', current_value=CurrentPower,
at=now)
]
feed.update()
#------------------Stuff I schedule to happen -----
sendsched = Scheduler()
sendsched.start()
# every 30 seconds send a house query packet to the XBee network
sendsched.add_interval_job(sendQueryPacket, seconds=30)
# every 15 seconds print the most current power info
sendsched.add_interval_job(printHouseData, seconds=15)
# every minute update the data store on Xively
sendsched.add_interval_job(updateXively, seconds=60)
# Create XBee library API object, which spawns a new thread
zb = ZigBee(ser, callback=message_received)
# Initialize api client
api = xively.XivelyAPIClient(API_KEY)
# and get my feed
feed = api.feeds.get(FEED_ID)
#Do other stuff in the main thread
while True:
try:
time.sleep(0.1)
if packets.qsize() > 0:
# got a packet from recv thread
# See, the receive thread gets them
# puts them on a queue and here is
# where I pick them off to use
newPacket = packets.get_nowait()
# now go dismantle the packet
# and use it.
handlePacket(newPacket)
except KeyboardInterrupt:
break
# halt() must be called before closing the serial
# port in order to ensure proper thread shutdown
zb.halt()
ser.close()
This gives the following output on the raspberry:
Console Output
pi@deserthome:~/src$ python powertoxively.py
Power Data: Current 0, Min 50000, Max 0
Outside Temp: Current 0, Min 200, Max -50
Power Data: Current 639, Min 639, Max 639
Outside Temp: Current 109, Min 109, Max 109
Power Data: Current 639, Min 639, Max 639
Outside Temp: Current 109, Min 109, Max 109
Power Data: Current 636, Min 636, Max 639
Outside Temp: Current 109, Min 109, Max 109
Updating Xively with value: 636 and 109
Power Data: Current 636, Min 636, Max 639
Outside Temp: Current 109, Min 109, Max 109
Power Data: Current 638, Min 636, Max 639
Outside Temp: Current 109, Min 109, Max 109
Power Data: Current 638, Min 636, Max 639
Outside Temp: Current 109, Min 109, Max 109
Not a huge bunch of impressive stuff, but it illustrates the point. I put as many comments in there as I could, both to help people understand and to nudge my own memory when I come back to this after doing something else for a while.Power Data: Current 0, Min 50000, Max 0
Outside Temp: Current 0, Min 200, Max -50
Power Data: Current 639, Min 639, Max 639
Outside Temp: Current 109, Min 109, Max 109
Power Data: Current 639, Min 639, Max 639
Outside Temp: Current 109, Min 109, Max 109
Power Data: Current 636, Min 636, Max 639
Outside Temp: Current 109, Min 109, Max 109
Updating Xively with value: 636 and 109
Power Data: Current 636, Min 636, Max 639
Outside Temp: Current 109, Min 109, Max 109
Power Data: Current 638, Min 636, Max 639
Outside Temp: Current 109, Min 109, Max 109
Power Data: Current 638, Min 636, Max 639
Outside Temp: Current 109, Min 109, Max 109
What I do is create a thread to catch XBee packets and queue them up to be handled. In the main thread, I grab the packets off the queue and take them apart, saving a couple of important items in global variables. I have an event scheduled to print the value of the global variables every 15 seconds and another event scheduled to run every minute and send updates to Xively. Yes, this is an odd way of doing it, but it's what I already do on my current house controller. I've found that scheduling things to happen is a much simpler way of handling tasks than anything else I've tried.
There's no internet handling in this module at all. I will do that next. As I mentioned before, I have two thermostats that are hooked to my local lan that can take commands and respond; I'll put the code in to query them every so often and save the results. Since the internet handling in python is so robust, that shouldn't be a problem at all.
The big problem is deciding how to store the house data in such a way that the web server I have running on the Pi can get at it. Everyone uses a database, but I'm not sure a big hunk of code like that is reasonable for a task like this. Gotta think about it and experiment a bit before I go that route.
Perseverance, or maybe bull-headedness, has gotten me this far and I truly hope other people that are thinking about doing something like this stumble across this site. It just might save them some of the headaches I've had.