Quantcast
Channel: Desert Home
Viewing all 218 articles
Browse latest View live

Adding a USB Stick to the Raspberry Pi

$
0
0
If you beat the heck out of your SD card on a Raspberry Pi by having code that constantly updates a database, you're eventually going to start having problems with the card.  These little SD cards are tough little critters and (some of them) work really well, but they have a finite life.

I started noticing increased problems with database locks, Occasional corrupted data, the kind of thing that goes along with the SD card starting to fail.  It wasn't a really bad problem, but I don't want to have to deal with a bad problem, so I got one of these:


I picked this particular one after reading about a thousand reviews of various USB sticks and took the plunge for this particular device.  The idea is that I wanted one that wouldn't die in a couple of days, would serve other purposes if I decided to do something else on the Pi, and had enough capacity to be relevant for a long time.  16GB should last me a while on the Pi, and if it became photo storage later, it would handle a lot of them.

I let it set on the shelf for quite a while before I finally decided to use it as the main storage on my Pi.  It's Saturday, I really don't want to dig on the drainage trench I've been working on for two weeks, and the neighbors blocked my driveway with a backhoe, so it's time to do this.

When I went looking on the web for instructions to do this, there were literally thousands of them out there.  Most would work, but I don't want to do most of it on the PC and then stick the device on the Pi and hope.  I was amazed at how many Mac instructions there are; I thought Mac users were all artists that couldn't understand real computers (heh).  But I found a set of simple instructions created by pauly that fit the bill nicely <link>.

Since I'm a coward, I backed up the SD card first, then rebooted the Pi and started down the instructions.  Every single step worked exactly as pauly described, and in about an hour I rebooted using the SD card for the boot partition and the USB card for the main file system.  I didn't do anything about expanding swap space yet, nor did I mess with removing the old file system on the SD card.  Those may become useful at some point, but I don't need to mess with it now.

The Pi booted faster, but that's not very important since I don't boot it very often.  I noticed a slight decrease in load average, and time will tell if I see a decrease in the number of problems I've been having.  It's only been up for a couple of hours so I can't speak from experience yet.

I do need to look at the mount point I used for the usb stick though.  I used the default /dev/sda for it, and if I plug in another USB stick, I may have a problem with the default name.  There's the by-id device to look at and I should use it so the name is constant.  But, for now, I'll just get some experience with the setup and see if it holds together.  That's mostly why I have a backup and also kept the file system on the SD card.

This particular Pi has two XBees on it.  As I mentioned in previous posts, I had to do this because the default Digi network and Lowe's Iris devices don't play well together.  This was the third device I added to the USB buss; so I have a powered hub in place to take care of the fact that my USB buss probably uses more power than the computer that runs it.  I'm currently using one of these:
It has seven ports; I hope I don't need all of them, and came with a power supply that can handle that many devices.  I've been bitten by devices like this that just can't cut it when I plugged in a few power hungry peripherals.  It's really annoying to have to hunt down a power supply when you want to try something.  This little guy has worked well.

So, here's a possibility for you folk that are nearing the failure point of the SD card.  Most people that play with the Pi's don't run them 24x7 for months on end with a database and web server beating on things, so they'll never get to this point.

But, some of us do.

OK, Now I Have a Question I Need Help With

$
0
0
I ran across a forum discussion that got me to thinking <link>.  I was researching (again) measuring power with some non intrusive method when I ran into a typical snarky set of responses to a person showing a possible solution.  The author of the post showed this picture of a clamp current meter and some wiring:
Granted, it looks like it won't measure anything since both lines go through the current clamp, but I'm here to tell you this works.

Yep, if the direction is reversed using a loop in the wire it will measure total current through the two lines.  I actually tried it with a current clamp and some 14 gauge wires to a 220V light bulb in an Edison socket.  As further proof, here's a picture of the wiring inside my smart meter:
Look closely, you can zoom if you need to, but notice how one of the power legs goes through the current transformer in the opposite direction?  Here's another picture of the wires removed from the meter:
One current transformer and two wires to measure the total current being used by my house.  I asked the guy from the power company why they didn't just put the current transformer around the neutral and he said people could avoid part of the bill by referencing to ground avoiding the neutral path.  That actually makes sense in a very unsafe fashion.  There's a number of things you can do with current transformers that I hadn't thought of until now.  You can run both wires through in the same direction and tell if there is current to ground; that's how the GFCI you have in the bathroom works.  You can loop a wire around the transformer and double the voltage out to measure smaller currents.  Heck, there's a bunch of configurations that can help us measure things if we learn how they work.

My question is, how the heck does this work?  I know it does, I've done it, and the meter manufacturer has also.  I just can't get my head around the physics of it.

This also points out how folk on the web jump on someone who is correct and belittle them.  I guess we've all been there some time or other.  The author of the picture showed a heck of a lot more restraint than I would have in the same circumstances.

Wemo Light Switches and Intermittent Failures ... Found it!

$
0
0
I have four Wemo light switches operating around the house on outside lights.  You know the ones.  They're the switches that you remember after you get in bed and have the covers pulled up.  In my case, I just grab a phone and turn the darn things off from the bed ... except ...

Every once in a while they don't respond.  This has driven me nuts ever since I got the switches and I'm not alone.  There are other folk out there that have been annoyed by this behavior.  After I simplified my code running the switches <link>, and the problem was still there; I set out to see what the heck was going on.  At first the code would hang waiting for a response from the switches, so I put a timeout on the socket read.

When the inevitable timeout came, I at least had a clue; the switch just quit responding.  So, I caught the timeout and printed the URL that was involved.  A few days later, it timed out again and I discovered that the IP address was ok, but the port number the Wemo was listening to had changed.  There's simply no reason for the port number to change ... unless the switch had done it intentionally.  Naw, no one would do that.

Then, the power failed to my house and everything rebooted.  Fine, I'd wait until it happened again.

One evening, one of the switches failed.  I got an email telling me the switch had gone off line.  Yes, an email.  One of the great things about working on a Pi is that it can do things like this.  I had put code in to send me an email when one of the switches got into this state.  I checked the logs (another great thing) and sure enough, the port number had changed.  A little while later, another switch failed, then the other two as well.  The period of working was roughly three days, and then, the switches would change the port number they responded to.

I don't have definitive proof, but I firmly believe they reboot the switches every three days.  Over time I could probably accumulate evidence to support this, but I don't want evidence; I want the darn switches to work.

What I had done in the code was to set a timeout on the socket receive from a polling request to the switch.  When it timed out, it would send me mail and then my process would exit.  Since the process is controlled by upstart (google it), it would start right back up and rediscover the switches.  Everything should be fine then because the new port number would be recorded and control would be set up correctly.  I actually rebooted the control process to just start over.  Not very elegant, but it worked.  When the four switches went south, the process restarted each time and got control back.

Ha, take that Belkin.

Of course there's a downside.  During the time the process is restarting, there could be a command to the switches missed.  It takes about 30 seconds to discover the switches and get all set up so the window is small, but it's there.  I think I can live with that.  I removed the code that sends mail to keep the changes I make to Miranda to an absolute minimum and have the changes running right now.  After a couple of weeks, I'll know how well it worked out.

It's very important to me to avoid relying on an external service to control something around my house, so things like If This Then That, are not an option.  Sure, I like the Wemo app for checking the lights from town, but I don't want to rely on it.  The Internet is just too flaky, companies change policies, and people like to muck with other people's stuff.  I want control inside my own house.

I'm getting closer now.

Taking Another Look at Process Communication - CherryPy

$
0
0
I've been using SysV interprocess communication between the various processes that control and monitor the devices around the house, and it's been working fine.  But, at some point I'm going to want to separate the processes into different machines.  A Pi for the lights, a Pi for the devices I hope to have somewhere else, and a pet project of mine, a Pi to control the functions of the pool.  SysV doesn't talk between machines.  I can use TCP for this, but I don't want to invent a protocol on top of it, I want something I can just pick up and use.  How about HTTP?

I could bring up a web server on each machine and hook it into the processes that are involved, but that would take a bunch of research.  So I got to looking around and there were two nice solutions, Tornado <link> and CherryPy <link>.  They're both nice, but CherryPy is simpler to use, so I chose it as my first experiment.  Besides, it's cool to be running CherryPy on a Raspberry Pi, ... right?

It came up on the first try using their tutorial and worked.  That looks promising, but what about doing something else besides just waiting for an HTTP request?  How to have several of them running on the same machine talking between processes?  How do I keep track of the various addresses?

In my usual form, I just started writing code to see what would happen.  CherryPy uses a publish, subscribe philosophy, and provides some basic published items.  I subscribed to their 'main' and used that to run my tiny timer, and suddenly, it could do something besides just wait for HTML to come in.  So, I have my timers, the ability to periodically check device status and probably do some other stuff.  Now would be a good time to expand the experiment.

The idea is to be able to communicate with each control or monitor process to see what is going on with a browser from my recliner as well as a running process that accumulates everything for presentation.  Hopefully, this will make it much easier to tell what is going on when something quits working properly.  I hope to use JSON as the data transfer protocol and then we'll be able to read it as well as use it easily in code. Eventually, I want to be able to separate the processes between Raspberry Pi's and have them keep working.  My very own tiny data center setting on a shelf over there somewhere.

But, I'm a bit sick of changing things, breaking them, and not having a clue what I did to break them.  Enter GitHub <link>.  I've put my current Raspberry Pi source on GitHub so I can track changes and back out things that just didn't work well.  I can also create a fork for massive experiments, load them on another Pi and play until it works.  A side effect is that you folk can grab the source without having to copy and paste from a window.  There have been many problems with python and its indentation rules, this should fix that.  Also, if you find a bug, you can fix it and then I can grab it as well.

The latest test of my move to CherryPy is in the file cherrytest.py in the other-stuff directory in my GitHub repository <link>.  The house directory is where I put the code I'm running right now on my Pi to control the house.  When I start testing using CherryPi to communicate, I'll fork off a new version and play there until I can tell if it is going to work out.

My very own little sandbox to play in.

Raspberry Pi, CherryPy, Process Communication, Part 2

$
0
0
Last post I talked about experimenting with a different method of process communication: using HTTP between the processes.  This will allow me to move some of my work to another machine, and increase my ability to monitor the various items.

I got into this mess because of the plethora of different devices and protocols I'm messing with.  Each set of devices has a different protocol, some invented by me, others invented by industry.  If you have two different ZigBee devices that can't talk to each other, you need something to translate in the middle.  This isn't too hard, but it gets complex when you're chasing a bug around and can't poke the pieces to see what happens.  So, I started with my Wemo devices, and added a minimal webserver to the code.  This allows me to talk to and check the status of the switches using a browser as well as a separate process that collects the data.

It turned out to be relatively easy, but I (again) had to learn a new jargon.  I'm finding over and over that the key to doing anything in the home automation realm is the language.  When you look at a library or implementation, be prepared for a few hours with google to understand what they're saying.

CherryPy turned out to be pretty slick.  It works differently than the big servers like Apache, but it can be used pretty easily.  I started off with the CherryPy tutorial and just started adding things I needed until I had a skeleton.  That's described in the last post.  Then I took the skeleton in one window and the existing code in another and started the cut-and-paste process of combining them.  It turned out pretty well.  I have several web-like interfaces to the Wemo code that I can interact with, and so far, it's holding up pretty well.  For example, I have a human interface that is really easy to use; I just type in 192.168.0.205:51001 as a URL and I get back:

Current Wemo Light Switch Status

Front Porch is Off  
Outside Garage Lights are Off  
Cactus Spot Lights are Off  
West Patio Lights are Off  

The status is immediate, meaning I actually send off a UPNP request and get the state.  The buttons work and will operate the lights.  It's basically a tiny web server to control the lights without all the hassle of bringing up Apache and a ton of other things.  Yes, it could be extended to use pictures and really cool looking interfaces, but that wasn't what I was going for, this is a nice side effect.

The machine interface is different.  If I type in 192.168.0.205:51001/status, I get:

[{"cactusspot": "Off"}, {"frontporch": "Off"}, {"patio": "Off"}, {"outsidegarage": "Off"}]

Just the bare essentials: the names of the switches and their states right now.  Notice that I have to type in less characters for the human interaction than I do for the machine interaction?  Yep, that's on purpose, I'm lazy.  If you want to see how lazy I am, I have to type in:

192.168.0.205:51001/pCommand?command=OutsideLightsOn

To turn on a single light with the process interface, and it doesn't return anything because it updates the SQLite3 database I use instead.  Sure, it's a painful thing to do, but who cares?  I don't have to type it, it's constructed by code.  To illustrate how a web page can be constructed, here is the code for the human readable page I put up:

    @cherrypy.expose
def index(self):
status = "<strong>Current Wemo Light Switch Status</strong><br /><br />"
status += "Front Porch is " + get("frontporch") + "&nbsp;&nbsp;"
status += '<a ><button>Toggle</button></a>'
status += "<br />"
status += "Outside Garage Lights are " + get("outsidegarage") + "&nbsp;&nbsp;"
status += '<a ><button>Toggle</button></a>'
status += "<br />"
status += "Cactus Spot Lights are " + get("cactusspot") + "&nbsp;&nbsp;"
status += '<a "><button>Toggle</button></a>'
status += "<br />"
status += "West Patio Lights are " + get("patio") + "&nbsp;&nbsp;"
status += '<a "><button>Toggle</button></a>'
status += "<br />"
return status

The entire page is constructed as a string and just sent back to my browser.  I used the simplest buttons and web techniques I could to make it as easy as possible.  Still, it works and doesn't look bad.  Notice that when you click on a button, it goes for the URL ./wemocommand ?  That's the code that will toggle a particular light.  I also use URL parameters (that's what the '?' is for) because they turned out to be easy to parse out.  I could have made this smaller by using a list, and I may some day, but this is much easier to understand for folk that are trying to copy the idea.  It also makes it simple to code for acting on the incoming HTML request:

    @cherrypy.expose
def wemocommand(self, whichone):
# first change the light state
toggle(whichone)
# now reload the index page to tell the user
raise cherrypy.InternalRedirect('/index')

Yes, that's all there was to it.  I already had the toggle() routine to support the old interface and I simply redisplay the first page because it will go get the new state of the switches.  At this point in my experiment, I was beginning to really like CherryPy.

Just to round out this a bit, here is the code for the JSON string return I do for the process interface:

    @cherrypy.expose
@cherrypy.tools.json_out() # This allows a dictionary input to go out as JSON
def status(self):
status = []
for item in lightSwitches:
status.append({item["name"]:get(item["name"])})
return status

Notice that CherryPy already has a JSON translator built in?  Is that cool or what?  All I had to do was get the lights state in a list and then return it.  It makes the code a little hard to understand, but it was so simple to put together.

If you want to see the entire module, it's under the branch wemoHTML in my GitHub repository.  If you haven't figured out how to get to that yet (took me some time), here's a link <link> that should get you directly to it.  The module is wemocontrol.py in the house directory.

I still haven't adapted the other pieces necessary to fully implement the idea.  I have to change the web code (php) that handles commands and also the presentation page for my control web page.  I don't expect to have too much trouble with it, but I'll probably have to learn the meaning of some new words.

AcuRite Weather Station, Raspberry Pi and a USB interface; Part 1

$
0
0
This is a long drawn out story.  Sorry.  I'm going to go into more detail than I usually do and the project will span more than one blog entry.  That's because it is really complex and deals with stuff that most of us would rather not mess with.  We want to monitor and control our house, not do a dissertation.  But, in this case, you can't have enough background information or advice.  There are things that can happen that you need a minimum of background to overcome.  I hope to give you enough to repeat the project.

It all started when one of my readers asked me if I had looked into a weather station for the house. Well, I had, but there were two problems, the good ones cost a fortune, and the others don't have a lot of capabilities. I was going to buy one of the sensor sets sold by SparkFun. They have a really nice one for about $70 that would do the job pretty well. As luck would have it, I was in Costco drooling over the multi-terrabyte disks and noticed they had a Acu-Rite weather station for $80, display and all. It had a sensor head with windspeed, rainfall, temperature, humidity and wind direction. The console was an LCD color display with a USB output that promised I could plug it into a computer and send data out on the web.


Yes, I left with it.

Heck, I couldn't buy the sensors and assemble them any cheaper, plus I'd have to learn all about reading rain gauges and anemometers.  This sounded like a great deal, and it was.  There was only one thing missing, being able to get the data into some kind of database so I could play with it.  Yes there was some pretty slick software that would upload the data to the AcuRite site, but you all know how I feel about relying on someone else for my data.  Also, the software ONLY runs on a Windows PC.  That means I have to keep a PC running all the time to get the data uploaded.  What?  This is the 21st century people, what's up with keeping a big old power hungry PC running 24/7?

I went looking for solutions and found a couple of interesting things.  A site called Weather Display <link> that had some nice software and a driver for Linux, but the poor developer had recently lost the source to his work <link>, and he wanted $70 bucks for it.  Another one, Valley Information Systems <link> who, it turns out, does the driver for the AcuRite software for the weather station.  V.I.S. had a forum as well <link>, so I prowled it and found out that they didn't have a Linux driver, and basically weren't interested in doing one.  I got this from a long rambling discussion with the author (you know me, I can't shut up) where he flat out told me it wasn't worth his time.  I spent several more hours looking for leads on a Linux driver with no luck at all.

So, I didn't want to spend $70 on a system that the source was being rebuilt from scratch for, and I didn't want to run it on a PC since that would tie up my machine and hook me to a wire 24/7; I guess the only solution was to write something myself.

But, what to do?  I could get a receiver and capture the RF sent from the 5 in 1 sensor I mounted on the roof:


(people tell me it looks like a rabbit setting up there)

I could also buy one of their 'bridge' devices.  This is basically an RF receiver hooked to an ethernet plug through some kind of processor.  This device was listed at $80, the same price as the weather station, and it wasn't even wireless.
It did meet the requirement of not having to have a PC plugged in and tethered to the display though.  I guess it's AcuRite's way of overcoming an obvious shortcoming ... for a price.

Or, I could try and conquer the USB interface to it on my Raspberry Pi.  Since I have years of experience dealing with RF in various forms, what do you think I did?  Right, I decided to hack into the USB interface.

The only problem is that I know exactly zero about USB and how to interact with it ... sigh; here I go again.

A while later, I had learned that there were several ways to interface with USB on the Pi, something called pyusb written in python, but there was so little documentation on it that frankly, I couldn't make heads or tails out of it.  There was libusb that had tons of documentation, but all of it was written in a language I couldn't understand --- USB jargonese.  There was something called libusbx, but it looked exactly like libusb.  There were a ton of examples, but nothing that showed how you could actually get data off the darn thing.  If I wanted to list the devices, I could find an example almost instantly, but actually get some data, basically nothing.

Fine, first I'd play with the USB, and get a feel for what was going on.  Maybe that would give me a few keywords that I could use in a better refined google search.  I learned all about lsusb and several other tools that can enumerate the devices hooked to a machine.  I looked at USB sniffers and how they worked.  I found out that HID stood for Human Interface Device and it had a special driver that was dealt with differently than other USB devices.  I learned all about how to modify udev rules to make a touchpad behave like a joystick; everything except how to get data from this weather station.

I decided to just start playing and see what happened.  I plugged the weather station into the Pi and started looking around.  The very first thing I found out was that a normal user on the Pi, like the user pi, can't read the device that is created when you plug the weather station in.  You can tell if this is the problem; look for the device in /dev and check its permissions

crw------- 1 root root 249, 0 Dec 31  1969 /dev/hidraw0

See, what happens is that the device isn't recognized by udev (the process that watches the USB bus) and it assigned the hidraw driver (the most basic generic driver of this type) to the device.  The hidraw driver defaulted like this can only be read and written to by root.  The situation was that the device would identify itself as a HID device, but it wasn't recognized properly and udev would default to the hidraw device driver.  OK, I started looking for how to use this device.  It turns out there's quite a bit of information on the hidraw driver and how to use it.  So I put together some code and gave it a try.

Nothing worked as advertised.  Nothing.  The weather station might have been assigned the hidraw driver, but it wouldn't act like one to Linux.  That was a bust.  I wrote some code to read the various descriptors from the device and noticed a couple of errors in its implementation that might have been causing the problem, but nothing that could solve the problem.  All of this was running as root on the Pi since a normal user couldn't talk to the device.  Yes, I could change the permissions on the device, but that would only last until it was unplugged.  When the device was plugged back in, the device was created root-only all over again.  I could also set the bit such that the code would run as root, but that could create a monster if I messed up in the code.  I really didn't want to risk that.  And, I was getting tired of dealing with the problems of a default device that wouldn't work, so I decided to conquer that item first.

When dealing with USB, the devices can come and go at will.  When we unplug the device, the system notices it and dismantles the connection to it.  Everything is restored when the device is plugged back in.  Although, the device may come up with a different mount point which means the device name changes.  For example on my Pi, since I don't have a keyboard or mouse, the weather station shows up as hidraw0.  On some other machine it could be hidraw1, hidraw2, whatever the next number after the devices already plugged in happens to be.  That's just the nature of USB.  This stuff works by a set of rules that are in the secret directory /etc/udev/rules.d.

I'm not going to drive you nuts with an in-depth discussion of udev and how it works, but I will tell you how to fix the permission problems so the device can be read by a normal Linux user (not root).

First, you have to find out what the OS sees the device as.  This is logged in the system log when you plug the device in and udev does its thing.  So, plug the weather station into the Pi and do the command 'dmesg'. You're interested in a device from Chaney Instrument, so look in the dmesg output for something like the following:
[    2.399920] usb 1-1.2: new low-speed USB device number 4 using dwc_otg
[ 2.536425] usb 1-1.2: New USB device found, idVendor=24c0, idProduct=0003
[ 2.538128] usb 1-1.2: New USB device strings: Mfr=0, Product=2, SerialNumber=0
[ 2.541669] usb 1-1.2: Product: Chaney Instrument
[ 3.127499] udevd[156]: starting version 175
[ 5.553287] bcm2708-i2s bcm2708-i2s.0: Failed to create debugfs directory
[ 12.559829] hid-generic 0003:24C0:0003.0001: usb_submit_urb(ctrl) failed: -1
[ 12.561497] hid-generic 0003:24C0:0003.0001: timeout initializing reports
[ 12.563452] input: Chaney Instrument as /devices/platform/bcm2708_usb/usb1/1-1/1-1.2/1-1.2:1.0/input/input0
[ 12.569076] hid-generic 0003:24C0:0003.0001: input,hidraw0: USB HID v1.11 Device [Chaney Instrument] on usb-bcm2708_usb-1.2/input0
I used the command 'dmesg | grep -i usb' to capture this, and it means the HID USB driver grabbed the device because it is listed as type 3 (HID) in its descriptor.  That needs to be fixed, but first a tiny bit about cmdline.txt.  This is used to feed parameters into the Linux boot up process, for example many folk have a cmdline.txt file like this when they first boot their Pi:

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait

My Pi, before this change looks like this:

dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait

I had to remove the console entries for other projects, so the command is smaller.  To stop the HID driver binding to the weather station USB port from happening, add this phrase to /boot/cmdline.txt, but be careful, messing up this line may cause the Pi not to boot back up.

usbhid.quirks=0xXXXX:0xYYYY:0x04

Replace XXXX with the idVendor above and the YYYY with the idProduct above, the 0x04 is HID_QUIRK_IGNORE, leave that alone. Just put the phrase somewhere in the line, you'll see what I mean when you look at it.  It seems the HID driver has allowances for 'quirks', things that just don't work the way you expect them to - like this device.  After I made the changes, my /boot/cmdline.txt looks like this:

dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline usbhid.quirks=0x24c0:0x0003:0x04 rootwait

Now you need to make a new usb device usable by someone other than root for the weather station, so go to the directory /etc/udev/rules.d (remember that from above) and create a file named 10-local.rules which contains the line:

SUBSYSTEM=="usb", ATTRS{idVendor}=="XXXX", ATTRS{idProduct}=="YYYY", SYMLINK="weather" MODE="666",GROUP="users"

The reason for the obscure file name is due to the way the file is read. The XXXX and YYYY are replaced with the idVendor and idProduct as described above. This will cause udev to make the device usable by a normal user.   My /etc/udev/rules.d/10-local.rules looks like this:

SUBSYSTEM=="usb", ATTRS{idVendor}=="24c0", ATTRS{idProduct}=="0003", SYMLINK="weather" MODE="666",GROUP="users"

Yours will probably look exactly the same since the vendor and product number won't change.  This rule will set the permission and create a symbolic link in the /dev/directory called 'weather' for the station.  Be careful with the equal and double equal signs; one is a comparison and the other is an assignment.  I fought this mistake for a hour before I noticed what I did wrong.  After you reboot the machine to make it read the changes, take a look at /dev/weather:

lrwxrwxrwx 1 root root 15 Dec 31  1969 /dev/weather -> bus/usb/001/004

The permissions are right and the link is into the USB bus structure.  To test that a normal user can get to the file, do 'hexdump < /dev/weather' while logged in as pi and make sure you get something on the screen.  The reason for hexdump is to prevent control characters from getting to the screen and messing up settings.  You're not really interested in what the data is, just that you can get it.

Once all these usb related changes are done you're all set with an entry in /dev called weather that can be read and written by a normal user.  You can unplug the device and /dev/weather will disappear; plug it back in and /dev/weather will come back.  This is the way it's supposed to work.  I actually did a 'tail -f  /var/log/messages' and watched as I plugged and unplugged the weather station.  It had taken a lot of research and experimentation to get to this point, and I wanted to enjoy it.  DO NOT DO THIS!  I now have a plug on the back of the weather station console that is flaky.  It's ok to unplug it, but don't be like me.

So, when do we get to the part about actually reading the darn thing.  Maybe in the next post, but there's much more to come.  We have to get libusb sorted out; that's a story in itself and will probably bore you to tears.  Then we have to discuss how to actually implement a user-space-usb-reader.  That's part of the jargon that I picked up learning about this.  Eventually, we'll actually decode the bit stream the weather station provides; yes, it's a dog gone bitstream.

I promise, I won't take months and months to get the rest of it up, but I don't want to make these too long.

Part 2 of this is now here <link>

AcuRite Weather Station, Raspberry Pi and a USB interface; Part 2

$
0
0
In part 1 <link> I described the AcuRite weather station I have and how I wanted to hook into the USB interface so I could have the data and do anything I wanted to with it.  I also gave instructions on how to modify certain files such that you have a device that a user can read.  Now, we have to hook some code to the USB port and actually deal with the device.  But first, we have to get a library that will help us along.

From my research, I only found one source that had the things I needed and enough documentation to actually stand a chance of implementing it, libusb.  Libusb is a multiplatform library to interface to a USB device from user space.  Let's discuss this a tiny bit since there are better sites out there to get this information from, but you need a tiny bit of information to understand what comes next.

On unix, there's users and the kernal.  The kernal is the ultimate controller of everything; it has access to all the facilities and takes care of portioning them out and keeping track of what goes on.  Users can't mess with the things the kernal controls, and that's a good thing.  We can't kill ourselves and have to restore the system from backup if we stay away from the things the kernal should be dealing with.  Unfortunately, the USB buss belongs to the kernal.  So, we need to use kernal interface routines to control the USB devices and those are really hard to understand.  Libusb gives us access to those interfaces in a reasonably simpler (not simple) way.  USB is very complex and nothing can make it easy, but it can, at least, be possible.

So where do we get libusb?  It's already on the machine, but the version there is several generations back and hard to link to.  I searched around and found out a few things that can help you keep from being as confused as I was when I first started this.

1. Almost all the web sites out there deal with a version of libusb that is outdated.  Libusb has gone through several changes and massive improvement this year (2014) and most sites don't work with the newest stuff.

2. As always, the newest stuff is more stable and has greater capabilities.

3.  In your searching, you'll run across libusb and libusbx.  These are two different things; libusbx is a fork of libusb that extended certain things.  The two were merged back together to form a new libusb that serves both projects.  So, web sites that talk about one of them, probably won't work with the latest.

4. You can't automatically install the latest libusb.  The packages on the Raspberry Pi are behind several versions.

5. Using the latest version, lots of the examples on the web won't even compile.

6. You WANT the latest version.

This all means you will have to get the latest version and compile it yourself.  There are two ways of doing this: get a download from the libusb website and install it, or get the source from github and install that.  I did both and they both worked, but I decided the best way for me is to grab it from github, because I know that represents what I want right now.  This may not be true in a couple of months, so consider a download from the libusb home site.

I used libusb-1.0.19 as my base.  You can get it from their (new) home site <link> , just click on 'download' and choose latest.  In a few weeks, this might not get you this version, so decide if you want the latest (I would) or if you want this one.  If you chose to go with 1.0.19, it'll be under the 'Previous Releases' area.  Download it and put it on the Pi somewhere.  Now you have to compile it.

You'll have to extract the sources; just use tar xvf filename and watch it fly by.  Then go into the new directory and read the file 'INSTALL'.  This is where they keep the build instructions, and for me, it was a simple configure, make, make install sequence and everything was done.  However, there's two caveats to this.  Make sure your Pi is up to date and install libudev-dev.

I had restored my Pi from backup first and the backup was from a while ago.  This meant that the software on the machine was months behind.  This gave me no end of trouble which I fixed by simply updating the machine:

sudo apt-get update
sudo apt-get upgrade

The trick to this is to do the update about 30 minutes before bedtime, then do the upgrade just before going to bed.  I don't know how long the upgrade took, I was asleep when it finished, but it took a long time.  You don't want to watch it, it's boring.  There is a confirmation prompt though, so wait for it to ask if you really want to do this, answer yes, and go to bed.  This works by the update going out and reading the various pacakages to see what the very latest is; it doesn't actually change anything except that.  The upgrade actually gets the new stuff and installs it.  So, the upgrade may have a whole lot to do and could take a while.

So, update, upgrade, get libusb but don't compile it.  You're still not ready.  Now you need to get libudev-dev.  This is an interface to the support functions that udev provides (remember udev from the last post?), and libusb needs them.  Fortunately, you can use apt-get to grab these:

sudo apt-get install libudev-dev

So, are you finally ready to compile libusb?  After reviewing the instruction in INSTALL in the directory you got with the release, go ahead and make the library.  However, be sure to sudo each step of the process.  If you don't errors happen and with the huge amount of stuff printed, it's real easy to miss something.  For me, it was:

sudo ./configure
sudo make
sudo make install

Now you're finally ready to actually do something.  Libusb is a 'c' library, so I used c code to deal with it.  I know, python is my usual preference, but when in Rome ...  Remember, most of the examples on the web won't work with the latest libusb, so it was a bit tough getting things to work.  I finally had to just step through some of them a line at a time and adapt them to work with the latest.  It was a royal pain, but it did help me to learn a lot about what I was doing.

So, here's the first code to deal with the USB device that we're going to eventually use:

usbexample1.c
/*
Experimentation with a USB interface to the Acu-Rite 5 in 1 Weatherstation
specifically for the Raspberry Pi.

Because there likely to be a version of libusb and the associated header file
on a Pi, use the command line below to build it since the build of libusb-1.0.19
places things in /usr/local

cc usbexample1.c -L/usr/local/lib -lusb-1.0
use ldd weatherstation to check which libraries are linked in.
If you still have trouble with compilation, remember that cc has a -v
parameter that can help you unwind what is happening.
*/

#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <time.h>
#include <unistd.h>
#include <sys/time.h>
#include <libusb-1.0/libusb.h>

// The vendor id and product number for the AcuRite 5 in 1 weather head.
#define VENDOR 0x24c0
#define PRODUCT 0x0003

// I store things about the weather device USB connection here.
struct {
libusb_device *device;
libusb_device_handle *handle;
int verbose;
} weatherStation;

/*
This code is related to dealing with the USB device
*/
// This searches the USB bus tree to find the device
int findDevice(libusb_device **devs)
{
libusb_device *dev;
int err = 0, i = 0, j = 0;
uint8_t path[8];

while ((dev = devs[i++]) != NULL) {
struct libusb_device_descriptor desc;
int r = libusb_get_device_descriptor(dev, &desc);
if (r < 0) {
fprintf(stderr,"Couldn't get device descriptor, %s\n", libusb_strerror(err));
return(1);
}

fprintf(stderr,"%04x:%04x (bus %d, device %d)",
desc.idVendor, desc.idProduct,
libusb_get_bus_number(dev), libusb_get_device_address(dev));

//r = libusb_get_port_numbers(dev, path, sizeof(path));
//if (r > 0) {
// fprintf(stderr," path: %d", path[0]);
// for (j = 1; j < r; j++)
// fprintf(stderr,".%d", path[j]);
//}
fprintf(stderr,"\n");

if (desc.idVendor == VENDOR && desc.idProduct == PRODUCT){
fprintf(stderr,"Found the one I want\n");
weatherStation.device = dev;
return (1);
}
}
return(0);
}

// to handle testing and try to be clean about closing the USB device,
// I'll catch the signal and close off.

// I do several things here that aren't strictly necessary. As I learned about
// libusb, I tried things and also used various techniques to learn about the
// weatherstation's implementation. I left a lot of it in here in case I needed to
// use it later. Someone may find it useful to hack into some other device.
int main(void)
{
libusb_device **devs;
int r, err;
ssize_t cnt;

err = libusb_init(NULL);
if (err < 0){
fprintf(stderr,"Couldn't init usblib, %s\n", libusb_strerror(err));
exit(1);
}
// This is where you can get debug output from libusb.
// just set it to LIBUSB_LOG_LEVEL_DEBUG
libusb_set_debug(NULL, LIBUSB_LOG_LEVEL_INFO);

cnt = libusb_get_device_list(NULL, &devs);
if (cnt < 0){
fprintf(stderr,"Couldn't get device list, %s\n", libusb_strerror(err));
exit(1);
}
// got get the device; the device handle is saved in weatherStation struct.
if (!findDevice(devs)){
fprintf(stderr,"Couldn't find the device\n");
exit(1);
}
// Now I've found the weather station and can start to try stuff
// So, I'll get the device descriptor
struct libusb_device_descriptor deviceDesc;
err = libusb_get_device_descriptor(weatherStation.device, &deviceDesc);
if (err){
fprintf(stderr,"Couldn't get device descriptor, %s\n", libusb_strerror(err));
exit(1);
}
fprintf(stderr,"got the device descriptor back\n");

// Open the device and save the handle in the weatherStation struct
err = libusb_open(weatherStation.device, &weatherStation.handle);
if (err){
fprintf(stderr,"Open failed, %s\n", libusb_strerror(err));
exit(1);
}
fprintf(stderr,"I was able to open it\n");
// Now that it's opened, I can free the list of all devices
libusb_free_device_list(devs, 1); // Documentation says to get rid of the list
// Once I have the device I need
fprintf(stderr,"Released the device list\n");
// Now I have to check to see if the kernal using udev has attached
// a driver to the device. If it has, it has to be detached so I can
// use the device.
if(libusb_kernel_driver_active(weatherStation.handle, 0) == 1) { //find out if kernel driver is attached
fprintf(stderr,"Kernal driver active\n");
if(libusb_detach_kernel_driver(weatherStation.handle, 0) == 0) //detach it
fprintf(stderr,"Kernel Driver Detached!\n");
}
int activeConfig;
err =libusb_get_configuration (weatherStation.handle, &activeConfig);
if (err){
fprintf(stderr,"Can't get current active configuration, %s\n", libusb_strerror(err));;
exit(1);
}
fprintf(stderr,"Currently active configuration is %d\n", activeConfig);

err = libusb_set_configuration (weatherStation.handle, 1);
if (err){
fprintf(stderr,"Cannot set configuration, %s\n", libusb_strerror(err));;
exit(1);
}
fprintf(stderr,"Just did the set configuration\n");


err = libusb_claim_interface(weatherStation.handle, 0); //claim interface 0 (the first) of device (mine had jsut 1)
if(err) {
fprintf(stderr,"Cannot claim interface, %s\n", libusb_strerror(err));
exit(1);
}
fprintf(stderr,"Claimed Interface\n");
fprintf(stderr,"Number of configurations: %d\n",deviceDesc.bNumConfigurations);
struct libusb_config_descriptor *config;
libusb_get_config_descriptor(weatherStation.device, 0, &config);
fprintf(stderr,"Number of Interfaces: %d\n",(int)config->bNumInterfaces);
// I know, the device only has one interface, but I wanted this code
// to serve as a reference for some future hack into some other device,
// so I put this loop to show the other interfaces that may
// be there. And, like most of this module, I stole the ideas from
// somewhere, but I can't remember where (I guess it's google overload)
const struct libusb_interface *inter;
const struct libusb_interface_descriptor *interdesc;
const struct libusb_endpoint_descriptor *epdesc;
int i, j, k;
for(i=0; i<(int)config->bNumInterfaces; i++) {
inter = &config->interface[i];
fprintf(stderr,"Number of alternate settings: %d\n", inter->num_altsetting);
for(j=0; j < inter->num_altsetting; j++) {
interdesc = &inter->altsetting[j];
fprintf(stderr,"Interface Number: %d\n", (int)interdesc->bInterfaceNumber);
fprintf(stderr,"Number of endpoints: %d\n", (int)interdesc->bNumEndpoints);
for(k=0; k < (int)interdesc->bNumEndpoints; k++) {
epdesc = &interdesc->endpoint[k];
fprintf(stderr,"Descriptor Type: %d\n",(int)epdesc->bDescriptorType);
fprintf(stderr,"Endpoint Address: 0x%0.2X\n",(int)epdesc->bEndpointAddress);
// Below is how to tell which direction the
// endpoint is supposed to work. It's the high order bit
// in the endpoint address. I guess they wanted to hide it.
fprintf(stderr," Direction is ");
if ((int)epdesc->bEndpointAddress & LIBUSB_ENDPOINT_IN != 0)
fprintf(stderr," In (device to host)");
else
fprintf(stderr," Out (host to device)");
fprintf(stderr,"\n");
}
}
}
//OK, done with it, close off and let it go.
fprintf(stderr,"Done with device, release and close it\n");
err = libusb_release_interface(weatherStation.handle, 0); //release the claimed interface
if(err) {
fprintf(stderr,"Couldn't release interface, %s\n", libusb_strerror(err));
exit(1);
}
libusb_close(weatherStation.handle);
libusb_exit(NULL);
exit(0);
}

Remember I told you that the installation process of libusb puts it in /usr/local?  Well, it's a little bit worse than that.  The actual directory it puts the header file libusb.h in is /usr/local/include/libusb-1.0 which makes it a bit more complicated to deal with.  In the source file above I used the line:

#include <libusb-1.0/libusb.h>

They also put the libraries we need in the directory /usr/local/lib, which is pretty normal.  These things make the command line to the compiler:

cc  usbexample1.c -L/usr/local/lib -lusb-1.0

because the compiler defaults to look in usr/local/include for the libusb.h file, but you have to get specific for the library.  Pay attention here, I fought this for a while because I was picking up the wrong library and some symbols weren't defined.    If you get into trouble and just can't figure out what is going wrong, the compiler has a -v flag that will print what it's doing step by step.  Carefully read this output and you should see where you're having a problem.  It outputs a lot of data though, so it isn't easy.

cc  -v usbexample1.c -L/usr/local/lib -lusb-1.0

Once you get it to compile, you'll be left with an a.out file that you can run to see what happens.  Just type in ./a.out and look at the output:

pi@deserthome:~/src$ ./a.out
24c0:0003 (bus 1, device 4)
Found the one I want
got the device descriptor back
I was able to open it
Released the device list
Currently active configuration is 1
Just did the set configuration
Claimed Interface
Number of configurations: 1
Number of Interfaces: 1
Number of alternate settings: 1
Interface Number: 0
Number of endpoints: 1
Descriptor Type: 5
Endpoint Address: 0x81
Direction is In (device to host)
Done with device, release and close it

This tells you that the code was able to find the device based on the vendor and id number, then through various arcane machinations, it was able to open it.  Then it just released it and closed it.

I put a lot of comments in the file, so you should be able to follow along and use the libusb api documentation <link> to understand it more fully.  Also notice that I sent all output to stderr.  For those of you that didn't understand that, Linux has both stderr and stdout.  Stdout is the normal place you send text you want to read and stderr is for errors.  There's a good reason for this, and I'll tell you all about it in the next post.

I will have the code above available on github soon, I just haven't done it yet.  So, if you're in a hurry, do a little copy and paste to create the file.  This code can also serve as a current (sort of, since things change pretty quickly) example of how to get to and reserve a USB device.  In the next installment I get more specific to the weather station.

So now we have the Pi set up with a USB device that is hooked to the weather station and some code that can attach to it and open it.  Next we have to read the data and decode it.

Have fun.

Part three of this series is here <link>.

AcuRite Weather Station, Raspberry Pi and a USB interface; Part 3

$
0
0
Part 2 of this is here <link>, and part 1 is here <link>.

Well, we've talked about the weather station, how to create a usb device for it and how to get a device driver that can talk to it.  Now we have to get the data off the usb and do something with it.

The usb device has two reports it provides, they're called Report 1 and Report 2 (duh).  R1 is a few bytes long and R2 is longer.  R1 has all the data from the weather head in it while R2 appears to have data gathered by the console.  For my purposes, I don't need anything from the console except the barometric pressure.  The sensor for that is inside the console piece and I haven't figured out how to get that part yet.

Yes, let me admit right up front I haven't figured out the barometric pressure reading.  The way you break into these things is to read the data, change something, look at the change, repeat until you have it figured out.  Barometric pressure doesn't change fast or far enough for me to have that ... yet.  I'll get it at some point.

However, wind speed, direction, a rain counter, temperature and humidity are all available and we can get that stuff from the device that's up on the roof.  It's really cool to be able to see that stuff on the Pi, so let's deal with R1 and save R2 for sometime later.  Here's the two reports that we can get out of the device:

R1 - 01 C0 5C 71 00 05 00 0C 03 FF
R1 - 01 C0 5C 78 00 08 1F 53 03 FF
R2 - 02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 01 47 68

R1 is 10 bytes long and R2 is 25 bytes long.  In each message the first byte is simply the report number and the last byte is the end marker, so the actual data for the two R1's are:

C0 5C 71 00 05 00 0C 03
C0 5C 78 00 08 1F 53 03

It's complicated by the fact that there are two types of R1 messages.  Which type each one is, is determined by the low order half of byte 2 -- remember, we start counting from 0.  In the first case, the lower 4 bits of byte 2 are 0001, this message contains wind speed, wind direction and rain counter.  For the second example, the lower 4 bits of byte 2 are 1000 and this message contains wind speed, temp and relative humidity.  To test for it in code:

if ((data[2] & 0x0f) == 8){ // this has wind speed, temp and relative humidity

and

if ((data[2] & 0x0f) == 1){ // this has wind speed, direction and rainfall

We've just obtained the readings of all five of the sensors in the 5 in 1 sensor head for the weatherstation.  Next we have to pick the bits out of each of them corresponding to the sensors reported.  So, I'll decode an item; the wind direction is the low order half of byte 4 (counting from 0) in message type 1, so the value would be 5.  Look above in the data, byte 2 is 71 so the low order half is 1 which means it holds wind speed, direction and rainfall; see how it works?

The 5 is one of the sixteen cardinal directions starting with NNW being 0 counting counter-clockwise around the compass and ending with N being 15.  That makes this SW, and 6 would be SSW and so on.  To decode this into something readable, just set up an array and index into it using the value you get from byte 4.

char *Direction[] = {"NNW", "NW", "WNW", "W", "WSW", "SW", "SSW", "S",
                                   "SSE", "SE",  "ESE", "E", "ENE", "NE", "NNE", "N" };

And then:

char *direction = Direction[data[4] & 0x0f];

Would give you the string "SW".  Not too hard, so let's look at something a bit harder to decode, the temperature.  Temperature is held in bytes 4 and 5 of message type 8.  Looking above at the data, the one with byte 2 = 78 (78 & 0x0f = 8) bytes 4 and five have the values 08 1F.  With these we have to combine the lower order 4 bits of byte 4 with the 7 lowest bits of byte 5.  The code for this looks like this:

    int highpart = (data[4] & 0x0f) << 7;
    int lowpart = data[5] & 0x7f;
    float combined = highpart | lowpart;

Notice how we can get a floating point number out of this?  We have to use the conversion capabilities of the compiler to make it easy, but it works nicely.  So in this case we get 08 & 0f which give us 0x08 then we shift it left 7 bits to result in 0x400.  Then we take the 0x1F & 0x7F to get 0x1F. Combining them 0x400 | 0x1f gets us 0x41F which is decimal 1055.  That seems a bit high, even for Arizona, so subtract 400 and divide by 10 to give a float value of 65.5 degrees Fahrenheit.  Fortunately, the compiler will do this by:

float temperature = (combined - 400) / 10.0;

Are you starting to see how this works?  I decoded the rest of them and the code that comes later will give you all the details for the other sensor data.  Never mind that every single thing is carefully designed to be confusing.  The compass cardinals count counter-clockwise; all the high order bits of the sensor data are for parity, not values; The readings have an offset of 400; etc.  This stuff is just details that, once figured out and coded, aren't even relevant any more.  You only have to be confused once, and I did that for you already.

We have the ability to decode the sensors, but how did I get the data?  Remember in the last episode I opened and closed the USB device but didn't read it?  It's time for me to show you how to read it.  With a USB device, you don't read it, you 'transfer' it; this is what it looks like:

actual = libusb_control_transfer(weatherStation.handle,
                    LIBUSB_REQUEST_TYPE_CLASS |
                    LIBUSB_RECIPIENT_INTERFACE |
                    LIBUSB_ENDPOINT_IN,
                    //These bytes were stolen with a USB sniffer
                    0x01,0x0100+whichOne,0,
                    data, 50, 10000);

We had already found the device, opened it and reserved it use for ourselves, so this transfer is the last piece.  The bits above are documented with libusb and the numeric values were totally stolen by sniffing the USB interaction and doing some copy and paste.  The very last three items in the call are the pointer to where the data is going to wind up, the maximum length acceptable and a maximum wait time of 10 seconds (in milliseconds).  Yep, that's all there is to it getting the data.  I didn't want to hang up in a hard loop banging the USB port as fast as I could because it wouldn't leave the weather station any time to actually process the incoming data from the sensor head.  I set up a really simple timer to read the USB port periodically to get the report R1 and process it into usable data.

I'm not real proud of the way I timed it, but it works.  I put the code in to read both of the reports, R1 and R2 because I haven't given up on the barometric pressure; I totally will decode that someday.  Once I got the code to read the data, convert it and print it, I decided to fancy it up a bit.  I like working on this kind of stuff in python, so I made the 'c' code send its output to stderr and stdout.  Stderr gets the informational, and error messages while stdout gets a ... you guessed it ... JSON ... message that can be read into python and further processed.  The idea is that you pipe one process into another using the Linux model.  First the code that I'm currently running to get a feel for what I want to do in the future:

/*
Experimentation with a USB interface to the Acu-Rite 5 in 1 Weatherstation
specifically for the Raspberry Pi. The code may work on other systems, but I
don't have one of those. Contrary to other folk's thinking, I don't care if
this ever runs on a Windows PC or an Apple anything.

I specifically used a model 2032 display with the sensor that I picked
up at one of those warehouse stores. The display has a usb plug on it and
I thought it might be possible to read the usb port and massage the data myself.

This code represents the result of that effort.

I gathered ideas from all over the web. I use the latest (for this second)
libusb and about a thousand examples of various things that other people have
done and posted about. Frankly, I simply can't remember all of them, so please,
don't be offended if you see your ideas somewhere in here and it isn't attributed.

I simply lost track of where I found what.

This module relies on libusb version 1.0.19 which, at this time, can only be
compiled from source on the raspberry pi.

Because there likely to be a version of libusb and the associated header file
on a Pi, use the command line below to build it since the build of libusb-1.0.19
places things in /usr/local

cc -o weatherstation weatherstation.c -L/usr/local/lib -lusb-1.0
use ldd weatherstation to check which libraries are linked in.
If you still have trouble with compilation, remember that cc has a -v
parameter that can help you unwind what is happening.
*/

#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <time.h>
#include <unistd.h>
#include <sys/time.h>
#include <libusb-1.0/libusb.h>

// The vendor id and product number for the AcuRite 5 in 1 weather head.
#define VENDOR 0x24c0
#define PRODUCT 0x0003

// I store things about the weather device USB connection here.
struct {
libusb_device *device;
libusb_device_handle *handle;
int verbose;
} weatherStation;
// These are the sensors the the 5 in 1 weather head provides
struct {
float windSpeed;
time_t wsTime;
int windDirection;
time_t wdTime;
float temperature;
time_t tTime;
int humidity;
time_t hTime;
int rainCounter;
time_t rcTime;
} weatherData;

// This is just a function prototype for the compiler
void closeUpAndLeave();

// I want to catch control-C and close down gracefully
void sig_handler(int signo)
{
if (signo == SIGINT)
fprintf(stderr,"Shutting down ...\n");
closeUpAndLeave();
}

/*
This tiny thing simply takes the data and prints it so we can see it
*/
// Array to translate the integer direction provided to text
char *Direction[] = {
"NNW",
"NW",
"WNW",
"W",
"WSW",
"SW",
"SSW",
"S",
"SSE",
"SE",
"ESE",
"E",
"ENE",
"NE",
"NNE",
"N" };

void showit(){

fprintf(stdout, "{\"windSpeed\":{\"WS\":\"%.1f\",\"t\":\"%d\"},"
"\"windDirection\":{\"WD\":\"%s\",\"t\":\"%d\"},"
"\"temperature\":{\"T\":\"%.1f\",\"t\":\"%d\"},"
"\"humidity\":{\"H\":\"%d\",\"t\":\"%d\"},"
"\"rainCounter\":{\"RC\":\"%d\",\"t\":\"%d\"}}\n",
weatherData.windSpeed, weatherData.wsTime,
Direction[weatherData.windDirection],weatherData.wdTime,
weatherData.temperature, weatherData.tTime,
weatherData.humidity, weatherData.hTime,
weatherData.rainCounter, weatherData.rcTime);
fflush(stdout);
}
/*
This code translates the data from the 5 in 1 sensors to something
that can be used by a human.
*/
float getWindSpeed(char *data){
int leftSide = (data[3] & 0x1f) << 3;
int rightSide = data[4] & 0x70 >> 4;
// Yes, I use mph, never got used to kilometers.
return((float)(leftSide | rightSide) * 0.62);
}
int getWindDirection(char *data){
return(data[4] & 0x0f);
}
float getTemp(char *data){
// This item spans bytes, have to reconstruct it
int leftSide = (data[4] & 0x0f) << 7;
int rightSide = data[5] & 0x7f;
float combined = leftSide | rightSide;
return((combined - 400) / 10.0);
}
int getHumidity(char *data){
int howWet = data[6] &0x7f;
return(howWet);
}
int getRainCount(char *data){
int count = data[6] &0x7f;
return(count);
}
// Now that I have the data from the station, do something useful with it.
void decode(char *data, int length, int noisy){
//int i;
//for(i=0; i<length; i++){
// fprintf(stderr,"%0.2X ",data[i]);
//}
//fprintf(stderr,"\n"); */
time_t seconds = time (NULL);
//There are two varieties of data, both of them have wind speed
// first variety of the data
if ((data[2] & 0x0f) == 1){ // this has wind speed, direction and rainfall
if(noisy)
fprintf(stderr,"Wind Speed: %.1f ",getWindSpeed(data));
weatherData.windSpeed = getWindSpeed(data);
weatherData.wsTime = seconds;
if(noisy)
fprintf(stderr,"Wind Direction: %s ",Direction[getWindDirection(data)]);
weatherData.wdTime = seconds;
weatherData.windDirection = getWindDirection(data);
if(noisy){
fprintf(stderr,"Rain Counter: %d ",getRainCount(data));
fprintf(stderr,"\n");
}
weatherData.rainCounter = getRainCount(data);
weatherData.rcTime = seconds;
}
// this is the other variety
if ((data[2] & 0x0f) == 8){ // this has wind speed, temp and relative humidity
if(noisy)
fprintf(stderr,"Wind Speed: %.1f ",getWindSpeed(data));
weatherData.windSpeed = getWindSpeed(data);
weatherData.wsTime = seconds;
if(noisy)
fprintf(stderr,"Temperature: %.1f ",getTemp(data));
weatherData.temperature = getTemp(data);
weatherData.tTime = seconds;
if(noisy){
fprintf(stderr,"Humidity: %d ", getHumidity(data));
fprintf(stderr,"\n");
}
weatherData.humidity = getHumidity(data);
weatherData.hTime = seconds;
}
}
/*
This code is related to dealing with the USB device
*/
// This searches the USB bus tree to find the device
int findDevice(libusb_device **devs)
{
libusb_device *dev;
int err = 0, i = 0, j = 0;
uint8_t path[8];

while ((dev = devs[i++]) != NULL) {
struct libusb_device_descriptor desc;
int r = libusb_get_device_descriptor(dev, &desc);
if (r < 0) {
fprintf(stderr,"Couldn't get device descriptor, %s\n", libusb_strerror(err));
return(1);
}

fprintf(stderr,"%04x:%04x (bus %d, device %d)",
desc.idVendor, desc.idProduct,
libusb_get_bus_number(dev), libusb_get_device_address(dev));

//r = libusb_get_port_numbers(dev, path, sizeof(path));
//if (r > 0) {
// fprintf(stderr," path: %d", path[0]);
// for (j = 1; j < r; j++)
// fprintf(stderr,".%d", path[j]);
//}
fprintf(stderr,"\n");

if (desc.idVendor == VENDOR && desc.idProduct == PRODUCT){
fprintf(stderr,"Found the one I want\n");
weatherStation.device = dev;
return (1);
}
}
return(0);
}

// to handle testing and try to be clean about closing the USB device,
// I'll catch the signal and close off.
void closeUpAndLeave(){
//OK, done with it, close off and let it go.
fprintf(stderr,"Done with device, release and close it\n");
int err = libusb_release_interface(weatherStation.handle, 0); //release the claimed interface
if(err) {
fprintf(stderr,"Couldn't release interface, %s\n", libusb_strerror(err));
exit(1);
}
libusb_close(weatherStation.handle);
libusb_exit(NULL);
exit(0);
}

// This is where I read the USB device to get the latest data.
unsigned char data[50]; // where we want the data to go
int getit(int whichOne, int noisy){
int actual; // how many bytes were actually read

// The second parameter is bmRequestType and is a bitfield
// See http://www.beyondlogic.org/usbnutshell/usb6.shtml
// for the definitions of the various bits. With libusb, the
// #defines for these are at:
// http://libusb.sourceforge.net/api-1.0/group__misc.html#gga0b0933ae70744726cde11254c39fac91a20eca62c34d2d25be7e1776510184209
actual = libusb_control_transfer(weatherStation.handle,
LIBUSB_REQUEST_TYPE_CLASS | LIBUSB_RECIPIENT_INTERFACE | LIBUSB_ENDPOINT_IN,
//These bytes were stolen with a USB sniffer
0x01,0x0100+whichOne,0,
data, 50, 10000);
if (actual < 0){
fprintf(stderr,"Read didn't work for report %d, %s\n", whichOne, libusb_strerror(actual));
}
else {
// If you want both of the reports that the station provides,
// just allow for it. Right this second, I've found every thing
// I need in report 1. When I look further at report 2, this will
// change
//fprintf(stderr,"R%d:%d:", whichOne, actual);
//int i;
//for(i=0; i<actual; i++){
// fprintf(stderr,"%0.2X ",data[i]);
//}
//fprintf(stderr,"\n");
if (whichOne == 1)
// The actual data starts after the first byte
// The first byte is the report number returned by
// the usb read.
decode(&data[1], actual-1, noisy);
}
}
// I do several things here that aren't strictly necessary. As I learned about
// libusb, I tried things and also used various techniques to learn about the
// weatherstation's implementation. I left a lot of it in here in case I needed to
// use it later. Someone may find it useful to hack into some other device.
int main(int argc, char **argv)
{
char *usage = {"usage: %s -u -n\n"};
int libusbDebug = 0; //This will turn on the DEBUG for libusb
int noisy = 0; //This will print the packets as they come in
libusb_device **devs;
int r, err, c;
ssize_t cnt;

while ((c = getopt (argc, argv, "unh")) != -1)
switch (c){
case 'u':
libusbDebug = 1;
break;
case 'n':
noisy = 1;
break;
case 'h':
fprintf(stderr, usage, argv[0]);
case '?':
exit(1);
default:
exit(1);
}
fprintf (stderr,"libusbDebug = %d, noisy = %d\n", libusbDebug, noisy);

if (signal(SIGINT, sig_handler) == SIG_ERR)
fprintf(stderr,"Couldn't set up signal handler\n");
err = libusb_init(NULL);
if (err < 0){
fprintf(stderr,"Couldn't init usblib, %s\n", libusb_strerror(err));
exit(1);
}
// This is where you can get debug output from libusb.
// just set it to LIBUSB_LOG_LEVEL_DEBUG
if (libusbDebug)
libusb_set_debug(NULL, LIBUSB_LOG_LEVEL_DEBUG);
else
libusb_set_debug(NULL, LIBUSB_LOG_LEVEL_INFO);


cnt = libusb_get_device_list(NULL, &devs);
if (cnt < 0){
fprintf(stderr,"Couldn't get device list, %s\n", libusb_strerror(err));
exit(1);
}
// got get the device; the device handle is saved in weatherStation struct.
if (!findDevice(devs)){
fprintf(stderr,"Couldn't find the device\n");
exit(1);
}
// Now I've found the weather station and can start to try stuff
// So, I'll get the device descriptor
struct libusb_device_descriptor deviceDesc;
err = libusb_get_device_descriptor(weatherStation.device, &deviceDesc);
if (err){
fprintf(stderr,"Couldn't get device descriptor, %s\n", libusb_strerror(err));
exit(1);
}
fprintf(stderr,"got the device descriptor back\n");

// Open the device and save the handle in the weatherStation struct
err = libusb_open(weatherStation.device, &weatherStation.handle);
if (err){
fprintf(stderr,"Open failed, %s\n", libusb_strerror(err));
exit(1);
}
fprintf(stderr,"I was able to open it\n");
// There's a bug in either the usb library, the linux driver or the
// device itself. I suspect the usb driver, but don't know for sure.
// If you plug and unplug the weather station a few times, it will stop
// responding to reads. It also exhibits some strange behaviour to
// getting the configuration. I found out after a couple of days of
// experimenting that doing a clear-halt on the device while before it
// was opened it would clear the problem. So, I have one here and a
// little further down after it has been opened.
fprintf(stderr,"trying clear halt on endpoint %X ... ", 0x81);
err = libusb_clear_halt(weatherStation.handle, 0x81);
if (err){
fprintf(stderr,"clear halt crapped, %s Bug Detector\n", libusb_strerror(err));;
}
else {
fprintf(stderr,"OK\n");
}

// Now that it's opened, I can free the list of all devices
libusb_free_device_list(devs, 1); // Documentation says to get rid of the list
// Once I have the device I need
fprintf(stderr,"Released the device list\n");
// Now I have to check to see if the kernal using udev has attached
// a driver to the device. If it has, it has to be detached so I can
// use the device.
if(libusb_kernel_driver_active(weatherStation.handle, 0) == 1) { //find out if kernel driver is attached
fprintf(stderr,"Kernal driver active\n");
if(libusb_detach_kernel_driver(weatherStation.handle, 0) == 0) //detach it
fprintf(stderr,"Kernel Driver Detached!\n");
}

int activeConfig;
err =libusb_get_configuration (weatherStation.handle, &activeConfig);
if (err){
fprintf(stderr,"Can't get current active configuration, %s\n", libusb_strerror(err));;
exit(1);
}
fprintf(stderr,"Currently active configuration is %d\n", activeConfig);

if(activeConfig != 1){
err = libusb_set_configuration (weatherStation.handle, 1);
if (err){
fprintf(stderr,"Cannot set configuration, %s\n", libusb_strerror(err));;
exit(1);
}
fprintf(stderr,"Just did the set configuration\n");
}

err = libusb_claim_interface(weatherStation.handle, 0); //claim interface 0 (the first) of device (mine had jsut 1)
if(err) {
fprintf(stderr,"Cannot claim interface, %s\n", libusb_strerror(err));
exit(1);
}
fprintf(stderr,"Claimed Interface\n");
fprintf(stderr,"Number of configurations: %d\n",deviceDesc.bNumConfigurations);
struct libusb_config_descriptor *config;
libusb_get_config_descriptor(weatherStation.device, 0, &config);
fprintf(stderr,"Number of Interfaces: %d\n",(int)config->bNumInterfaces);
// I know, the device only has one interface, but I wanted this code
// to serve as a reference for some future hack into some other device,
// so I put this loop to show the other interfaces that may
// be there. And, like most of this module, I stole the ideas from
// somewhere, but I can't remember where (I guess it's google overload)
const struct libusb_interface *inter;
const struct libusb_interface_descriptor *interdesc;
const struct libusb_endpoint_descriptor *epdesc;
int i, j, k;
for(i=0; i<(int)config->bNumInterfaces; i++) {
inter = &config->interface[i];
fprintf(stderr,"Number of alternate settings: %d\n", inter->num_altsetting);
for(j=0; j < inter->num_altsetting; j++) {
interdesc = &inter->altsetting[j];
fprintf(stderr,"Interface Number: %d\n", (int)interdesc->bInterfaceNumber);
fprintf(stderr,"Number of endpoints: %d\n", (int)interdesc->bNumEndpoints);
for(k=0; k < (int)interdesc->bNumEndpoints; k++) {
epdesc = &interdesc->endpoint[k];
fprintf(stderr,"Descriptor Type: %d\n",(int)epdesc->bDescriptorType);
fprintf(stderr,"Endpoint Address: 0x%0.2X\n",(int)epdesc->bEndpointAddress);
// Below is how to tell which direction the
// endpoint is supposed to work. It's the high order bit
// in the endpoint address. I guess they wanted to hide it.
fprintf(stderr," Direction is ");
if ((int)epdesc->bEndpointAddress & LIBUSB_ENDPOINT_IN != 0)
fprintf(stderr," In (device to host)");
else
fprintf(stderr," Out (host to device)");
fprintf(stderr,"\n");
}
}
}
fprintf(stderr,"trying clear halt on endpoint %X ... ", (int)epdesc->bEndpointAddress);
err = libusb_clear_halt(weatherStation.handle, (int)epdesc->bEndpointAddress);
if (err){
fprintf(stderr,"clear halt crapped, %s SHUCKS\n", libusb_strerror(err));;
closeUpAndLeave();
}
else {
fprintf(stderr,"OK\n");
}

// So, for the weather station we now know it has one endpoint and it is set to
// send data to the host. Now we can experiment with that.
//
// I don't want to just hang up and read the reports as fast as I can, so
// I'll space them out a bit. It's weather, and it doesn't change very fast.
int tickcounter= 0;
while(1){
sleep(1);
if(tickcounter++ % 10 == 0){
getit(1, noisy);
}
if(tickcounter % 30 == 0){
getit(2, noisy);
}
if (tickcounter % 15 == 0){
showit();
}
}
}
Yes, I'll get it up on github so you can download it, I just want to finish this series before I put it there.  This compiles like the last example where we had to specify the library location:

cc -o weatherstation  weatherstation.c -L/usr/local/lib -lusb-1.0

I called it weatherstation, if you want a shorter name, feel free.  If you take a quick glance at the code you'll notice that it accepts parameters:

weatherstation -[unh]

-u will turn on libusb debug so you can see what is happening behind the scenes.
-n (noisy) will print the various decoded sensor data as it comes in
-h simple help for the options because I won't remember them

The output of the code looks like this combining without parameters:

libusbDebug = 0, noisy = 0
24c0:0003 (bus 1, device 6)
Found the one I want
got the device descriptor back
I was able to open it
trying clear halt on endpoint 81 ... OK
Released the device list
Currently active configuration is 1
Claimed Interface
Number of configurations: 1
Number of Interfaces: 1
Number of alternate settings: 1
Interface Number: 0
Number of endpoints: 1
Descriptor Type: 5
Endpoint Address: 0x81
Direction is In (device to host)
trying clear halt on endpoint 81 ... OK
{"windSpeed":{"WS":"0.0","t":"1417653478"},"windDirection":{"WD":"NNW","t":"0"},"temperature":{"T":"62.6","t":"1417653478"},"humidity":{"H":"92","t":"1417653478"},"rainCounter":{"RC":"0","t":"0"}}
{"windSpeed":{"WS":"3.7","t":"1417653488"},"windDirection":{"WD":"NNE","t":"1417653488"},"temperature":{"T":"62.6","t":"1417653478"},"humidity":{"H":"92","t":"1417653478"},"rainCounter":{"RC":"12","t":"1417653488"}}
^CShutting down ...
Done with device, release and close it

Notice that you get the informational stuff and errors will show up also, but if you use a redirect to get rid of the stderr output, it will look like this:

pi@deserthome2:~/src$ weatherstation 2>/dev/null
{"windSpeed":{"WS":"0.0","t":"1417653739"},"windDirection":{"WD":"SSE","t":"1417653739"},"temperature":{"T":"62.6","t":"1417653729"},"humidity":{"H":"92","t":"1417653729"},"rainCounter":{"RC":"12","t":"1417653739"}}
{"windSpeed":{"WS":"0.0","t":"1417653749"},"windDirection":{"WD":"SSE","t":"1417653749"},"temperature":{"T":"62.6","t":"1417653729"},"humidity":{"H":"92","t":"1417653729"},"rainCounter":{"RC":"12","t":"1417653749"}}
{"windSpeed":{"WS":"0.0","t":"1417653769"},"windDirection":{"WD":"SSE","t":"1417653749"},"temperature":{"T":"62.6","t":"1417653769"},"humidity":{"H":"92","t":"1417653769"},"rainCounter":{"RC":"12","t":"1417653749"}}

Notice that I used the line:

weatherstation 2>/dev/null

This will throw away any information or error, so you may want to send it to a file instead:

weatherstation 2>somefilename

But you can read up on that if you don't already have it down pat.  The JSON string has each device, its value and the time it was read.  So, suppose you pipe this into a little piece of python code that looks like this:

#!/usr/bin/python
import sys
import json
import time

buff = ''
while True:
try:
buff += sys.stdin.read(1)
if buff.endswith('\n'):
data = json.loads(buff[:-1])
print "On", time.strftime("%A, %B, %d at %H:%M:%S",time.localtime(float(data["windSpeed"]["t"]))),
print "the wind was blowing from the", data["windDirection"]["WD"],
print "at\n", data["windSpeed"]["WS"], "mph,",
print "and it is", data["temperature"]["T"], "degrees F Outside.",
print "The humidity is", data["humidity"]["H"], "percent",
print "and \nthe rain counter is", data["rainCounter"]["RC"],
print
sys.stdout.flush()
buff = ''
except KeyboardInterrupt:
sys.stdout.flush()
sys.exit()

Then you'll get a cute little update that looks like this:

On Wednesday, December, 03 at 17:51:01 the wind was blowing from the E at
0.0 mph, and it is 62.6 degrees F Outside. The humidity is 92 percent and
the rain counter is 12
On Wednesday, December, 03 at 17:51:21 the wind was blowing from the ENE at
2.5 mph, and it is 62.6 degrees F Outside. The humidity is 92 percent and
the rain counter is 12
On Wednesday, December, 03 at 17:51:31 the wind was blowing from the ENE at
2.5 mph, and it is 62.6 degrees F Outside. The humidity is 92 percent and
the rain counter is 12
On Wednesday, December, 03 at 17:51:51 the wind was blowing from the ENE at
0.0 mph, and it is 62.6 degrees F Outside. The humidity is 92 percent and
the rain counter is 12

I called the little python script 'readit.py', so the command looked like this:

 weatherstation 2>/dev/null | readit.py

Yes, I threw away the information and errors, trust me, when I run this for real, I'll be saving that stuff in /var/log/house and running logrotate on it.

There it is, the whole project to get the data, and do something with it.  Are you starting to understand why I cut it up into pieces?  You just had a primer on breaking into devices that they've been keeping secret, creating your very own USB driver for a new device in user space, using udev to create a custom file mount point specific to a vendor device, creating a piece of code that pipes its output so other pieces of code can work on it further, building a simple python script to format the data, did I forget anything?  Not bad for three settings.

Now, go forth and conquer.


Using CherryPy for interprocess Communication Part 3

$
0
0
Part 1 is here <link>, and part 2 is here <link>.

I've been devoting my project time to the weather station, and completely forgot about finishing my testing of using HTTP to communicate between processes.  It's time to finish the experiment and actually implement it for the various processes I have running on the Pi.  As I mentioned before, I want to do this so I can move any of the processes to a different machine when I finally outgrow the single Raspberry PI.  No, I'm not changing machines, I'm going to add others to the collection.

I started with the code that controls the Wemo light switches I have and let it run for a while to make sure there wasn't something weird going to pop up; it didn't.  So now is the time to make corresponding changes to the code that sends signals to control the lights.  Currently there are two places I do this: a process that runs and handles scheduled events around the house and the web interface I use to manually control and look at things.  I'll start with the event controller since it's the easiest piece of code.

Basically, it's just a matter of sending an HTML request to the address and port number I assigned to the Wemo code.  I use different port numbers so I can have multiple addresses on a single machine.  This way, the  regular web server watches on port 80 like all the web sites out there, and the control processes watch on whatever port I decided to put them on.  To do this and maintain versatility, I put the address in my .houserc file and read it when the process first comes up.  Here's the entry I have:

"wemocontrol":{
"ipAddress":"192.168.0.205",
"port": 51001},

I just read the file using the code I already posted here <link> and set the Cherry Pi server to listen to it.  Now I need to have some code to send to it.  Here's the code fragment I came up with:

#! /usr/bin/python
import datetime
import time
import urllib2
import BaseHTTPServer
import os, sys
lib_path = os.path.abspath('../house')
sys.path.append(lib_path)
from houseutils import lprint, getHouseValues, timer, checkTimer

def openSite(Url):
#print Url
try:
webHandle = urllib2.urlopen(Url, timeout=5) #if it doesn't answer in 5 seconds, it won't
except urllib2.HTTPError, e:
errorDesc = BaseHTTPServer.BaseHTTPRequestHandler.responses[e.code][0]
print "Error: cannot retrieve URL: " + str(e.code) + ": " + errorDesc
raise
except urllib2.URLError, e:
print "Error: cannot retrieve URL: " + e.reason[1]
raise
except urllib2.HTTPError as e:
print e.code
print e.read()
raise
except: #I kept getting strange errors when I was first testing it
e = sys.exc_info()[0]
print ("Odd Error: %s" % e )
raise
return webHandle

def talktoWemo(ip, command):
website = openSite("HTTP://" + ip + '/' + command)
# now read the status that came back from it
websiteHtml = website.read()
# After getting the status from the little web server,
# strip off the trailing cr,lf
# and separate the values into a list that can
# be used to tell what is going on
return websiteHtml

# Get the ip address and port number you want to use
# from the houserc file
ipAddress=getHouseValues()["wemocontrol"]["ipAddress"]
port = getHouseValues()["wemocontrol"]["port"]
wemoController = ipAddress + ":" + str(port)
print(talktoWemo(wemoController, "status"))

print(talktoWemo (wemoController, "pCommand?command=patioToggle"))

Well, it's actually more than a fragment, This code will actually control my patio light.  Each time I run it, it will toggle the switch.  It turned out to be pretty simple when done in python, I just used the normal facilities for reading a web site.  In there are also examples of how I read the address and port number from the .houserc file, and how I handle possible errors in the HTTP request.  I came up with that huge set of error conditions as I was learning how to send and receive HTTP; there's a lot that can go wrong and I hated having to look up the errors each time I messed something up.

Now I need to understand how to do the same thing from php so I can command it with a request over the internet.  I guess while I'm in there I'll add some code to read the .houserc file; here's how I send the HTML request to the Wemo control software:

function ipControl($whichOne,$command){
echo "<br />";
echo "URL will be: $whichOne/$command";
$response = file_get_contents("http://$whichOne/$command");
return(true);

};

This is how I read the .houserc file to get the address I assigned it:

$config = file_get_contents("/home/pi/.houserc");

Then in the case statement for controlling the lights I decode the JSON request and grab the address and port number I put in there.

case "lights":
$wemoIp = json_decode($config,true)["wemocontrol"]["ipAddress"];
$wemoPort = json_decode($config,true)["wemocontrol"]["port"];
$c = $commandParts[1];
ipControl("$wemoIp:$wemoPort", "pCommand?command=$c");
break;

This set of changes is merged into my php script that handles web commands, 'command.php'.  Yes, I managed to get it into github so you can grab the entire thing from there and see how it fits together.
The file is in the 'wwwsrc' directory and here's a link to the page <link>.

The very last thing is to include the fragment above into the scheduled event process.  This separate process handles events like: turn the outside lights on at 7PM; turn them off at 10PM, shut the pool motor off in case I forget, etc.  I haven't talked about this process much previously because it had several things in it that I didn't want to put on the web.  When I moved my various keys and such into the .houserc file, that problem disappeared, so this file has the changes above added to it and is called 'events.py' and is in the directory 'house' in the github page above.

So, one set of conrols is done.  I'll have to visit each of the others over time and make similar changes to them, but I won't be limited to one machine anymore.  To move a process to another machine, just change the address in the .houserc file; HTTP interactions don't care that the process is on another machine.

AcuRite Weather Station, Raspberry Pi and a USB interface On the Web Part 4

$
0
0
Part 3 of this is here <link>

After I finished up changing how my various processes communicate in the last post, I got back to the weather station.  Frankly, it's no fun reading lines of text, it's much better to put things in guages or charts.  I took the easy way out, and just put the entire JSON string I created in the data data base as a string, not separate records.

No, there wasn't any technical reason for this, I just didn't want to think about how to organize the schema to handle it.  It does mean that I haven't done anything with the rain fall counter though.  This part of the station is a simple accumulator that counts rainfall in 0.01 inch increments.  We have to save the measurement periodically and calculate from the latest reading to have it make sense.  I'll get to that some time this winter, but I have the wind direction and speed, temperature, and humidity working just fine.

It took some experimentation to get the JSON string out of the database and converted into values in PHP though.  Take a look at the code involved:

$ws = timedQuerySingle(
'select "json" from "weather";');
$db->close();
# The weather string is a pain, this is converting it, and
# since I can reuse variables and I'm tired of thinking up names
# I use the same name over and over again just to confuse
# anyone reading this.
# First, there an extra set of quotes around the string
$ws=substr($ws,1,strlen($ws)-2);
# Now I have to get rid of the \" that I had to use to put it in
# the database
$ws=str_replace("\\","",$ws);
#Now, convert the json into variables
$ws = json_decode($ws,true);

The entire string is surrounded by quotes and every single quote inside is 'escaped' with a backslash.  So, I cut out the middle between the quotes and then remove the backslashes.  This won't work if there's a backslash in the string, but I built the string and know what's in it.  Once I'm done with this, the various items in the string are available as a named array index. Here's how to refer to the four values temperature, wind speed, wind direction and humidity:

$ws["windSpeed"]["WS"]
$ws["windDirection"]["WD"]
$ws["humidity"]["H"]
$ws["temperature"]["T"]);

If you remember, I stored the time I recorded the reading with each item, so the times would be something like,

$ws["windDirection"]["t"]

for each measurement.  I don't know if I need it, but it's better to have it now while I'm figuring out what I want to keep.

To display it I added even more SteelSeries gauges to my display for the items.  I now looks like this:


Yes, it's a little busy, but I can glace at it and tell what's going on in a second; besides, it's mine, not some other piece of software that I have to mess with forever to get what I want.  I suspect it will get even more elaborate when I add in something for rainfall.  Heck, I may use one of the web authoring tools and really fancy up the display to impress people with, but this is adequate for taking a look at what it's like outside, and it's only a few seconds behind real time.  I'm actually pretty proud of it.

And yes, it's raining today; the humidity is usually around 12 or so.  The difference between the two temperatures is related to location.  The weather station is up on the roof, and the other temperature monitor is on a fence post at about 5 feet.  On sunny days I expect the temperature difference to be even larger.  As before, the code is available on github to grab and change to suit your needs.

Have fun.

Acurite Weather Station, Raspberry Pi, USB driver, I messed up, Part 5

$
0
0
Part 4 of this series is here <link>

Yep, I messed up, here's the story:  I was logging data so I could try and decode the barometric sensor and was getting nowhere, so I thought I'd take the console apart to see what kind of pressure sensor it had in it.  So, about an hour later I plugged it back in and it didn't work.

No, I didn't suspect I broke it, everything worked except the actual read.  The reads timed out and then started responding that I was sending the wrong command.  No code had changed, what could be wrong?  After the usual routine of changing cables, checking power, plugging it in and out a bunch of times, it still didn't work.  I moved the console to my other Pi and started taking apart the code.

About a day later, I had tried everything I could think of then it dawned on me that something might be wrong with the initialization.  I found the problem, and it was with the way I set it up.  I removed the attachment of the hidraw driver from the port, and it was providing the initialization to the kernal that allowed the weather station to work.  That was a bad thing, since without this, the kernal couldn't talk to the station.

When I took the command out of the command.txt file and prevented the hidraw driver from connecting, the communication path was fine.  But (there's always one of those), my code couldn't connect a second time.  The first time I plugged it in, everything was fine, but if the device was reset, it started failing again.  This turned out to be caused by my code disconnecting the hidraw driver and not putting it back so it could initialize the device again.

Fortunately, there's fix for that; usblib has a call that will detach the kernal driver automatically and restore it when we're done.  I added that and the station can be plugged in and work, then work again the next time.  While I was in there, I cleaned up some comments, removed the superfluous code that I left in, and generally cleaned up a bit.

It's working fine now and back to everyday use while I look further into the barometric pressure, but that's yet another post on this device.  Meanwhile, here's the updated code for the usb driver portion, and the changes will be in Github in an hour or so.

/*
Documentation at desert-home.com

This is the actual weather module I run for controlling the house, as such, this
may not be what you want. You may want other-stuff/weatherstation.c

Experimentation with a USB interface to the Acu-Rite 5 in 1 Weatherstation
specifically for the Raspberry Pi. The code may work on other systems, but I
don't have one of those. Contrary to other folk's thinking, I don't care if
this ever runs on a Windows PC or an Apple anything.

I specifically used a model 2032 display with the 5 in 1 sensor that I picked
up at one of those warehouse stores. The display has a usb plug on it and
I thought it might be possible to read the usb port and massage the data myself.

This code represents the result of that effort.

I gathered ideas from all over the web. I use the latest (for this second)
libusb and about a thousand examples of various things that other people have
done and posted about. Frankly, I simply can't remember all of them, so please,
don't be offended if you see your ideas somewhere in here and it isn't attributed.

I simply lost track of where I found what.

This module relies on libusb version 1.0.19 which, at this time, can only be
compiled from source on the raspberry pi.

Because there's likely to be a version of libusb and the associated header file
on a Pi, use the command line below to build it since the build of libusb-1.0.19
places things in /usr/local

cc -o weatherstation weatherstation.c -L/usr/local/lib -lusb-1.0
use ldd weatherstation to check which libraries are linked in.
If you still have trouble with compilation, remember that cc has a -v
parameter that can help you unwind what is happening.
*/

#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <time.h>
#include <unistd.h>
#include <sys/time.h>
#include <libusb-1.0/libusb.h>

// The vendor id and product number for the AcuRite 5 in 1 weather head.
#define VENDOR 0x24c0
#define PRODUCT 0x0003

// I store things about the weather device USB connection here.
struct {
libusb_device *device;
libusb_device_handle *handle;
} weatherStation;
// These are the sensors the the 5 in 1 weather head provides
struct {
float windSpeed;
time_t wsTime;
int windDirection;
time_t wdTime;
float temperature;
time_t tTime;
int humidity;
time_t hTime;
int rainCounter;
time_t rcTime;
} weatherData;

// This is just a function prototype for the compiler
void closeUpAndLeave();

// I want to catch control-C and close down gracefully
void sig_handler(int signo)
{
if (signo == SIGINT)
fprintf(stderr,"Shutting down ...\n");
closeUpAndLeave();
}

/*
This tiny thing simply takes the data and prints it so we can see it
*/
// Array to translate the integer direction provided to text
char *Direction[] = {
"NNW",
"NW",
"WNW",
"W",
"WSW",
"SW",
"SSW",
"S",
"SSE",
"SE",
"ESE",
"E",
"ENE",
"NE",
"NNE",
"N" };
// this is a bitmapped byte to tell if the various styles of reports have
// come in. Bit 0 is R1 first type, bit 2 is R1 type 2 and bit 3 is R2
// even though I don't use R2 yet
uint8_t reportsSeen = 0;

void showit(){

// make sure enough reports have come in before reporting
if( reportsSeen >= 3){ // Change this when report 3 is decoded
fprintf(stdout, "{\"windSpeed\":{\"WS\":\"%.1f\",\"t\":\"%d\"},"
"\"windDirection\":{\"WD\":\"%s\",\"t\":\"%d\"},"
"\"temperature\":{\"T\":\"%.1f\",\"t\":\"%d\"},"
"\"humidity\":{\"H\":\"%d\",\"t\":\"%d\"},"
"\"rainCounter\":{\"RC\":\"%d\",\"t\":\"%d\"}}\n",
weatherData.windSpeed, weatherData.wsTime,
Direction[weatherData.windDirection],weatherData.wdTime,
weatherData.temperature, weatherData.tTime,
weatherData.humidity, weatherData.hTime,
weatherData.rainCounter, weatherData.rcTime);
fflush(stdout);
}
}
/*
This code translates the data from the 5 in 1 sensors to something
that can be used by a human.
*/
float getWindSpeed(char *data){
int leftSide = (data[3] & 0x1f) << 3;
int rightSide = data[4] & 0x70 >> 4;
// Yes, I use mph, never got used to kilometers.
return((float)(leftSide | rightSide) * 0.62);
}
int getWindDirection(char *data){
return(data[4] & 0x0f);
}
float getTemp(char *data){
// This item spans bytes, have to reconstruct it
int leftSide = (data[4] & 0x0f) << 7;
int rightSide = data[5] & 0x7f;
float combined = leftSide | rightSide;
return((combined - 400) / 10.0);
}
int getHumidity(char *data){
int howWet = data[6] &0x7f;
return(howWet);
}
int getRainCount(char *data){
int count = data[6] &0x7f;
return(count);
}
// Now that I have the data from the station, do something useful with it.

void decode(char *data, int length, int noisy){
//int i;
//for(i=0; i<length; i++){
// fprintf(stderr,"%0.2X ",data[i]);
//}
//fprintf(stderr,"\n"); */
reportsSeen |= 0x04; // just pretend I've seen report 2 already
time_t seconds = time (NULL);
//There are two varieties of data, both of them have wind speed
// first variety of the data
if ((data[2] & 0x0f) == 1){ // this has wind speed, direction and rainfall
if(noisy)
fprintf(stderr,"Wind Speed: %.1f ",getWindSpeed(data));
weatherData.windSpeed = getWindSpeed(data);
weatherData.wsTime = seconds;
if(noisy)
fprintf(stderr,"Wind Direction: %s ",Direction[getWindDirection(data)]);
weatherData.wdTime = seconds;
weatherData.windDirection = getWindDirection(data);
if(noisy){
fprintf(stderr,"Rain Counter: %d ",getRainCount(data));
fprintf(stderr,"\n");
}
weatherData.rainCounter = getRainCount(data);
weatherData.rcTime = seconds;
reportsSeen |= 0x01; //I've seen report 1 now
}
// this is the other variety
if ((data[2] & 0x0f) == 8){ // this has wind speed, temp and relative humidity
if(noisy)
fprintf(stderr,"Wind Speed: %.1f ",getWindSpeed(data));
weatherData.windSpeed = getWindSpeed(data);
weatherData.wsTime = seconds;
if(noisy)
fprintf(stderr,"Temperature: %.1f ",getTemp(data));
weatherData.temperature = getTemp(data);
weatherData.tTime = seconds;
if(noisy){
fprintf(stderr,"Humidity: %d ", getHumidity(data));
fprintf(stderr,"\n");
}
weatherData.humidity = getHumidity(data);
weatherData.hTime = seconds;
reportsSeen |= 0x02; // I've seen report 2 now

}
}
/*
This code is related to dealing with the USB device
*/
// This searches the USB bus tree to find the device
int findDevice(libusb_device **devs)
{
libusb_device *dev;
int err = 0, i = 0, j = 0;
uint8_t path[8];

while ((dev = devs[i++]) != NULL) {
struct libusb_device_descriptor desc;
int r = libusb_get_device_descriptor(dev, &desc);
if (r < 0) {
fprintf(stderr,"Couldn't get device descriptor, %s\n", libusb_strerror(err));
return(1);
}

fprintf(stderr,"%04x:%04x (bus %d, device %d)",
desc.idVendor, desc.idProduct,
libusb_get_bus_number(dev), libusb_get_device_address(dev));
fprintf(stderr,"\n");
if (desc.idVendor == VENDOR && desc.idProduct == PRODUCT){
fprintf(stderr,"Found the weather station\n");
weatherStation.device = dev;
return (1);
}
}
return(0);
}

// to handle testing and try to be clean about closing the USB device,
// I'll catch the signal and close off.
void closeUpAndLeave(){
//OK, done with it, close off and let it go.
fprintf(stderr,"Done with device, release and close it\n");
int err = libusb_release_interface(weatherStation.handle, 0); //release the claimed interface
if(err) {
fprintf(stderr,"Couldn't release interface, %s\n", libusb_strerror(err));
}
fprintf(stderr, " Released interface and restored kernal driver\n");
libusb_close(weatherStation.handle);
fprintf(stderr, " Closed Weatherstation device\n");
libusb_exit(NULL);
fprintf(stderr, " Closed USB interface\n");
exit(0);
}

// This is where I read the USB device to get the latest data.
unsigned char data[50]; // where we want the data to go

int getit(int whichOne, int noisy){
int actual; // how many bytes were actually read

// The second parameter is bmRequestType and is a bitfield
// See http://www.beyondlogic.org/usbnutshell/usb6.shtml
// for the definitions of the various bits. With libusb, the
// #defines for these are at:
// http://libusb.sourceforge.net/api-1.0/group__misc.html#gga0b0933ae70744726cde11254c39fac91a20eca62c34d2d25be7e1776510184209

actual = libusb_control_transfer(weatherStation.handle,
LIBUSB_REQUEST_TYPE_CLASS | LIBUSB_RECIPIENT_INTERFACE | LIBUSB_ENDPOINT_IN,
//These bytes were stolen with a USB sniffer
0x01,0x0100+whichOne,0,
data, sizeof(data), 10000);
if (actual < 0){
fprintf(stderr,"Read didn't work for report %d, %s\n", whichOne, libusb_strerror(actual));
}
else {
// If you want both of the reports that the station provides,
// just allow for it. Right this second, I've found every thing
// I need in report 1. When I look further at report 2, this will
// change
//fprintf(stderr,"R%d:%d:", whichOne, actual);
//int i;
//for(i=0; i<actual; i++){
// fprintf(stderr,"%0.2X ",data[i]);
//}
//fprintf(stderr,"\n");
if (whichOne == 1)
// The actual data starts after the first byte
// The first byte is the report number returned by
// the usb read.
decode(&data[1], actual-1, noisy);
}
}
// I do several things here that aren't strictly necessary. As I learned about
// libusb, I tried things and also used various techniques to learn about the
// weatherstation's implementation. I left a lot of it in here in case I needed to
// use it later. Someone may find it useful to hack into some other device.
int main(int argc, char **argv)
{
char *usage = {"usage: %s -u -n\n"};
int libusbDebug = 0; //This will turn on the DEBUG for libusb
int noisy = 0; //This will print the packets as they come in
libusb_device **devs;
int r, err, c;
ssize_t cnt;

while ((c = getopt (argc, argv, "unh")) != -1)
switch (c){
case 'u':
libusbDebug = 1;
break;
case 'n':
noisy = 1;
break;
case 'h':
fprintf(stderr, usage, argv[0]);
case '?':
exit(1);
default:
exit(1);
}
fprintf(stderr,"%s Starting ... ",argv[0]);
fprintf(stderr,"libusbDebug = %d, noisy = %d\n", libusbDebug, noisy);
// The Pi linker can give you fits. If you get a linker error on
// the next line, set LD_LIBRARY_PATH to /usr/local/lib
fprintf(stderr,"This is not an error!! Checking linker, %s\n", libusb_strerror(0));

if (signal(SIGINT, sig_handler) == SIG_ERR)
fprintf(stderr,"Couldn't set up signal handler\n");
err = libusb_init(NULL);
if (err < 0){
fprintf(stderr,"Couldn't init usblib, %s\n", libusb_strerror(err));
exit(1);
}
// This is where you can get debug output from libusb.
// just set it to LIBUSB_LOG_LEVEL_DEBUG
if (libusbDebug)
libusb_set_debug(NULL, LIBUSB_LOG_LEVEL_DEBUG);
else
libusb_set_debug(NULL, LIBUSB_LOG_LEVEL_INFO);


cnt = libusb_get_device_list(NULL, &devs);
if (cnt < 0){
fprintf(stderr,"Couldn't get device list, %s\n", libusb_strerror(err));
exit(1);
}
// go get the device; the device handle is saved in weatherStation struct.
if (!findDevice(devs)){
fprintf(stderr,"Couldn't find the device\n");
exit(1);
}
// Now I've found the weather station and can start to try stuff
// So, I'll get the device descriptor
struct libusb_device_descriptor deviceDesc;
err = libusb_get_device_descriptor(weatherStation.device, &deviceDesc);
if (err){
fprintf(stderr,"Couldn't get device descriptor, %s\n", libusb_strerror(err));
exit(1);
}
fprintf(stderr,"got the device descriptor back\n");

// Open the device and save the handle in the weatherStation struct
err = libusb_open(weatherStation.device, &weatherStation.handle);
if (err){
fprintf(stderr,"Open failed, %s\n", libusb_strerror(err));
closeUpAndLeave(); }
fprintf(stderr,"I was able to open it\n");
// Now that it's opened, I can free the list of all devices
libusb_free_device_list(devs, 1); // Documentation says to get rid of the list
// Once I have the device I need
fprintf(stderr,"Released the device list\n");
err = libusb_set_auto_detach_kernel_driver(weatherStation.handle, 1);
if(err){
fprintf(stderr,"Some problem with detach %s\n", libusb_strerror(err));
closeUpAndLeave();
}
int activeConfig;
err =libusb_get_configuration(weatherStation.handle, &activeConfig);
if (err){
fprintf(stderr,"Can't get current active configuration, %s\n", libusb_strerror(err));;
closeUpAndLeave();
}
fprintf(stderr,"Currently active configuration is %d\n", activeConfig);
if(activeConfig != 1){
err = libusb_set_configuration (weatherStation.handle, 1);
if (err){
fprintf(stderr,"Cannot set configuration, %s\n", libusb_strerror(err));;
closeUpAndLeave();
}
fprintf(stderr,"Just did the set configuration\n");
}

err = libusb_claim_interface(weatherStation.handle, 0); //claim interface 0 (the first) of device (mine had just 1)
if(err) {
fprintf(stderr,"Cannot claim interface, %s\n", libusb_strerror(err));
closeUpAndLeave();
}
fprintf(stderr,"Claimed Interface\n");
// I don't want to just hang up and read the reports as fast as I can, so
// I'll space them out a bit. It's weather, and it doesn't change very fast.
sleep(1);
fprintf(stderr,"Setting the device 'idle' before starting reads\n");
err = libusb_control_transfer(weatherStation.handle,
0x21, 0x0a, 0, 0, 0, 0, 1000);
sleep(1);
int tickcounter= 0;
fprintf(stderr,"Starting reads\n");
while(1){
sleep(1);
if(tickcounter++ % 10 == 0){
getit(1, noisy);
}
if(tickcounter % 30 == 0){
getit(2, noisy);
}
if (tickcounter % 15 == 0){
showit();
}
}
}

There's more details on what not to do in the post where I described this in detail <link>, yep, I fessed up right in the post where the instructions are.

For the folk that have already grabbed this, sorry, I'll try a bit harder next time.

AcuRite Weather Station, Barometric Pressure Part 6

$
0
0
Part 5 of this interminable series is here <link>

Way back in part 1 of this project I mentioned that I got into a little spat with a software developer about bringing up a linux version of his code <link>?  Well, I prowled around (a lot) in his forum and discovered that he really likes to talk.  That's a good thing, a very good thing, and I certainly can't say anything bad, I've got what? 200 or so posts on various technical items on this blog, and a trail a mile wide across the internet, so something about glass houses comes to mind.

At any rate, I ran across this post of his where he discusses the barometric pressure reading from the AcuRite console <link>.  The sensor is inexpensive, but it's a high sensitivity device and capable of measuring altitude to within 20cm if it is properly used.  What does altitude have to do with barometric pressure?  Everything.

I'm not going to go into the ins and outs of altitude vs temperature vs air pressure because there's entire sites devoted to this that will do a much better job of explaining it, except where it involves getting a reading worth using from the Acurite sensor in the console of my weather station.  It's the last piece of weather information I want to grab from the station; I want battery level from the weather head, but that's not weather info.  The rest of the stuff like rainfall this week, highest recorded temperature, those things can all be derived from the other data, and is a big piece of why I broke into the unit.

But, the barometric pressure still eludes me.  So, here's what I've learned so far and I'm quite open to suggestions.  The barometric sensor is: Measurement Specialities MS5607-02BA03 and here's its spec sheet PDF file <link> and the manufacturer's web page <link>.  It's a very nice device that uses a piezo crystal to sense pressure.  Since it's subject to temperature variance, it's read in the factory and calibration parameters are stored inside the chip.  The idea is that the developer resets the chip, reads the calibration parameters and then applies them to the reading from the chip and comes out with an extremely accurate measurement for altitude.  For a device in a fixed location, the altitude reading can be directly translated to atmospheric pressure.  That would give you an extremely accurate barometer.

Getting back to the forum post above, it's pretty clear that AcuRite doesn't do this.  They have some secret algorithm that is supposed to take the need for reading the calibration out by sampling readings for about a month to derive a value that will allow them to predict the weather more accurately for the console's location.  Pardon me if I'm a bit skeptical about that. Weather forecasters have been trying to do that for centuries, and I hardly think AcuRite achieved the 'Holy Grail' of atmospheric science.  I think they couldn't figure out how to work the chip and gave up and called marketing in for a story to cover up the problem.

Who wants to buy a barometer that you have to wait a month before the readings start making sense?

We don't need accuracy in the absolute reading, we need accuracy in the changing of the reading, and a linear response that we can calibrate and forget for a while.  Remember the old physical barometers that our farmer grandparents had in the front room where you could check it on the way out the door to the barn?  Those devices worked well.  When you got them, you checked the local weather and set the reading to be the same by turning a screw on the back.  Then you set a movable needle (ours was red) to point to the same reading.  Then if the pressure went up (clear weather) you could tell because the needle moved up from the red one.  Simple, and did the job nicely.

I logged the data from the R2 message (where the barometer is supposed to show up) for quite a while, but still haven't figured it out.  In a fit of pique, I pulled the console apart to see if I could read the chip's markings:

Tearing open the AcuRite Console


Above is an over all view of the guts of the thing.  Circled on the left near the middle is the USB port and power input; above it is the board for the local temperature and humidity sensor; and way on the right is the radio receiver.  The antenna is a wire that runs up and to the left along the top.  Yes, they covered the logic chips to try and keep us from reverse engineering the hardware.


Here's the local temperature sensor (black) and local humidity sensor (white) next to an air vent. Simple setup and should work well.


The lower logic board with the reset and radio channel select switches.


The upper logic board.  The arrow points to the pressure sensor and I'm sorry it's hard to see, but you already know what it looks like from the data sheet.  This tiny little thing was hard to read and took my best magnifying glass and a good flashlight to see the markings.

All in all, it looks like a reasonable device, it makes me wonder why they took the path they chose for the barometric sensor; they seem to have enough horsepower in there to get a really good reading.

I went back to prowling through the website above, looking specifically for hints on how to derive the pressure reading from the data the device sends and came across this explanation:
In fact, the sensors used in these consoles aren't accurate; you can't expect to get an "accurate" barometric sensor for anything less than about $1,600.  That's the reason your sensor is reporting over 40 inches of mercury; we adjust for the variations between sensors using the adjustment you were asked to do.
But I don't buy that explanation.  I've seen a bunch of gauges that cost far less than $1600, with specs that are pretty impressive, and this device claims accuracy to mere centimeters.  Nope, it's simply that the data is not corrected using the burned in parameters that should be used.

So, given the various graphs provided by the manufacturer the device is relatively linear, just way off because the calibration hasn't been done, so how the heck do I find the pressure in the data?  If I can get the pressure, then rely on the temperature inside a house to be relatively stable, we should be able to derive an offset that gives us a pretty reasonable value.  Here's a small sample of the data sent in the R2 message.

02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 07 47 E0
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 05 47 E0
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 05 47 DC
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 05 47 D9
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 03 47 D9
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 03 47 D6
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 05 47 D6
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 07 47 D6
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 09 47 D6
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 09 47 D1

The reading on the console for the last line was 77 F and 30.09 in/Hg.  Notice how only the last three bytes change?  Actually though the last four bytes change.  Heres the highest and lowest values sorted from the file I'm logging to:

02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 8F DF 47 B0
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 3B 47 BB

Yes, the other values in the data do not change.  It was reported on the forum that the calibration values were being sent, but this isn't the case I my examination.  If these are calibration values, they don't resemble the items described by the manufacturer, nor do they make sense. So, we're left with the last four 8 bit values.  The data supplied by the sensor is in 24 bit values, so these last bytes could not be all of the temperature and pressure.  Since this is made in Europe, it's likely the data is in degrees C and mbar, and just looking at the four bytes as pressure, temperature, the last line of the first block could be, 0x9009 = 38873 which doesn't seem to be reasonable.  It might be somewhat reasonable for in/Hg though as in 38.8 in/Hg; a bit high, but sort of there.  I don't get much sense out of the other number 0x47d1 = 18385 either.

But notice that the second 16 bit number is roughly half the first number; it might be that the first number is a higher conversion from the A/D convertor than the second and both of them represent the same thing, just different settings on the A/D convertor.  The chip supports this, so it's possible.

And remember, I can't compare the numbers to the reading on the front of the display since AcuRite is using some secret algorithm to compute that because their way is more accurate, meaningful, whatever...

The low pressure for the last 24 hours was 952 mbar (28.11 in/Hg) and the high was 956 (28.23) and from the graph I looked at the average was 954 (28.17), so suppose the first 16 bit number was the pressure reading (uncompensated) and I just took a 24 hour average to see what came up, a change of 23 in the number would be a 1 millibar change in pressure.  If I just go with the average of 954 and consider the mean of my high and low reading 36877 and bump it up by one tenth for every 2.3  change in the number, to see what happens:

36831 to 36923 avg 36877  difference = 92
952 to 956 avg is 954  difference = 4
That would make each milibar increase equal to a change of 23 in the reading.

So, if I set 954 as being the same as 36877 and a reading of 8fe7 (right this second) would be 36839, subtract 36877 to get -38 divide by 23 to get -1.65 and the pressure would be 952.4 mbar.  Which is more reasonable than the display which says 1017 mbar, which would put me below seal level and a bunch higher than all the stations around me.  Also, remember that what I'm really interested in is the change over time as a forecast indicator.

All told, it looks to me like the barometric pressure is a total waste of time to continue trying to decode and make sense of beyond this.  It appears that AcuRite ignores the correction factors in favor of some secret technique, and we all know how I feel about 'secrets'.  I'll try this for a few days to see how it goes, but my big inclination is to get a pressure sensor and use some of the example code out there to read it and properly apply the compensation factors to get a really good reading.  I may not use the same chip, because there's a lot of breakout boards for the BMP180 and they're cheap.

Hmm, I could hook a BMP180 up to an Arduino, get both the temperature and barometric pressure from it, send it through an XBee to my Pi controller and replace the configuration I have outside in the Stephenson screen.  That would actually be a cool project.

If you notice something I overlooked, misunderstood, or messed up, let me know.  

AcuRite Weather Station, Raspberry Pi, Refining the readings Part 7

$
0
0
The previous part of this project is here <link>

Yes, another post on this device.  Even with this one, I'm not done; there will be others since I want to sample the RF from the sensor at some point and also bring up a separate little computer for just the weather station.  But I was extremely lucky and one of my readers double checked my work and found a couple of mistakes.

Thanks go to Heath Paddock <link>; he double checked the wind speed and direction and found mistakes, then gave me code to correct my mistakes.  Thank you Heath.  Due to a couple of factors completely within my control I misread the wind direction conversion.  He noticed it when he was testing the device and dropped me a line.  Then he double checked the wind speed and noticed a mistake there as well.

I was getting the bits correctly, but the conversion was off; let's look at the wind direction first.  The correct table for this sensor to convert from a number to direction looks like this:

// Array to translate the integer direction provided to text
char *Direction[] = {
"NW", // 0
"WSW", // 1
"WNW", // 2
"W", // 3
"NNW", // 4
"SW", // 5
"N", // 6
"SSW", // 7
"ENE", // 8
"SE", // 9
"E", // 10
"ESE", // 11
"NE", // 12
"SSE", // 13
"NNE", // 14
"S" };

Every single value is different; when I mess up, I do it in style.  When he sent me the code, I climbed up on the roof and brought down the station to check and see what is happening.  Yep, North was a six coming out of the USB port in report R1.  I went through each of the other directions and they matched his numbers exactly, then I looked at my console and it was wrong.  That just didn't make sense at all, so like all good computer users, I reset the darn thing.  It was still wrong.  I finally unplugged the console from the USB port on the Pi, pulled the batteries out of the console, and just to be complete, pulled the batteries out of the weather head.  This is a sure sign of desperation, when you do something that you know won't help just because you can't think of anything else.

But that did it.  A value of six (the pointer on the weather station was taped in place by now) corresponded to a N reading on the console.  I have no explanation for what was going on, but it works fine now.  No, I'm not going to try and recreate the situation.  I'm quite happy with the way it works and will double check the readings each time I have to change batteries in either the console or the weather head.

At any rate, the wind direction seems to be working really well now.  Next is the wind speed; Heath noticed an odd reading and asked me about it.  I had seen the reading the day before I left for Christmas, so I knew what he was talking about, but I didn't have time to look into it.  He took the time to examine the data he was receiving and found the solution.  The wind speed sensor is granular to .5 MPH; yes MPH, not KPH, and is reported back doubled.  The console does the usual 8 bit math and puts up a reading that is slightly off and loses the .5 granularity.  It also does some degree of smoothing and only reports the speed at intervals so you have to be somewhat patient to see what's going on.  Here's the code I'm currently using:

int getWindSpeed(char *data){
int leftSide = (data[3] & 0x1f) << 3;
int rightSide = (data[4] & 0x70) >> 4;
float speed = (leftSide | rightSide) / 2.0;
return(speed);
}

Notice that I'm using a float to preserve the .5 granularity?  I did that to get the most out of my sensor and because it looks cool on my web display.  It won't match the console exactly, but that isn't really my goal.

There will probably be something else some clever person turns up that I missed or interpreted wrong, but that's part and parcel of hacking into a device with little or no information up front.  The point is that multiple people have participated in this project over many miles of separation, and we all get to use it.  I'll have the code updated in Github in an hour or so as well.

Now I have to climb back up on the roof and put it in place.  There's a storm coming and I don't want to miss it.

AcuRite Weather Station, Raspberry Pi, Fun Stuff, Part 8

$
0
0
Part 7 of this series is here <link>

I mentioned way back in part 1 that I visited a site and discussed a linux version of the AcuRite weather station interface with a guy on Valley Information Systems discussion board <link>.  The person was the professed author of the VIS sofware that is provided by AcuRite and he has a greatly expanded and pretty cotton pickin' slick version that can be subscribed to as well.  Michael Walsh is his name and he's pretty sharp, but very, very snarky.

You've all seen this, especially from some of the dweebs on the various forums that belittle every post that is put up on the site.  I'd show you the post so you could get a feel for what I'm talking about, but he removed it.  Yep, when I went back and posted about my success with reading the USB port on linux and starting to decode the various items, I got a subscription notice that my post had been moved to the archives.  Which generally means it can still be found by a search on the forum, but it doesn't appear in the lists of posts.

Today, I got another notice and when I clicked on it:

The topic or board you are looking for appears to be either missing or off limits to you.

Heh, it looks like it's off the board entirely.  When I looked for 'linux' on the board, there's been a couple of folk looking around for a linux version recently; I may have become competition for him.

Keep in mind, I don't know why he moved the posts, or why he eventually made them disappear, it could just be weird traffic or some accident.  But it is cool that the posts of a successful, FREE, version that reads the AcuRite console's USB output on a little Raspberry Pi that doesn't tie up your home PC have disappeared.

But, once it's on the internet, it gets read, copied, backed up, mailed about, etc.  Especially since a few folk have helped me on this project by finding bugs and running it a lot in their own home.  I just typed "acurite raspberry pi" into Goodle and guess what site came up as the first entry?  hee hee.

So, you weather station folk out there, folk that got one of these for Christmas, people that succumbed to the impulse of the cool packaging at Costco (I'm in that group) spread the word.  If enough folk get involved, this could lead to a great weather station package.  I just don't have the hardware, facilities, time, or diversity in my network to do a really good job.

But wouldn't it be cool

Wemo Light Switches, a Completely New Approach

$
0
0
I have several posts on implementing the Wemo switches into my house <link, link, link> and of course a couple of other modifications to make things a little easier; lastly, where I finally found out what was causing them to just go away from time to time <link>.  Even with all this attention, the things still are an aggravation.  But not just for me; one of my readers, Glenn Tracy, has several of the switches, and one Wemo Insight plug that have been driving him nuts.  Glenn made the serious mistake of mentioning arp, so I enlisted him to try the various ideas out as I coded them (he probably regrets that now), and I started exploring possibilities.

The devices pretty much stay on line since I found a cause of the intermittent failure, but then they get hard to find because they don't respond well to Upnp discovery messages.  Since the IP address is assigned by DHCP, I can't rely on that to find them, plus the port they respond to can change any time the switch feels cranky.  So how the heck can I find them reliably ... arp seemed to work.  TCP/IP relies on the arp protocol to communicate, that should be reliable enough.  So, how to use it?

Fortunately, someone else has addressed this problem with a couple of tools, arping and arp-scan.  Arping relies on some knowledge in advance, but arp-scan doesn't.  You can issue an arp-scan on your home network and every machine on the network will be found.  Naturally, this isn't perfect, some devices guard against response to prevent hacking, and some devices don't respond on the first interaction.  But, I'm looking for Wemo switches, they respond, so now I have to figure out how to get the port number that the switch currently listens to.

Taking a chapter from my hacks on various other switches and devices, I decided to just try all the ports and see which one worked.  Since the switches I have only use ports 49152, 49153, and 49154, I can try all of them in a matter of seconds and not worry about it taking a ton of time searching.  If it turns out later that they go higher from time to time, I can simply add another port to the list.  Now, armed with this possibility, I tried out arp-scan.  Arp-scan is a normal linux tool, so all we have to do is install it:

sudo apt-get install arp-scan

And let the installation run.  Now we have to make it run as a root process because raw sockets need root permission to be effective:

sudo chmod +s /usr/bin/arp-scan

That will give the executable the 'suid' bit and since it's owned by root, it will get all the capabilities of a root process.  Nice, now run it and see what happens:

 arp-scan -q -v 192.168.0.0-192.168.0.255
Interface: eth0, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.8.1 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
192.168.0.1 40:4a:03:de:41:87
192.168.0.2 00:1d:7e:91:12:0a
192.168.0.28 ec:1a:59:ce:82:71
192.168.0.34 4c:82:cf:83:c5:05
192.168.0.45 ec:e0:9b:b9:31:11
192.168.0.50 00:1d:7e:91:12:0a
192.168.0.29 ec:1a:59:ce:7c:8d
192.168.0.44 78:4b:87:41:5d:53
192.168.0.43 ec:1a:59:cd:99:55
192.168.0.22 2c:d0:5a:f6:dc:65
192.168.0.26 ec:1a:59:e8:fe:81
192.168.0.47 18:22:7e:d2:04:a0
192.168.0.202 de:ad:be:ef:fe:ed
192.168.0.203 de:ad:be:ef:fe:ee
192.168.0.204 de:ad:be:ef:fe:ef
--- Pass 1 complete
--- Pass 2 complete

18 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.8.1: 256 hosts scanned in 1.258 seconds (203.50 hosts/sec). 15 responded

Notice a few things: I carefully chose the options to give me the information I needed while excluding the built-in capability of finding the manufacturer of the devices found by table lookup.  I was only interested in Belkin devices, and the table lookup depends on having up to date tables.  I really don't want to have to worry about keeping some table updated, so I'll handle that in the code for controlling the switch.  Also, arp-scan will take a range of addresses; since I limit my DHCP devices to the range 0-50, I should be able to scan all addresses in a few seconds.  That should be much, much faster than waiting for devices to respond to the (often ineffective) upnp discovery command.

The devices have been found, and a little code can isolate out the Belkin devices, so now it's time to find the port number the switch listens to.  This serves another purpose, to separate the switches from other Belkin devices that may show up over time.  It wouldn't do much good to tell a router or ethernet switch to turn off thinking it was a switch.  We've already discovered many intimate details of the switches including the location of the switch's setup xml description.  So a simple HTML request to switch-ip-address:possible-port/setup.xml should return a description of the switch.  Once we have the html in hand, just look inside it for the keywords and we know what we have.  I'm only interested in an Insite or a lightswitch; the other devices can wait for a later hack.  If nothing comes back, then I have the wrong port, just try the next one.  Here's an example of the setup.xml output that I sucked out of one of my switches:


<root xmlns="urn:Belkin:device-1-0">
<specVersion>
<major>1</major>
<minor>0</minor>
</specVersion>
<device>
<deviceType>urn:Belkin:device:lightswitch:1</deviceType>
<friendlyName>outsidegarage</friendlyName>
<manufacturer>Belkin International Inc.</manufacturer>
<manufacturerURL>http://www.belkin.com</manufacturerURL>
<modelDescription>Belkin Plugin Socket 1.0</modelDescription>
<modelName>LightSwitch</modelName>
<modelNumber>1.0</modelNumber>
<modelURL>http://www.belkin.com/plugin/</modelURL>
<serialNumber>221332K130012B</serialNumber>
<UDN>uuid:Lightswitch-1_0-221332K130012B</UDN>
<UPC>123456789</UPC>
<macAddress>EC1A59E8FE80</macAddress>
<firmwareVersion>WeMo_WW_2.00.6395.PVT</firmwareVersion>
<iconVersion>3|49153</iconVersion>
<binaryState>0</binaryState>
<iconList>
<icon>
<mimetype>jpg</mimetype>
<width>100</width>
<height>100</height>
<depth>100</depth>
<url>icon.jpg</url>
</icon>
</iconList>
<serviceList>
<service>
<serviceType>urn:Belkin:service:WiFiSetup:1</serviceType>
<serviceId>urn:Belkin:serviceId:WiFiSetup1</serviceId>
<controlURL>/upnp/control/WiFiSetup1</controlURL>
<eventSubURL>/upnp/event/WiFiSetup1</eventSubURL>
<SCPDURL>/setupservice.xml</SCPDURL>
</service>
<service>
<serviceType>urn:Belkin:service:timesync:1</serviceType>
<serviceId>urn:Belkin:serviceId:timesync1</serviceId>
<controlURL>/upnp/control/timesync1</controlURL>
<eventSubURL>/upnp/event/timesync1</eventSubURL>
<SCPDURL>/timesyncservice.xml</SCPDURL>
</service>
<service>
<serviceType>urn:Belkin:service:basicevent:1</serviceType>
<serviceId>urn:Belkin:serviceId:basicevent1</serviceId>
<controlURL>/upnp/control/basicevent1</controlURL>
<eventSubURL>/upnp/event/basicevent1</eventSubURL>
<SCPDURL>/eventservice.xml</SCPDURL>
</service>
<service>
<serviceType>urn:Belkin:service:firmwareupdate:1</serviceType>
<serviceId>urn:Belkin:serviceId:firmwareupdate1</serviceId>
<controlURL>/upnp/control/firmwareupdate1</controlURL>
<eventSubURL>/upnp/event/firmwareupdate1</eventSubURL>
<SCPDURL>/firmwareupdate.xml</SCPDURL>
</service>
<service>
<serviceType>urn:Belkin:service:rules:1</serviceType>
<serviceId>urn:Belkin:serviceId:rules1</serviceId>
<controlURL>/upnp/control/rules1</controlURL>
<eventSubURL>/upnp/event/rules1</eventSubURL>
<SCPDURL>/rulesservice.xml</SCPDURL>
</service>
<service>
<serviceType>urn:Belkin:service:metainfo:1</serviceType>
<serviceId>urn:Belkin:serviceId:metainfo1</serviceId>
<controlURL>/upnp/control/metainfo1</controlURL>
<eventSubURL>/upnp/event/metainfo1</eventSubURL>
<SCPDURL>/metainfoservice.xml</SCPDURL>
</service>
<service>
<serviceType>urn:Belkin:service:remoteaccess:1</serviceType>
<serviceId>urn:Belkin:serviceId:remoteaccess1</serviceId>
<controlURL>/upnp/control/remoteaccess1</controlURL>
<eventSubURL>/upnp/event/remoteaccess1</eventSubURL>
<SCPDURL>/remoteaccess.xml</SCPDURL>
</service>
<service>
<serviceType>urn:Belkin:service:deviceinfo:1</serviceType>
<serviceId>urn:Belkin:serviceId:deviceinfo1</serviceId>
<controlURL>/upnp/control/deviceinfo1</controlURL>
<eventSubURL>/upnp/event/deviceinfo1</eventSubURL>
<SCPDURL>/deviceinfoservice.xml</SCPDURL>
</service>
<service>
<serviceType>urn:Belkin:service:smartsetup:1</serviceType>
<serviceId>urn:Belkin:serviceId:smartsetup1</serviceId>
<controlURL>/upnp/control/smartsetup1</controlURL>
<eventSubURL>/upnp/event/smartsetup1</eventSubURL>
<SCPDURL>/smartsetup.xml</SCPDURL>
</service>
<service>
<serviceType>urn:Belkin:service:manufacture:1</serviceType>
<serviceId>urn:Belkin:serviceId:manufacture1</serviceId>
<controlURL>/upnp/control/manufacture1</controlURL>
<eventSubURL>/upnp/event/manufacture1</eventSubURL>
<SCPDURL>/manufacture.xml</SCPDURL>
</service>
</serviceList>
<presentationURL>/pluginpres.html</presentationURL>
</device>
</root>


The tag <modelName>LightSwitch</modelName> tells me that this is a light switch, as opposed to an Insight, which would look like <modelName>Insight</modelName>.  This matters because the Insight switch returns a different value when it's turned on. The MAC address is in there, and a lot of other things that are used during operation.  For example to find the friendly name (the name assigned by the user) look at the tag <friendlyName>outsidegarage</friendlyName>.

The very fact that I got the XML back means I found the port and the tags tell me what it is, so now I have enough information, just some code to control the switches and I should be done.  Yeah, right.  When my victim er ... uh ... partner Glenn in this effort tested it, there were a number of problems that came up.  I had to put the IP address range in the .houserc file because there just wasn't any way to reliably discover the network range we were interested in.  Then, we found out that the wemo switches took to long to respond to the arp request and it had to be retried, sometimes several times to get them to talk.  When the switches decide to change addresses, there has to be a rediscovery to get the new port number.  You know, the usual list of things that will drive you nuts using a device differently than it was intended.

But, we overcame the ton of 'glitches' that almost appear to have been designed into the switch to keep people like me from doing things like this, and in the process made it much faster, much more reliable, and able to leap tall buildings with a single bound.  There are some caveats though:  The switches can still disappear.  Yes, even with a different method of discovery, and port change retries, the things will fail to respond long enough to evade detection.  To overcome this, the control process will simply exit after a few retries, and some controlling tool can automatically restart it to do the entire discovery process all over again.  This will find the switches and set them up to be used all over again.  A simple script like this:


#!/bin/bash

while :
do
        echo "Starting wemoccontrol"
        wemocontrol.py
        sleep 10
done

can restart the process.  Yes, you have to replace the commands with something suitable to your environment, and the sleep is to keep it under control in case you make a mistake, but you can use this to test the idea.  I use upstart and a config file to accomplish this for all the processes I run on the Pi, why invent something new when a great tool already exists?

So, instead of using upnp discovery, I switched to arp discovery, I find the Belkin device, and among those I try the various ports to discover Wemo light switches and Insite devices, then I bring up a CherryPy server to control them.  When they fail to respond to a request, I try three more times to make them respond and then give up and let the process die to be restarted by some other control software.  The controls are the same as my previous tries, I record the state in a Sqlite3 database and that in turn can be read by anything running on the machine.  Sounds pretty complicated, but it's much, much simpler than the other techniques I came up with.  This is very lightweight and works quite a bit faster than the others, and so far, much more reliably.  Here's the code, and it has also been placed in Github to make it easier to grab if you want <link>.

#! /usr/bin/python
# Checking Wemo Switches
#
import subprocess
import commands
from datetime import datetime, timedelta
import time
import urllib2
import BaseHTTPServer
from socket import *
import sys
import json
import re
import argparse
import sqlite3
import cherrypy
from houseutils import lprint, getHouseValues, timer, checkTimer

#--------This is for the HTML interface
def openSite(Url):
#lprint (Url)
webHandle = None
try:
webHandle = urllib2.urlopen(Url, timeout=2) # give up in 2 seconds
except urllib2.HTTPError, e:
errorDesc = BaseHTTPServer.BaseHTTPRequestHandler.responses[e.code][0]
#print "Error: (opensite) cannot retrieve URL: " + str(e.code) + ": " + errorDesc
raise
except urllib2.URLError, e:
#print "Error: (openSite) cannot retrieve URL: " + e.reason[1]
raise
except: #I kept getting strange errors when I was first testing it
e = sys.exc_info()[0]
#print ("(opensite) Odd Error: %s" % e )
raise
return webHandle

def talkHTML(ip, command):
website = openSite("HTTP://" + ip + '/' + urllib2.quote(command, safe="%/:=&?~#+!$,;'@()*[]"))
# now (maybe) read the status that came back from it
if website is not None:
websiteHtml = website.read()
return websiteHtml

# and this is for the SOAP interface
# Extract the contents of a single XML tag from the data
def extractSingleTag(data,tag):
startTag = "<%s" % tag
endTag = "</%s>" % tag

try:
tmp = data.split(startTag)[1]
index = tmp.find('>')
if index != -1:
index += 1
return tmp[index:].split(endTag)[0].strip()
except:
pass
return None

def sendSoap(actionName, whichOne, actionArguments):
argList = ''
soapEnd = re.compile('<\/.*:envelope>')
if not actionArguments:
actionArguments = {}
for item in switches:
if item["name"] == whichOne:
thisOne = item
break;
switchIp = item["ip"]
switchPort = item["port"]

for arg,(val,dt) in actionArguments.iteritems():
argList += '<%s>%s</%s>' % (arg,val,arg)

soapRequest = 'POST /upnp/control/basicevent1 HTTP/1.1\r\n'
# This is the SOAP request shell, I stuff values in it to handle
# the various actions
# First the body since I need the length for the headers
soapBody = '<?xml version="1.0"?>\n'\
'<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">\n'\
'<SOAP-ENV:Body>\n'\
'\t<m:%s xmlns:m="urn:Belkin:service:basicevent:1">\n'\
'%s\n'\
'\t</m:%s>\n'\
'</SOAP-ENV:Body>\n'\
'</SOAP-ENV:Envelope>' % (actionName,argList,actionName)

#These are the headers to send with the request
headers = {
'Host':'%s:%s' % (switchIp, switchPort),
'Content-Length':len(soapBody),
'Content-Type':'text/xml',
'SOAPAction':'"urn:Belkin:service:basicevent:1#%s"' % (actionName)
}
#Generate the final payload
for head,value in headers.iteritems():
soapRequest += '%s: %s\r\n' % (head,value)
soapRequest += '\r\n%s' % soapBody
if showXml:
print stars
print "***REQUEST"
print soapRequest

try:
sock = socket(AF_INET,SOCK_STREAM)
sock.connect((switchIp,int(switchPort)))
sock.settimeout(3); # don't want to hang forever, ever
sock.send(soapRequest)
soapResponse = ""
while True:
data = sock.recv(1024)
if not data:
break
else:
soapResponse += data
if soapEnd.search(soapResponse.lower()) != None:
break
if showXml:
print "***RESPONSE"
print soapResponse
print stars
print ''
sock.close()
(header,body) = soapResponse.split('\r\n\r\n',1)
if not header.upper().startswith('HTTP/1.') and ' 200 ' in header.split('\r\n')[0]:
print 'SOAP request failed with error code:',header.split('\r\n')[0].split('',1)[1]
errorMsg = self.extractSingleTag(body,'errorDescription')
if errorMsg:
print 'SOAP error message:',errorMsg
return None
else:
return body
except Exception, e:
lprint ('Caught exception in sending:', e, switchIp, switchPort)
sock.close()
return None
except KeyboardInterrupt:
print "Keyboard Interrupt"
sock.close()
return None

# This will look at the result from sendSoap, and if the
# switch disappeared, it will try and get the new port number
# and update the various items. This should allow the code
# to continue as if the switch never decided to change its
# port number
def sendCommand(actionName, whichOne, actionArguments):
result = sendSoap(actionName, whichOne, actionArguments)
if result is not None:
return result
# it failed, now we have to do something about it
# first, get the switch entry to check for a port change
for item in switches:
if item["name"] == whichOne:
thisOne = item
break;
switchIp = item["ip"]
switchPort = item["port"]
# try to get the port number from the switch a few times
for i in range(0,3): # Only try this three times
lprint ("Trying to recover the switch %s"%whichOne)
# getPort doesn't use sendSoap, so this call won't recurs
newEntry = getPort(switchIp)
# if the port changed, try and get the new one
if newEntry is not None:
# fine, it's at least alive, grab the port number,
# print something, and and stuff it in the database
# if it didn't change this won't break it, but if
# it did change, this will fix it.
item["port"] = newEntry["port"]
lprint ("Switch", whichOne, "changed ip from", switchPort, "to", newEntry["port"])
dbconn = sqlite3.connect(DATABASE)
c = dbconn.cursor()
try:
c.execute("update lights "
"set port=? where name = ?;",
(newEntry["port"], whichOne))
except sqlite3.OperationalError:
lprint("Database is locked, record skipped")
dbconn.commit()
dbconn.close()
# now try the command again
# if it died completely it may have come back by now,
# or if the port changed, this will try it one more time
# it needs a limit this because this call will recurs
result = sendSoap(actionName, whichOne, actionArguments)
if result is not None:
lprint("Switch recovered")
return result
time.sleep(1) #give the switch time to catch its breath
else:
# this means the switch is not responding to HTML
# so try the getPort again to see if it's back yet
# There's no point in sending the soap command yet
time.sleep(1) #give the switch time to catch its breath
continue
# it failed three times, just give up, die and let the system
# restart the process.
exit("The switch %s went away"% whichOne)


# Step through each light and see get its current state
# then record the state in the database.
def doLights():
for switch in switches:
thisOne = switch['name']
updateDatabase(thisOne,get(thisOne))

def keepAlive():
'''
I update the database periodically with the time so I can check to see
if things are holding together. I currently use the time in the light
switch records for this.
'''
lprint(" keep alive")
for switch in switches:
thisOne = switch['name']
updateDatabase(thisOne, get(thisOne), force=True)


def get(whichone):
'''
Returns On or Off
'''
resp = sendCommand('GetBinaryState', whichone, {})
if resp is not None:
tagValue = extractSingleTag(resp, 'BinaryState').split('|')[0]
return 'Off' if tagValue == '0' else 'On'
return 'Off'

def on(whichone):
"""
BinaryState is set to 'Error' in the case that it was already on.
"""
resp = sendCommand('SetBinaryState', whichone, {'BinaryState': (1, 'Boolean')})
if resp is not None:
tagValue = extractSingleTag(resp, 'BinaryState').split('|')[0]
status = 'On' if tagValue in ['1', '8', 'Error'] else 'Off'
handleUpdate(whichone, status)
lprint("turned %s on"%(whichone))
return status
return 'Off'

def off(whichone):
"""
BinaryState is set to 'Error' in the case that it was already off.
"""
resp = sendCommand('SetBinaryState', whichone, {'BinaryState': (0, 'Boolean')})
if resp is not None:
tagValue = extractSingleTag(resp, 'BinaryState').split('|')[0]
status = 'Off' if tagValue in ['0', 'Error'] else 'On'
handleUpdate(whichone, status)
lprint("turned %s off"%(whichone))
return status
return 'Off'

def toggle(whichOne):
if (get(whichOne) == 'On'):
off(whichOne)
else:
on(whichOne)

def outsideLightsOn():
lprint (" Outside lights on")
on("outsidegarage")
on("frontporch")
on("cactusspot")

def outsideLightsOff():
lprint (" Outside lights off")
off("outsidegarage")
off("frontporch")
off("cactusspot")


def handleUpdate(whichone, status):
for i in switches:
if i['name'] == whichone:
i['status'] = status
updateDatabase(whichone, status)

def updateDatabase(whichone, status, force=False):
'''
This is running on a Pi and is not event driven, so polling like
this will result in considerable wear to the SD card. So, I'm going to
read the database to see if it needs to be changed before I change it.
According to everything I've read, reads are free, it's the writes that
eat up the card.
'''
dbconn = sqlite3.connect(DATABASE)
c = dbconn.cursor()
c.execute("select status from lights where name = ?;",
(whichone,))
oldstatus = c.fetchone()
if oldstatus[0] != status or force == True:
lprint ("Had to update database %s, %s"%(whichone, status))
try:
c.execute("update lights "
"set status = ?, utime = ? where name = ?;",
(status, time.strftime("%A, %B, %d at %H:%M:%S"), whichone))
dbconn.commit()
except sqlite3.OperationalError:
lprint("Database is locked, record skipped")
dbconn.close()

# If a command comes in from somewhere, this is where it's handled.
def handleCommand(command):
lprint(str(command))
# the command comes in from php as something like
# ('s:17:"AcidPump, pumpOff";', 2)
# so command[0] is 's:17:"AcidPump, pumpOff'
# then split it at the " and take the second item
try:
c = str(command[0].split('\"')[1]).split(',')
except IndexError:
c = str(command[0]).split('') #this is for something I sent from another process
lprint(c)
if (c[0] == 'OutsideLightsOn'):
outsideLightsOn()
elif (c[0] == 'OutsideLightsOff'):
outsideLightsOff()
elif (c[0] == 'fPorchToggle'):
toggle("frontporch")
elif(c[0] == 'garageToggle'):
toggle("outsidegarage")
elif (c[0] == 'cactusToggle'):
toggle("cactusspot")
elif (c[0] == 'patioToggle'):
toggle("patio")
else:
lprint("Weird command = " + str(c))

# First the process interface, it consists of a status report and
# a command receiver.
class WemoSC(object):
@cherrypy.expose
@cherrypy.tools.json_out() # This allows a dictionary input to go out as JSON
def status(self):
status = []
for item in switches:
status.append({item["name"]:get(item["name"])})
return status

@cherrypy.expose
def pCommand(self, command):
handleCommand((command,0));

@cherrypy.expose
def index(self):
status = "<strong>Current Wemo Light Switch Status</strong><br /><br />"
for item in switches:
status += item["name"] +" is " + get(item["name"]) + "&nbsp;&nbsp;"
status += '<a href="wemocommand?whichone='+item["name"]+'"><button>Toggle</button></a>'
status += "<br />"
return status

@cherrypy.expose
def wemocommand(self, whichone):
# first change the light state
toggle(whichone)
# now reload the index page to tell the user
raise cherrypy.InternalRedirect('/index')

# given the ip of a Belkin device this will try the ports that
# are used on the Wemo switches to see which one works. The assumption
# is that if none of the ports work, it's not a switch, it's a modem or
# something else.
def getPort(ip):
entry = []
for p in ["49153", "49154", "49155"]:
try:
resp = talkHTML(ip + ':' + p + "/setup.xml", "")
if debug:
print "\tfound one at", b[0], "port", p
if showXml:
print stars
print "response from switch"
print resp
print stars
name = extractSingleTag(resp, 'friendlyName')
model = extractSingleTag(resp, 'modelName')
entry = {"mac":b[1],"ip":b[0], "port":p, "name":name, "model":model}
return entry
except timeout:
continue
except urllib2.URLError:
continue
except:
e = sys.exc_info()[0]
print ("Unexpected Error: %s" % e )
continue
return None

####################### Actually Starts Here ################################
debug = False
showXml = False
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("-d", "--debug",
action = "store_true",
help='debug flag')
parser.add_argument("-x", "--xml",
action = "store_true",
help='show xml')
parser.add_argument('count',type=int);
args = parser.parse_args()
if args.debug:
print "Running with debug on"
debug = True
if args.xml:
print "Running with showXML on"
showXml = True
targetNumber = args.count

stars = "*********************************************"

#-------------------------------------------------
# the database where I'm storing stuff
DATABASE=getHouseValues()["database"]
lprint("Using database ", DATABASE);
# Get the ip address and port number you want to use
# from the houserc file
ipAddress=getHouseValues()["wemocontrol"]["ipAddress"]
port = getHouseValues()["wemocontrol"]["port"]
lprint("started looking for {} switches".format(targetNumber))

# This works on my machine, but you may have to mess with it
# The arp-scan below tells it not to look up the manufacturer because I
# didn't want to worry about keeping the tables that are used up to date,
# the -l tells it to find the local net address on its own, and
# -v (verbose) will print that net address so I can show it for debugging
# I take the scan range out of the .houserc file, it's an entry under wemocontrol
# that looks like "scanRange":"192.168.0.1-192.168.0.50" adjust this as
# needed
try:
scanRange = getHouseValues()["wemocontrol"]["scanRange"]
arpCommand = "arp-scan -q -v %s 2>&1" %(scanRange)
except KeyError:
print "No entry in .houserc for wemocontrol scanRange"
exit();

while True:
devices = [];
# first the devices on the network
if debug:
print "arp-scan command is:", arpCommand
theList = subprocess.check_output(arpCommand,shell=True);
# split the output of the arp-scan into lines instead of a single string
lines = theList.splitlines()
# this looks at each line and grabs the addresses we're interested in
# while ignoring the lines that are just information.
for line in lines:
allowedDigits = set("0123456789abcdef:. \t")
if all(c in allowedDigits for c in line):
d = line.split()
try:
devices.append([d[0], d[1]])
except IndexError: # an empty line will pass the test
continue
# arp-scan can give the same addresses back more than once
# step through the list and remove duplicates
temp = []
for e in devices:
if e not in temp:
temp.append(e)
devices = temp
if debug:
print devices
# for each device, look up the manufacturer to see if it was registered
# to belkin
bDevices = []
# I got this list direct from the IEEE database and it may
# need to be updated in a year or two.
belkinList = ("001150", "00173F", "001CDF", "002275", "0030BD",
"08863B", "94103E", "944452", "B4750E", "C05627", "EC1A59")
for d in devices:
if d[1].replace(':','')[0:6].upper() in belkinList:
bDevices.append([d[0],d[1]])
if debug:
print "These are the Belkin devices on the network"
print bDevices
if len(bDevices) < targetNumber:
lprint ("Only found", len(bDevices), "Belkin devices, retrying")
time.sleep(1)
continue
# Got all that were asked for, continue to the next step

# Now that we have a list of the Belkin devices on the network
# We have to examine them to be sure they are actually switches
# and not a modem or something else. This will also assure that
# they will actually respond to a request. They still may not work,
# but at least we have a chance.
switches = []
for b in bDevices:
result = getPort(b[0])
if result is not None:
switches.append(result)
# Did we find enough switches ?
if len(switches) < targetNumber:
lprint ("Only found", len(switches), "of them, retrying")
devices = []
continue
# Yes we did, break out.
break;
# Now I'm going to check the database to see if it has been
# adjusted to hold all the items (older version didn't have
# ip, port, and mac addresses
dbconn = sqlite3.connect(DATABASE)
c = dbconn.cursor()
c.execute("pragma table_info(lights);")
dbrow = c.fetchall()
if not any('ip' and 'mac' and 'port' in r for r in dbrow):
lprint ("Database needs to be adjusted")
lprint ("to hold ip, port, and MAC")
try:
print "adding ip if needed"
c.execute("alter table lights add column ip text;")
except sqlite3.OperationalError:
print "ip was already there"
try:
print "adding mac if needed"
c.execute("alter table lights add column mac text;")
except sqlite3.OperationalError:
print "mac was already there"
try:
print "adding port if needed"
c.execute("alter table lights add column port text;")
except sqlite3.OperationalError:
print "port was already there"
dbconn.commit()
else:
lprint ("Database already adjusted")
for item in switches:
try:
c.execute("update lights "
"set ip = ?, mac=?, port=?where name = ?;",
(item["ip"], item["mac"], item["port"], item["name"]))
dbconn.commit()
except sqlite3.OperationalError:
lprint("Database is locked, record skipped")
dbconn.commit()
dbconn.close()
lprint ("")
lprint ("The list of", len(switches), "switches found is")
for item in switches:
lprint ("Friendly name:", item["name"])
lprint ("Model:", item["model"])
lprint ("IP address:", item["ip"])
lprint ("Port number:", item["port"])
lprint ("MAC:", item["mac"])
lprint ('')

# timed things.
checkLightsTimer = timer(doLights, seconds=2)
keepAliveTimer = timer(keepAlive, minutes=4)
# Now configure the cherrypy server using the values
cherrypy.config.update({'server.socket_host' : ipAddress.encode('ascii','ignore'),
'server.socket_port': port,
'engine.autoreload.on': False,
})
# Subscribe to the 'main' channel in cherrypy with my timer
cherrypy.engine.subscribe("main", checkTimer.tick)
lprint ("Hanging on the wait for HTTP message")
# Now just hang on the HTTP server looking for something to
# come in. The cherrypy dispatcher will update the things that
# are subscribed which will update the timers so the light
# status gets recorded.
cherrypy.quickstart(WemoSC())

sys.exit("Told to shut down");

Notice that it is self contained.  It doesn't require Miranda.py anymore, nor any of the Ouimeaux libraries.  That's what makes it appealing to me; it's nice to have everything in one module and reasonably easy to understand.  I added a few parameters:

wemocontrol.py -h
usage: wemocontrol.py [-h] [-d] [-x] count

positional arguments:
count

optional arguments:
-h, --help show this help message and exit
-d, --debug debug flag
-x, --xml show xml

I start it with wemocontrol.py 4, because I have four switches.  It will keep looking until it finds all four switches which could be a problem if one of them is disconnected, but I'll deal with that problem as it happens.  The -x parameter displays the SOAP messages as they are sent and received.  This can be used to find problems in the protocol or to understand how it works.  The -d is for debugging when you can't get things to work.  If you turn them both on, the volume of output is huge, so be prepared for it.  The -h is because I can never remember this stuff and wanted a reminder.  To run it, you must have an entry in a file called .houserc in the home directory of the user.  I use this file for all the entries for my processes.  This way I can keep things like keys, directory paths, etc in one place easily found.  The entries I have look like this:

"wemocontrol":{
"ipAddress":"192.168.0.205",
"port": 51001,
"scanRange":"192.168.0.1-192.168.0.50"},

I have a complete description of the file and how to use it here <link>.

Now, I must thank Glenn for his help in this effort.  This thing would have been a bear to test without him.  The rest of you, have fun with it.


I Want to Whine About Shipping Costs.

$
0
0
I don't know about the rest of you, but sometimes one of my projects costs more in shipping fees than components.  Getting a couple of chips and resistors across the country can make a project cost more than the fun of doing it is worth.  That's why more and more of my projects are based on cheap Chinese products shopped from one of the Chinese suppliers or eBay.

Sure, I'd much rather buy locally, if there was a locally available to buy from, but the industry has moved completely away from that model...  There are no electronic shops within any reasonable driving distance of my place.  The cost of getting something from somewhere else has trained me to look at the shipping cost very carefully, and my big mouth has just gotten a thread closed at one of my favorite supplier's forum.

The story: I want to follow up on my threat of getting a barometric sensor and setting it up to read temperature and air pressure inside a Stevenson screen I have on a fence post in the yard <link>, so I went to Adafruit and found what I wanted.  It was a:

BMP180 Barometric Pressure/Temperature/Altitude Sensor- 5V ready PID: 1603  $9.95

This is a great little breakout board (from the description) and would be a lot of fun to work up in the project.  However, as I was checking out, the shipping price came up:

UPS Ground: $13.23

There were other options, all more expensive than this.  What?  That's almost 150% of the cost of the item.  So, I went to their forum and looked to see if anyone else complained about this.  Of course someone did, so I did too.  I never want to be outdone in the complaint department.  When it became clear to the folk running the forum that I wasn't going to be shuffled aside by sycophants citing how hard and expensive it was to ship items, the main purpose of their business, they closed the thread. Maybe I should start a forum (not likely), because I could get the last word every time.  All you have to do is type something in, then close the thread so no one can respond.

No, I wasn't abusive, I think I was polite and reasonable, but in your leisure (those of you that have any left), take a look for yourselves <link>.

Yes, yes, I know all about how much it costs to staff and support a shipping department.  I understand completely the inconvenience of getting stuff to the post office and organizing everything.  But, isn't that what got this business going?  The huge problem of shipping is why the distributor model came to be. One company skilled in making a product sells it to another that is skilled in warehousing and shipping.  I took that class in college too.

And, they just announced a solution that is actually pretty good; they have joined with DigiKey.  So, let's see what that would do to solve my particular problem: Darn, the pressure sensor isn't listed as an Adafruit item.  Ok, I'll just pretend it was, I'll pick some other item that weighs roughly the same and compare it between Adafruit and DigiKey; I chose the:

ADS1015 12-Bit ADC - 4 Channel with Programmable Gain Amplifier PID: 1083  $9.95

and the shipping was the same at $13.23.  At DigiKey, the shipping for it was $3,23, exactly $10.00 less.

Yes, I know that ordering more stuff would spread the shipping cost across multiple items making it easier to justify, but I don't want to order a bunch of items just to stockpile them for later, they get lost or chewed on by the squirrels that sometimes get into my garage.

But just to fill in the blanks, suppose I bought two of them; would that make it better?  Nope, two boards cost exactly the same for shipping at both Adafruit and DigiKey: $13.23 and $3.23. Too bad I don't need two barometric pressure sensors.

I was going to go back and compliment them on the deal they made with DigiKey, but they had closed the thread already, so I couldn't.  But, not to be forestalled, I commented instead on the cool looking blog post they did on DigiKey (just today mind you, after I complained publicly), but the comment hasn't appeared yet.  It's a moderated blog, so it may take a while for the comment to make it past the censor (free speech doesn't apply to someone else's blog), or it may not show up at all.  I'll check back later to see if they let the comment through <link>, or maybe I'll forget about it.

Don't misunderstand, I absolutely love Adafruit.  They constantly surprise me at their offerings and the support they give to their customers.  Great company, and I'll continue to deal with them, but order from one of their distributors because I just don't want to throw money down the drain on shipping costs. Note that SparkFun still uses USPS shipping and a similar purchase would be comparable to DigiKey shipping.  And, of course, sometimes the additional shipping cost is worth it; times when you want it before the weekend, something broke and you need it fixed right now, or the ton of other times when waiting a week or more is inconvenient. That's not the case most of the time.

The barometric sensor is on its way from China, but that ADC listed above looks like it would work for another project I have in mind.  And, I have an active DigiKey account.

Barometric Pressure, Wow, that was easy

$
0
0
My barometric pressure sensor came in today.  I posted about ordering it a short while ago because the shipping costs at Adafruit (my preference) were too cotton-pickin' high, and man, I was totally surprised when it came in today.  The other thing that surprised me was how small this thing is.

Sure, the pictures compare it to a coin and that should give a person a good idea how small it is, but when you unwrap it and have to actually look around to find it inside the shipping envelope, it comes home full force.  This thing, breakout board and all, is tiny.


Notice how small it is compared to the Arduino?  That means I have some choices in exactly how I implement the final installation.  Another thing that surprised me was how incredibly easy it was to implement the code to read both temperature and pressure from the device.  Yes, I was standing on the shoulders of giants that have trod before me, but that's what open source is all about.  First, I grabbed the Adafruit Unified Sensor Driver and tried it out.  I had compile errors in the examples supplied.  Frankly, that was annoying since I've never encountered that before at Adafruit, so I grabbed their older, deprecated, version 1 library specifically for the BMP-085 and tried it.

Worked first time.

So, fine, I'll just use the older library because it was getting late and I really don't want to hunt down some big company's compile error.  A deprecated library that works is perfectly OK with me.  Of course I made a few changes to the example because it gives the pressure in Pascals and the temperature in Celcius, I wanted mBar and Fahrenheit.  I also want the string I use to be a JSON string when I send it over an XBee to my Raspberry Pi.

So, about twenty or so minutes later I had this script:

#include "Wire.h"
#include "Adafruit_BMP085.h"

Adafruit_BMP085 bmp;
char buff1[50];

void setup() {
Serial.begin(9600);
bmp.begin();
}

void loop() {
char t[10], p[10];

float temperature = bmp.readTemperature()* 9/5 +32;
float pressure = bmp.readPressure()/100.0;
sprintf(buff1, "{barometer:{temperature:%s,pressure:%s}}\n",
dtostrf(temperature, 3, 1, t),dtostrf(pressure, 4, 1, p));
Serial.print(buff1);
delay(500);
}

Yes, that's all there is to implementing one of these devices.  How much easier can it possibly get? Here's what the output looks like:

{barometer:{temperature:74.5,pressure:930.4}}
{barometer:{temperature:74.5,pressure:930.4}}
{barometer:{temperature:74.5,pressure:930.4}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.4}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.4}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.4}}

I still have a bunch of work to do with this device, I have to hook it to an XBee, and install it into my Stephenson Screen out on the fence.  That will require waiting for the XBee shield I ordered for this project to come in.  I intend to put an Arduino, the shield, the XBee and this new BMP-085 all out on the fence in the weather and see what happens over time.  So expect at least one more post on this part of the project.

These days a basic Arduino clone can be bought for less than $10, an XBee shield for about the same and the BMP-085 for a few more dollars.  The accuracy on these things is quite good, and if necessary over time, calibration is a simple code change.  Nice project for a beginning weather buff that wants to go digital.

Have fun.

My Barometer, and Tips and Tricks With XBee Device Code.

$
0
0
In the last post <link> I put together a basic barometer to substitute for the silly one on the AcuRite console that came with my 5n1 weatherhead.  My XBee shield still hasn't come in, but I got a wild hair and decided to go ahead and put together the code while I waited.  As I was stealing pieces from various other projects I've done and combining it to implement this latest one, I realized there was a significant piece of code I haven't shared yet.

Early on I got tired of having my XBee set up wrong because I forgot something when I configured it and created a little piece of code to check the configuration to some degree at boot time on the Arduino.  By sending AT commands to the XBee, you can check things like software version, operating mode, etc.  Then if something was messed up, you can correct it easily without a long debugging session to discover some minor setting error.  I also use this to get the name of the device, and the address that it normally sends to.  Using this piece of code makes the entire device more configurable.  For example, you need to change to broadcast, instead of a specific address, to monitor the packets, just use XCTU to remotely change the DL, DH attributes and reset the Arduino so it reads the new address.  If you decide to change the device name, remotely change the NI attribute, and again, reboot the arduino.  I've been using this in one form or another now for a while to help work the kinks out of the latest device I'm working with.

Yes, you can change the configuration of the XBee that you stuck in the corner of the attic remotely,  This has been a capability of XCTU for a long time, and the latest version of XCTU makes it even easier.  To remotely reset an Arduino, I simply decode a command I send and then call location zero.

These two tiny tricks makes minor changes to the network really easy and can be done from my recliner.

The code that follows (below somewhere) is a simple thing with complete XBee interaction,  I have examples of sending formatted strings containing float data through the XBee, parsing a long unsigned integer out of an XBee message and using it, simulating the data portion of an XBee message for debugging, setting up a timer to send status periodically, reading some of the configuration out of the XBee to be sure it's OK before continuing the work.  That kind of thing.

Since the barometer code is so easy to understand, I thought that would be a perfect device to put all this stuff in and use as an illustration.  Go take a look at the barometer <link> then come back here and I'll step through some of the things that you can use in your own code to make the XBee a valuable device in do it yourself home automation.

First, just to get it out of the way, resetting the Arduino from software.  If you haven't hit it yet, you will, that time when you know a reset will cure a malfunction on the Arduino, but its up high, out in the yard, in the dark on the side of the house; somewhere hard or inconvenient to get to.  Wouldn't it be nice to just send a command to it and have it reset itself?  A software reset is easy to do.  If you need a hard reset (similar to pulling the plug) this won't do it; but those are not often needed.  Just put this line somewhere at the top of the code:

void(* resetFunc) (void) = 0; //declare reset function @ address 0

and when you need it, call it like this:

resetFunc();  //This will reset the board and start all over.

Yep, that's all there is to it.  I use this a lot for overcoming memory leaks in libraries that haven't been tested long enough, memory loss from using Streams in c++, that kind of problem.  It also helps overcome my own bugs from time to time.  As an example, in the code below I use it twice.  I have one software reset that is on a timer to reset the board every day around midnight.  This is to recover memory lost to whatever code does a malloc() and then doesn't remember to do a free() to release the memory back.  This is a very common problem (called memory leak) and has plagued tiny computers as long as they've existed.  When you want to run an Arduino 24/7, you have to code defensively.  The other instance is when I send a reset command over the XBee network to cause the board to reset.  I use this to test things like power failures, how long it will take to settle down, and to see if a reset will work before I get the ladder out to take a physical look at it.

Since I mentioned remotely resetting the board, I'll describe how I do that, but I need to explain how I use the IDE itself first.  Most of you already know that the Arduino IDE supports 'tabs', but the new folk haven't caught up to that yet.  Tabs are actually separate source files that the IDE stores in the same directory as the sketch.  These are extremely nice for organizing code so you don't have to scroll through a couple of hundred lines of code to get to something you want to work on.


What you're looking at above is the Arduino IDE with three tabs.  One is for the bulk of the barometer code, the next is for some utility routines I use a lot, and the one selected is the XBee code I use on a lot of my devices.  I opened the window that controls tabs so the new folk can get a look at the tool provided for working with tabs.  Remember, each tab is actually a separate file in the sketch directory.  When you compile something, the files are all concatenated together and passed to the compiler, this is just a convenience to help you organize your project.  There's a couple of tutorials out there that mention tabs, and I've heard of a book or two that do, but most of the instructions are great at telling you how to install it, but not how to use it effectively.  So, don't be afraid to poke around and experiment.

No, I'm not going to write a book about it.  If I did, they'd change it and I'd have to have a second revision.  I've got projects to work on instead.

Anyway, The picture shows how I decode the command and reset the board.  Over time I've worked out my own particular method of sending and receiving packets that works well for me.  Everything is done in a string; I don't try to send integers, floats or compact the data.  For home automation, interactions are measured in a minimum of seconds and plain text traveling between the machines is perfectly adequate.  It also makes monitoring what's going on much easier since I don't have to decode anything to read it, it's already text.  So my command to reset the barometer is:

Barometer,Reset

I can type that right into an XCTU session from my recliner and the barometer out on the fencepost (where it will eventually reside) will reset itself.  Yes, I have to construct a packet in API mode 2 format, but XCTU has a packet generator that works pretty well to help with that.  

Now, let's talk about AT commands to the XBee itself.  When you look at the various tutorials on the web, they love to type AT into the XBee console and look for the OK that comes back.  Then they teach you how to set up the XBee using these commands.  This is good, but not very useful if you're going to actually do a project that will run all the time and have to survive a lot of the normal problems such as RF noise, distance, flaky devices, etc.  For that you have to go to API mode and send fully constructed packets between machines.  I have already covered the various pitfalls and have instructions on how to set this up on my XBee page, so I won't go over it again here, go here <link> for that when you need it.  What I want to show you is how to send an AT command, get something from the XBee and then use it.  Doing this you can double check the configuration, get the device name, etc.  It's a really good feature to understand.  As an example, here are the first few lines executed on my barometer:


I know, it's a dumb screenshot.  I did this on purpose, to illustrate again how tabs can work for you.  I have selected the barometer tab and scrolled down a bit to the setup() routine.  See how you can flip between tabs to do things instead of scrolling though tons of lines of code and losing your train of thought?  I like this feature.

Anyway, I set up the XBee code, then call getDeviceParameters().  This routine sends a series of AT commands to gather various items and then returns.  Then I display the parameters so I can tell if I have the XBee set up the way I want it.  I actually store the string 'Barometer' in the NI command; this is the name of the device and I use it to check incoming packets to see if they're for this device and include it in the out going status packets I send.  This way I can change the name of the device by simply changing the parameter on the XBee.  Here's the code that does this:

void getDeviceParameters(){
uint32_t addrl = 0;
uint32_t addrh = 0;
uint32_t daddrl = 0;
uint32_t daddrh = 0;

if(sendAtCommand((uint8_t *)"VR")){
if (atResponse.getValueLength() != 2)
Serial.println(F("Wrong length in VR response"));
deviceFirmware = atResponse.getValue()[0] << 8;
deviceFirmware += atResponse.getValue()[1];
}
if(sendAtCommand((uint8_t *)"AP")){
if (atResponse.getValue()[0] != 0x02)
Serial.println(F("Double Check the XBee AP setting, should be 2"));
}
if(sendAtCommand((uint8_t *)"AO")){
if (atResponse.getValue()[0] != 0)
Serial.println(F("Double Check the XBee A0 setting, should be 0"));
}

if(sendAtCommand((uint8_t *)"NI")){
memset(deviceName, 0, sizeof(deviceName));
for (int i = 0; i < atResponse.getValueLength(); i++) {
deviceName[i] = (atResponse.getValue()[i]);
}
if (atResponse.getValueLength() == 0){
Serial.println(F("Couldn't find a device name"));
}
}
if(sendAtCommand((uint8_t *)"SH")){
for (int i = 0; i < atResponse.getValueLength(); i++) {
addrh = (addrh << 8) + atResponse.getValue()[i];
}
}
if(sendAtCommand((uint8_t *)"SL")){
for (int i = 0; i < atResponse.getValueLength(); i++) {
addrl = (addrl << 8) + atResponse.getValue()[i];
}
}
ThisDevice=XBeeAddress64(addrh,addrl);
if(sendAtCommand((uint8_t *)"DH")){
for (int i = 0; i < atResponse.getValueLength(); i++) {
daddrh = (daddrh << 8) + atResponse.getValue()[i];
}
}
if(sendAtCommand((uint8_t *)"DL")){
for (int i = 0; i < atResponse.getValueLength(); i++) {
daddrl = (daddrl << 8) + atResponse.getValue()[i];
}
}
Controller=XBeeAddress64(daddrh,daddrl);
}

uint8_t frameID = 12;

boolean sendAtCommand(uint8_t *command) {
while(1){
frameID++;
atRequest.setFrameId(frameID);
atRequest.setCommand(command);
// send the command
xbee.send(atRequest);
//Serial.println("sent command");
// now wait a half second for the status response
if (xbee.readPacket(500)) {
// got a response!

// should be an AT command response
if (xbee.getResponse().getApiId() == AT_COMMAND_RESPONSE) {
xbee.getResponse().getAtCommandResponse(atResponse);

if (atResponse.isOk()) {
//Serial.println("response OK");
if (atResponse.getFrameId() == frameID){
//Serial.print("Frame ID matched: ");
//Serial.println(atResponse.getFrameId());
return(true);
}
else {
Serial.println("Frame Id did not match");
}
}
else {
Serial.print(F("Command returned error code: "));
Serial.println(atResponse.getStatus(), HEX);
}
}
else {
Serial.print(F("Expected AT response but got "));
Serial.println(xbee.getResponse().getApiId(), HEX);
}
}
else {
// at command failed
if (xbee.getResponse().isError()) {
Serial.print(F("Error reading packet. Error code: "));
Serial.println(xbee.getResponse().getErrorCode());
}
else {
Serial.println(F("No response from radio"));
}
}
}
}

It's not perfect code, and it doesn't serve the 'general case', but it is easy to understand and does the job.  Notice in the routine sendAtCommand() that I hang up in an infinite loop waiting for a response to each AT comand I send?  Yes, that's on purpose.

On a network like mine where there are a number of XBees sending data at essentially random intervals, traffic can be mixed in with the responses to the AT commands.  I send a request for the software version at the same time as a time update is received from my house clock.  The XBee will send the packet from the clock first and I could miss the response to the AT command.  Also, the receive for the packets is interrupt driven and characters can be missed by simultaneous sends and receives.  To overcome these kinds of problems, I wait for the response about a half second, if it doesn't come back, I try it again.  This guarantees that I will accumulate the items I need during initialization regardless of traffic conditions, it just may take a couple of tries to do it.  It's actually kind of fun to watch it during high traffic conditions; it tries each command as many times as necessary to gather the data and then returns to let setup() print them out.  Here's what the output looks like when I run it:

Barometer Version 1 Init...
This is device Barometer
Device Firmware version 2370
This Device Address is 0x0013A200 0x4089D915
Default Destination Device Address is 0x0013A200 0x406F7F8C
Setup Complete
Mem = 276
Barometer: temperature: 77.5, uncorrected pressure: 937.8
See how the name of the device, long address, version, destination long address can be gathered directly from the XBee?  This is nice to guarantee that you set things up correctly when you try your code out.

Now, let's look at sending the actual data that I gathered from the barometer chip.  I've learned to like JSON encoding as a method of passing data and am converting all my devices to send data using this format.  I haven't gotten very far on that project, but the barometer is new, so I use it here.  I use this line to construct the status text that is sent:

void sendStatusXbee(){
float temperature = bmp.readTemperature()* 9/5 +32;
float pressure = bmp.readSealevelPressure(altitude)/100.0;
sprintf(Dbuf, "{\"%s\":{\"temperature\":\"%s\",\"pressure\":\"%s\",\"utime\":\"%ld\"}}\n",
deviceName, // originally read from the XBee
dtostrf(temperature, 1, 1, t), // This is a text conversion of a float
dtostrf(pressure, 1, 1, p), // same as above
now()); // current time in seconds
Serial.print(Dbuf); // notice this is only for the serial port
sendXbee(Dbuf); // out to the XBee
Serial.println("Pressure message sent");
}

I use sprintf() because the documentation is all over the web and there are tons of examples out there for people to follow.  The problem is that the Arduino sprintf() doesn't support floating point, but does have a handy routine dtostrf() that will turn a float variable into a nice little string that can be used.  Pay particular attention to the format string I used; you don't have to do it this way.  You can assemble any string you want to and send it instead.  Like I said though, JSON is already documented on the web and I don't have to invent anything this way.

Now, I want to show you how I control where the message is sent.  I read the default address from the XBee DH and DL parameters during initialization and put it into the variable 'Destination', now I get to use the variable:

void sendXbee(const char* command){
ZBTxRequest zbtx = ZBTxRequest(Destination, (uint8_t *)command, strlen(command));
xbee.send(zbtx);
}

This will take the string I just constructed and put it into an XBee packet, set the address I mentioned and send it.  To change the address such that the message is broadcast to all devices, I decode a command and just change the value of Destination:

    // The ability to broadcast is sometimes used to see what it going on
// using an XBee plugged into a laptop to monitor the data.
else if (strcmp(nxtToken, "SendBroadcast") == 0){
Serial.println(F("Switching Address to Broadcast"));
Destination = Broadcast; // set XBee destination address to Broadcast for now
}
else if (strcmp(nxtToken, "SendController") == 0){
Serial.println(F("Switching Address to Controller"));
Destination = Controller; // set XBee destination address to Broadcast for now
}

This is just like the Reset command above; I send

Barometer,SendBroadcast

and it switches the Destination variable to the broadcast address and from then on every XBee in the network will see the message.  To put it back to normal:

Barometer,SendController

And, this brings up another feature I like to have on my remote devices, a nice way to test the commands without having to drag out an XBee breakout board every time I want to add or try something.  Wouldn't it be nice to be able to simulate the receipt of a command through the IDE serial display?  I worked up this code to allow that very thing:

  // This allows emulation of an incoming XBee command from the serial console
// It's not a command interpreter, it's limited to the same command string as
// the XBee. Can be used for testing
if(Serial.available()){ //
char c = Serial.read();
if(requestIdx < sizeof(requestBuffer)){
requestBuffer[requestIdx] = c;
requestIdx++;
}
else{ //the buffer filled up with garbage
memset(requestBuffer,0,sizeof(requestBuffer)); //start off empty
requestIdx = 0; // and at the beginning
c = 0; // clear away the just-read character
}
if (c == '\r'){
requestBuffer[requestIdx] = '\0';
// Serial.println(requestBuffer);
// now do something with the request string
processXbee(requestBuffer,strlen(requestBuffer));
//start collecting the next request string
memset(requestBuffer,0,sizeof(requestBuffer)); //start off empty
requestIdx = 0; // and at the beginning
}
}

This fragment will allow me to type a command that would normally come in over the XBee right into the IDE serial monitor and send it to the code.  It does this by just gathering the characters and calling the same routine I use to decode XBee commands.  This cuts short the time it takes to add another command to the device.

Now, normally, I would post the code of the entire module, but I bet you've had enough code samples in windows for one session.  Instead, I finally got the Arduino code I'm using for the various things into GitHub.  This entire module is in the 'Barometer' directory so you can grab it and modify away.  It's been running for a few days and hasn't blown up yet, but I'll certainly be adding to it over time.

If you go looking in the other directories don't expect all of these features to be in each thing.  Like I said, I developed these tricks over time and only recently started to go back and update the older devices.  Heck, I haven't changed the thermostats at all in months.  One of the modules hasn't even been compiled under an IDE version in a couple of years.  I'll work through the devices one at a time as I have the chance, and probably come up with another trick that I'll want to fit into the others.  It never ends.

Here's the link to GitHub  <link>. Have fun.

Let's Finish Off the Barometer Project.

$
0
0
In the last entry <link> I posted the code for my barometer; I took a simple program of a few lines and turned it into a monster that does everything imaginable.  Now, I should actually set it up to work and put it in service.  Yes, my XBee shield came in and worked fine first try, so I mounted the little breakout board on it and added four wires to connect the barometer chip to the Arduino pins and tested it.


Made a nice little package when I got it all put together, so I dismantled the Stephenson Screen on my fence and tie wrapped it inside.


I had the old temperature sensor in there for years with no problems, so I don't think I need to get any fancier than that.

One thing I did in the code that I didn't mention in the last post was how to correct the reading to adjust for sea level comparisons.  See, to make barometer reading useful around the world, weather folk adjust them for the altitude so that there is a common base for comparison.  There's some really good explanations for why and how to do this out there, so if you don't already understand this, do a quick search.

From our perspective, this is incredibly easy; just call a different routine to read the data:

float pressure = bmp.readSealevelPressure(altitude)/100.0;

This statement will get the pressure adjusted for 'altitude' and the division by 100 will convert it from Pascals to millibars.  If you want inches of mercury, just look up the conversion factor.  The Adafruit library has already got the code for the conversion, all you have to do is look up your altitude in Google Maps, or read it off a GPS at the location where you put the barometer.

I've already adjusted my house monitor code to use this device and if you look at my house display, the outside temp and barometric pressure are taken from the new set up. I was lucky enough to catch a small storm so I could compare it with the other stations in my area, and it was right on.  It's really great when a sensor does exactly what the datasheet says it will.

I really like this sensor; the temperature measurement is very stable as is the barometer.  It's a snap to read and only takes two signal wires for all of it.  Since there's room on the device for more sensors, I may think about luminance or something in the future.

All this playing with the AcuRite weatherstation has been fun, and now I'm going to look at conquering the RF signal it sends.  What I'm going for now is far, far away from what I started off to do, but isn't that how it goes?  I want to grab the RF directly and use that data along with any other sensors I need to add and bring up a separate Raspberry Pi to handle weather presentation. Remember, the code is already in GitHub for you to grab <link>.

Funny how these projects tend to take on a life of their own.

Suddenly, I'm Famous

$
0
0
Heh, this blog is the "Engineering Site of the Day" at EEWeb.com.  I only get this privilege for a day, so check it out quickly:

http://www.eeweb.com/websites/desert-home

I've used the site several times looking for cool devices and ideas.  You know, when you just want to step through the various advertisements and announcements to see what the new things are.  They also have a forum where you can ask questions like "How do I ...," and "what device will ..." The nice thing is there isn't the collection of snarky replies that usually come from a question like that.

No, I have no idea why they chose me other than they recognize tenacity.  I can't be the beautiful prose and superior artistry of my blog.

So, go to their site and prowl around. Then, for folk like us, go to the embedded tab and really look around, There are some articles in the embedded section that really caught my eye.  Especially one on sequencing power supplies <link>.  I'm looking at that problem right now where I have a 5V device hooked to a 3V device and I want to put them to sleep.  How the heck does one do that?

Now, a screenshot in case they decide they made a horrible mistake and pull it.  I want proof.


I'm so proud.
Viewing all 218 articles
Browse latest View live