Controlling Wemo Light Switches, Another Look

No, I'm not dead, and yes, I still work on electronic projects.  I've been busy with projects that are much more mundane and normal.  Things like trying to build a pad for my motorcycle trailer, fighting with my chlorine generator for the pool, and recovering from my house flooding during the monsoon rains here in AZ.  Yep, I had water come in one side of the house and go out the other.  I'm still working on the damage from that episode.

But people don't read my posts to hear about that kind of thing, they probably have worse problems around the house than I do ... I don't get snow or tornadoes.

For months now I've had Wemo light switches running.  They're a nice little device that Belkin puts out and seem to be gaining in popularity.  Heck, Home Depot sells them in the electrical section now.  However, they rely on an app on the cell phone and a cloud server.  That may not be a concern for most people, but I personally hate having things I depend on subject to the whim of a company that might just decide to start charging me for the service in the future.  That's why I put together some software to run the switches from my Raspberry Pi based house controller <link><link><link>.

However the library and toolset I used, while a great effort and really classy, was too complex and kept failing.  Let me be perfectly clear, the ouimeaux library I used was good, the problem is that the wemo switches don't always work like Universal Plug and Play (upnp) devices are supposed to.  I don't want to get into a lengthy description of upnp here, there are a bunch of those out there, but suffice it to say that the subscription capabilities of the switches fails.

So, you discover the devices, set up a subscription where the devices let you know when the state changes happen, and then sometime later, the switch quits responding.  That means sometime or other, the light doesn't turn on or off and you have to restart the process that controls them to get the switch working again.  Additionally, the ouimeaux library uses a LOT of other tools and provides features that I'm just not interested in.  It uses gevent to handle asynchronous operations, has a little web server, client software, enough stuff for a major project.  I just want to turn them on and off and look to see which way they are now.  Using something large like this to do a simple job leads to huge problems when something fails.  There's just so much there that you can go nuts chasing a problem.

You guessed it, I wrote my own.  Well, that's actually an exaggeration.  I stole some stuff and used it would be closer to the truth.  I started out by writing a handler to discover upnp devices on my network.  If you've never done that, I recommend you look into it just for fun.  I discovered my satellite receiver, my Roku, the DSL modem, all the Wemo switches, and several other devices that I didn't know had plug and play interfaces.  That was a royal pain.  I had to learn about SSDP, SOAP, multicast, XML, and the list was growing.  Shoot, all I want to do was control the darn switches, not create a dissertation on half the jargon on the web.  I did learn a lot about discovering the devices on my network, but that wasn't my goal.

That led me to a few days of searching on the web for upnp libraries.  They're out there, and some of the are really good, but they're written in something besides python.  Sure a couple  have interfaces, but the documentation went totally over my head.  There were a some in python, but a couple had disappeared off the web, and one just didn't work (the samples even had syntax errors).  What should I do?  I remembered reading somewhere that python can import modules, and that the python tool Miranda worked reasonably well to experiment with upnp devices ... hmmm.

I got the latest Miranda and combined it with my own code and came up with a way to read and control the switches that uses only the basic python libraries.  It was a challenge hooking into someone else's code that was designed to run as a standalone tool, but it worked.  That way I leveraged the huge amount of work the author of Miranda did to make my job easier.  The big bonus was that Miranda is also really good as a upnp learning tool.  Ha, I have a debugger and a controller all in one.

Since I had spent a lot of time debugging the switch operation to determine why they would quit, I didn't want to use the subscription service.  The subscription facility is asynchronous, so the switch will send an XML message over UDP to a registered URL any time the state changes (off to on); too bad it doesn't work all the time.  That means that I would have to poll the switches to see if something or someone other than my code on the controller had changed their state.  Let me elaborate on that a bit.  These are wall switches; I can walk over and push the button and turn off the lights.  Since my software can't see that, it has to ask the switch what state it is in right now.

That's fine, I can poll it every couple of seconds and see what is happening, BUT there's a problem with that.  I love APScheduler a great timer tool in python; I have it in almost everything I do, but it uses threads to control when things happen.  Using threads and database updates is a path to disaster.  When I tried my new code out, about half the time it would leave the database locked because something jerked execution away at an inopportune time.  I even updated my code and libraries to the latest version of APScheduler which allows jobs to be paused, and still had the problem.  That meant that I had to figure out how to schedule polling of the switches within a single thread.

After a few magnificent failures I came up with a pretty simple way to do it; my method is in the code below.  I may have to use this in some of the previous work I've done, it's really small and simple, so it could decrease the load on the poor little Pi.

Even though this took an inordinate amount of time and study, it's still pretty darn nice.  I haven't ran it for months, but it's held up for a short time pretty well.  I could well develop problems with sockets hanging or something, but the code is simple and straight forward, so I should be able to isolate whatever problems turn up and fix them.  I don't have eight libraries of very complex code written by someone else to step through trying to find out how something works.  It's me and Miranda here, ... well there are the system libraries, but nothing terribly exotic.

Here's the code.  It's written in my usual style: variable names that are too darn long, comments for practically every darn line, over indentation, etc.  Basically everything I need to pick it up a year from now to fix a bug or add a feature.

#! /usr/bin/python
from miranda import upnp
from miranda import msearch
from miranda import set
import sys
#from apscheduler.schedulers.blocking import BlockingScheduler
import datetime
from datetime import timedelta
from datetime import datetime
import time
import sysv_ipc
import logging
import sqlite3
import pdb #yes, I had trouble and had to use this !!
#-------------------------------------------------
# the database where I'm storing stuff
DATABASE='/home/pi/database/desert-home'

def lprint(text):
 print time.strftime("%A, %B, %d at %H:%M:%S"),text
 sys.stdout.flush()

def _send(action, whichone, args):
 if not args:
  args = {}
 entry = (item for item in lightSwitches if item["name"] == whichone).next()
 index =entry['index']
 host_info = conn.ENUM_HOSTS[index]
 device_name = 'lightswitch'
 service_name = 'basicevent'
 controlURL = host_info['proto'] + host_info['name']
 controlURL2 = host_info['deviceList'][device_name]['services'][service_name]['controlURL']
 if not controlURL.endswith('/') and not controlURL2.startswith('/'):
  controlURL += '/'
 controlURL += controlURL2

 resp = conn.sendSOAP(
  host_info['name'],
'urn:Belkin:service:basicevent:1',
  controlURL,
  action,
  args
 )
 return resp

def get(whichone):
'''
 Returns True if the light is on and False if not
'''
 resp = _send('GetBinaryState', whichone, {})
 tagValue = conn.extractSingleTag(resp, 'BinaryState')
 return 'On' if tagValue == '1' else 'Off'

def handleUpdate(whichone, status):
 for i in lightSwitches:
  if i['name'] == whichone:
   i['status'] = status
 updateDatabase(whichone, status)

def on(whichone):
"""
 BinaryState is set to 'Error' in the case that it was already on.
"""
 resp = _send('SetBinaryState', whichone, {'BinaryState': (1, 'Boolean')})
 tagValue = conn.extractSingleTag(resp, 'BinaryState')
 status = 'On' if tagValue in ['1', 'Error'] else 'Off'
 handleUpdate(whichone, status)
 lprint("turned %s on"%(whichone))
 return status

def off(whichone):
"""
 BinaryState is set to 'Error' in the case that it was already off.
"""
 resp = _send('SetBinaryState', whichone, {'BinaryState': (0, 'Boolean')})
 tagValue = conn.extractSingleTag(resp, 'BinaryState')
 status = 'Off' if tagValue in ['0', 'Error'] else 'On'
 handleUpdate(whichone, status)
 lprint("turned %s off"%(whichone))
 return status

def doLights():
 for switch in lightSwitches:
  thisOne = switch['name']
  updateDatabase(thisOne,get(thisOne))

def doComm():
 global firstTime
 #global scheditem

 try:
  if (firstTime):
   while(True):
    try:
     # commands could have piled up while this was 
     # not running. Clear them out.
     junk = Cqueue.receive(block=False, type=0)
     print "purging leftover commands", str(junk)
    except sysv_ipc.BusyError:
     break
   firstTime=False
  while(True):
   newCommand = Cqueue.receive(block=False, type=0)
   # type=0 above means suck every message off the
   # queue.  If I used a number above that, I'd
   # have to worry about the type in other ways.
   # note, I'm reserving type 1 messages for 
   # test messages I may send from time to 
   # time.  Type 2 are messages that are
   # sent by the php code in the web interface.
   # Type 3 are from the event handler. This is just like
   # the house monitor code in that respect.
   # I haven't decided on any others yet.
   handleCommand(newCommand)
 except sysv_ipc.BusyError:
  pass # Only means there wasn't anything there 


def handleCommand(command):
 #lprint("" + str(command))
 # the command comes in from php as something like
 # ('s:17:"AcidPump, pumpOff";', 2)
 # so command[0] is 's:17:"AcidPump, pumpOff'
 # then split it at the "  and take the second item
 try:
  c = str(command[0].split('\"')[1]).split(',')
 except IndexError:
  c = str(command[0]).split('')   #this is for something I sent from another process
 #lprint(c)
 if (c[0] == 'OutsideLightsOn'):
  outsideLightsOn()
 elif (c[0] == 'OutsideLightsOff'):
  outsideLightsOff()
 elif (c[0] == 'fPorchToggle'):
  toggle("frontporch")
 elif(c[0] == 'garageToggle'):
  toggle("outsidegarage")
 elif (c[0] == 'cactusToggle'):
  toggle("cactusspot")
 elif (c[0] == 'patioToggle'):
  toggle("patio")
 else:
  lprint(" Weird command = " + str(c))

def outsideLightsOn():
 lprint (" Outside lights on")
 on("outsidegarage")
 on("frontporch")
 on("cactusspot")

def outsideLightsOff():
 lprint (" Outside lights off")
 off("outsidegarage")
 off("frontporch")
 off("cactusspot")

def toggle(whichOne):
 if (get(whichOne) == 'On'):
  off(whichOne)
 else:
  on(whichOne)

def keepAlive():
'''
 For my own purposes, I update the database periodically with the time
 so I can check to see if things are holding together.  I currently use the
 time in the light switch records for this.
'''
 lprint(" keep alive")
 for switch in lightSwitches:
  thisOne = switch['name']
  updateDatabase(thisOne, get(thisOne), force=True)

def updateDatabase(whichone, status, force=False):
'''
 This is running on a Pi and is not event driven, so polling like
 this will result in considerable wear to the SD card.  So, I'm going to 
 read the database to see if it needs to be changed before I change it.  
 According to everything I've read, reads are free, it's the writes that
 eat up the card.
'''
 dbconn = sqlite3.connect(DATABASE)
 c = dbconn.cursor()
 c.execute("select status from lights where name = ?;",
  (whichone,))
 oldstatus = c.fetchone()
 if oldstatus[0] != status or force == True:
  lprint ("Had to update database %s, %s"%(whichone, status))
  c.execute("update lights "
"set status = ?, utime = ? where name = ?;",
   (status, time.strftime("%A, %B, %d at %H:%M:%S"), whichone))
  dbconn.commit()
 dbconn.close()

if __name__ == "__main__":
 #When looking at a log, this will tell me when it is restarted
 lprint (" started")
 firstTime = True
 debug = False
 if not debug:
  conn = upnp(False,False,None,0)
'''
  I don't want the search for devices to run forever 
  So, I set the timeout for miranda to some number of seconds
  to limit it.
'''
  set(3, ["set","timeout", "10"], conn)
'''
  This looks at the devices that responded and gathers more data about
  them by sending them a request to itemize their capabilities.

  Sometimes a upnp device goes nuts and responds way out of
  proportion.  You can get the same device in the tables
  many times, so set the uniq to True

  Also, the Wemo switches don't always respond to a discover specific to 
  them.  That means I have to do a general discover and get all the devices
  on the network.  This sucks because it slows things down, so if anyone
  overcomes this problem, let me know how.
'''
  set(3, ["set","uniq", True], conn)
''' This is the actual search '''
  msearch(1,[msearch],conn)
''' and now do the interaction '''
  for index, hostInfo in conn.ENUM_HOSTS.iteritems():
   #print "************** ", index, " of ", len(conn.ENUM_HOSTS) - 1
''' on my network, I have a rogue device that reports badly '''
   if hostInfo['name'].find('192.168.16.254') == 0:
    print "Odd device, ignoring"
    continue
''' if you want to see them as they come in, uncomment this '''
   #print hostInfo
   if hostInfo['dataComplete'] == False:
    xmlHeaders, xmlData = conn.getXML(hostInfo['xmlFile'])
    conn.getHostInfo(xmlData,xmlHeaders,index)

'''
  now to select only the light switches from the various devices 
  that responded 
'''
  lightSwitches=[]
  for index, host_info in conn.ENUM_HOSTS.iteritems():
   if "deviceList" in host_info:
    if "lightswitch" in host_info["deviceList"]:
     name = host_info["deviceList"]["lightswitch"]["friendlyName"]
     lightSwitches.append({"name": name, "index": index, "status" : 'unknown'})
'''
  OK, now I have the list of Wemo light switches that are around the 
  house, so print it and show the state of each one 
'''
  print "this is the list of the", len(lightSwitches), "Wemo switches found."
  for switch in lightSwitches:
   switch['status'] = get(switch['name'])
   print switch

 # Create the message queue where commands can be read
 # I just chose an identifier of 13 because the house monitor
 # already took the number 12.
 Cqueue = sysv_ipc.MessageQueue(13, sysv_ipc.IPC_CREAT,mode=0666)
'''
 This is a poor man's timer for task control.  I may put this in a class
 after I've run it for a while.  The reason I did it this way is that 
 APSscheduler creates a separate thread to handle timers and I don't 
 want the contention to the database of separate threads doing things
 that way.

 To use it, just put another entry into the table.
'''
 lprint (" Setting up timed items")
 startTime = int(time.time())
 tasks = [{'whichOne': doLights,'interval' : 2, 'next': startTime+10},
  {'whichOne': keepAlive, 'interval' : (4*60), 'next' : startTime+15}]
 lprint ("going into the processing loop")
 print startTime
 while True:

  #pdb.set_trace()
  now = int(time.time())
  for task in tasks:
   if task['next'] <= now:
    task['next'] = now + task['interval']
    #print task['whichOne']
    task['whichOne']()
  doComm()
'''
  doing a sleep here releases the cpu for longer than the program runs
  That way I reduce the load on the machine so it can do more stuff
'''
  time.sleep(0.25) 
  pass 

 sys.exit("done");

It's only fair to thank Ian McCracken who wrote the Ouimeaux library, Isaac Kelly who did much of the initial discovery work, and Craig Heffner the author of Miranda.  My work is built on theirs.

No, I don't have support in here for the Wemo motion sensor, crock pot, or other items they may come up with.  If you want to grab it and extend it to support those devices, feel free; just let me know so I can get the improvements as well.

There's also support in here for the sysv messages I use to communicate between processes on the machine.  Feel free to rip that out and put in anything you want.  I wanted to give you an example that I'm actually running right now, not something I cut up to look pretty.

Just for fun, here's the output of the code that is captured to my log:

Sunday, October, 05 at 16:22:06  started
action timeout
action uniq
Show unique hosts set to: True
argc 1
argv [<function msearch at 0xb69d94f0>]
Entering discovery mode for 'upnp:rootdevice', Ctl+C to stop...

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.34:49312
XML file is located at http://192.168.0.34:49312/device.xml
Device is running Linux/2.6.37.6-4.0, UPnP/1.0, Portable SDK for UPnP devices/1.
6.17
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.34:49200
XML file is located at http://192.168.0.34:49200/device.xml
Device is running Linux/2.6.37.6-4.0, UPnP/1.0, Portable SDK for UPnP devices/1.
6.17
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.28:49154
XML file is located at http://192.168.0.28:49154/setup.xml
Device is running Unspecified, UPnP/1.0, Unspecified
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.29:49153
XML file is located at http://192.168.0.29:49153/setup.xml
Device is running Unspecified, UPnP/1.0, Unspecified
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.43:49154
XML file is located at http://192.168.0.43:49154/setup.xml
Device is running Unspecified, UPnP/1.0, Unspecified
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.26:49153
XML file is located at http://192.168.0.26:49153/setup.xml
Device is running Unspecified, UPnP/1.0, Unspecified
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.22:2869
XML file is located at http://192.168.0.22:2869/upnphost/udhisapi.dll?content=uu
id:3a682b82-e803-4105-bfeb-114ed775cab1
Device is running Microsoft-Windows/6.3 UPnP/1.0 UPnP-Device-Host/1.0
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.14:8060
XML file is located at http://192.168.0.14:8060/
Device is running Roku UPnP/1.0 MiniUPnPd/1.4
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.1:80
XML file is located at http://192.168.0.1:80/DeviceDescription.xml
Device is running ZyXEL-RomPlug/4.51 UPnP/1.0 IGD/1.00
****************************************************************


Discover mode halted...
Failed to get argument name for OpenInstaAP
Failed to get argument name for CloseInstaAP
Failed to get argument name for OpenInstaAP
Failed to get argument name for CloseInstaAP
Failed to get argument name for OpenInstaAP
Failed to get argument name for CloseInstaAP
Failed to get argument name for OpenInstaAP
Failed to get argument name for CloseInstaAP
Caught exception while parsing device service list: list index out of range
At index  7
Caught exception while parsing device service list: list index out of range
At index  8
this is the list of the 4 Wemo switches found.
{'status': 'Off', 'index': 2, 'name': 'cactusspot'}
{'status': 'Off', 'index': 3, 'name': 'frontporch'}
{'status': 'Off', 'index': 4, 'name': 'patio'}
{'status': 'Off', 'index': 5, 'name': 'outsidegarage'}
Sunday, October, 05 at 16:22:42  Setting up timed items
Sunday, October, 05 at 16:22:42 going into the processing loop
1412551362
Sunday, October, 05 at 16:22:56 Had to update database patio, On
Sunday, October, 05 at 16:22:56 turned patio on
Sunday, October, 05 at 16:22:57  keep alive
Sunday, October, 05 at 16:22:57 Had to update database cactusspot, Off
Sunday, October, 05 at 16:22:57 Had to update database frontporch, Off
Sunday, October, 05 at 16:22:57 Had to update database patio, On
Sunday, October, 05 at 16:22:58 Had to update database outsidegarage, Off
Sunday, October, 05 at 16:23:04 Had to update database patio, Off
Sunday, October, 05 at 16:23:04 turned patio off
Sunday, October, 05 at 16:23:12  Outside lights on
Sunday, October, 05 at 16:23:12 Had to update database outsidegarage, On
Sunday, October, 05 at 16:23:12 turned outsidegarage on
Sunday, October, 05 at 16:23:12 Had to update database frontporch, On
Sunday, October, 05 at 16:23:12 turned frontporch on
Sunday, October, 05 at 16:23:12 Had to update database cactusspot, On
Sunday, October, 05 at 16:23:12 turned cactusspot on
Sunday, October, 05 at 16:23:17  Outside lights off
Sunday, October, 05 at 16:23:17 Had to update database outsidegarage, Off
Sunday, October, 05 at 16:23:17 turned outsidegarage off
Sunday, October, 05 at 16:23:17 Had to update database frontporch, Off
Sunday, October, 05 at 16:23:17 turned frontporch off
Sunday, October, 05 at 16:23:17 Had to update database cactusspot, Off
Sunday, October, 05 at 16:23:17 turned cactusspot off