Well, I did it. I finally broke down and bought a NAS (network attached Storage). I researched these things for days looking for something that would fit my needs and couldn't make up my mind. I was tempted to just take a Raspberry Pi and add a disk drive, but I wanted more capability in the device than that would have given me. I asked other folk, and it came down to two manufacturers of this kind of device: Synology and QNAP. Frankly, they both sound great and have more good reviews than bad.
I settled on QNAP, not because it won out in the specification, but because I caught a nice sale. TigerDirect had the QNAP TS-231 on sale for $30 bucks off, so I ordered it. I got two 3TB WD red drives from Amazon and set it up. It's up in my attic data center working its little heart out.
When I was looking at what could be run on it, MySQL was listed. That meant that I could put my house databases up there and get them off the Pi's I have running. It also meant that I could expand the database and actually log data locally for my house.
Wow, that would mean all my griping about cloud services would end.
First step, set up a MySQL database on the server; that took about 10 minutes. Then understand how to put data on it; that took a day. Then convert my code to use the new databases on the NAS; that took a stinking week. The very basic use of MySQL is different enough from the way to use SQLite3 that I had to learn a whole lot to get it working.
But, when I did, it really sped things up. I'm not clear on why exactly; I'm making data base transactions across an Ethernet link instead of locally, but it runs faster. Since the NAS is set up Raid 1, I have 3TB to play with and I expect the databases to use a hunk of that so it may slow down over time as the data bases grow.
My impetus for finally biting the bullet on one of these devices was storage failures on the Raspberry Pi. That's the most significant problem I've had with the Pi, the SD cards wear out, and I had worse luck using a USB stick. Well, I won't be wearing the cards out as fast, and I don't need the extra space of a USB stick anymore. However, the SD cards will still wear out because all the system logs and such are beating on them.
To this end, I'm going to put the operating system on the NAS as well, but I haven't gotten that far yet. The idea is to put everything I can on the NAS and only boot the Pi from the SD card. That should make it last a while. Sort of the best of both worlds; the separate processor and hardware of the Pi combined with the storage and reliability of a NAS. I'll inherit the possibility of things quitting if the Ethernet fails, but my house already relies on Ethernet anyway. I have the various Arduinos set up to work in a default mode, so that problem hasn't gotten any worse ... I think.
One of the side-effects (affects?? I can never remember which one to use) is that I cleaned up a bunch of code, removed things I really wasn't using any more and restructured a lot of my processes. So, most of the processes that wrote data to the various cloud services I looked into are gone now. The code is still out there on GitHub (the internet is forever), but it's not in the latest update. If you need it, just look at a previous commit.
Another thing that changed is since I'm not updating the various clouds, the live graphs I have scattered in earlier posts may not work any more. I replaced some of them with pictures to keep it somewhat coherent, but gave up after a few. Also, the architecture of the overall house control and monitoring is a bit different. I guess I'll delete the page on that and try to create a new picture and description soon.
I'm keeping my legacy feeds on Xively running for now since the majority of my data is there. I wouldn't mind losing that data, I'll make more, but I want to annoy them as long a possible with a personal feed. They've been hinting that they may remove the personal feeds for a while now, and I just plan on staying there until they kick me off.
But, this thing is so cool. It just works and accepts transactions without complaining at all. MySOL is way capable and can do things I hadn't even thought of.
So, Dave's place has Arduinos running things like the garage, pool and air conditioning that talk to a Raspberry Pi that saves it's status to a NAS. There's also a weather head that sends RF data to another Raspberry Pi that saves weather data cumulatively to the NAS as well.
The code for the new way of doing things is out on GitHub, if you want to grab something.
Unfortunately, this exercise has given me a bunch of ideas on things I can do to improve the operation. Now that I can essentially have unlimited Raspberry Pi devices doing things and coordinating activities of other devices, I can put the house configuration file on the NAS and read it from any Ethernet connected device, I can separate the web services from everything else and increase house Ethernet security. I can save data on each room using the new temperature sensor in various ways. I can ... I can go completely nuts,
If I haven't already.
I settled on QNAP, not because it won out in the specification, but because I caught a nice sale. TigerDirect had the QNAP TS-231 on sale for $30 bucks off, so I ordered it. I got two 3TB WD red drives from Amazon and set it up. It's up in my attic data center working its little heart out.
When I was looking at what could be run on it, MySQL was listed. That meant that I could put my house databases up there and get them off the Pi's I have running. It also meant that I could expand the database and actually log data locally for my house.
Wow, that would mean all my griping about cloud services would end.
First step, set up a MySQL database on the server; that took about 10 minutes. Then understand how to put data on it; that took a day. Then convert my code to use the new databases on the NAS; that took a stinking week. The very basic use of MySQL is different enough from the way to use SQLite3 that I had to learn a whole lot to get it working.
But, when I did, it really sped things up. I'm not clear on why exactly; I'm making data base transactions across an Ethernet link instead of locally, but it runs faster. Since the NAS is set up Raid 1, I have 3TB to play with and I expect the databases to use a hunk of that so it may slow down over time as the data bases grow.
My impetus for finally biting the bullet on one of these devices was storage failures on the Raspberry Pi. That's the most significant problem I've had with the Pi, the SD cards wear out, and I had worse luck using a USB stick. Well, I won't be wearing the cards out as fast, and I don't need the extra space of a USB stick anymore. However, the SD cards will still wear out because all the system logs and such are beating on them.
To this end, I'm going to put the operating system on the NAS as well, but I haven't gotten that far yet. The idea is to put everything I can on the NAS and only boot the Pi from the SD card. That should make it last a while. Sort of the best of both worlds; the separate processor and hardware of the Pi combined with the storage and reliability of a NAS. I'll inherit the possibility of things quitting if the Ethernet fails, but my house already relies on Ethernet anyway. I have the various Arduinos set up to work in a default mode, so that problem hasn't gotten any worse ... I think.
One of the side-effects (affects?? I can never remember which one to use) is that I cleaned up a bunch of code, removed things I really wasn't using any more and restructured a lot of my processes. So, most of the processes that wrote data to the various cloud services I looked into are gone now. The code is still out there on GitHub (the internet is forever), but it's not in the latest update. If you need it, just look at a previous commit.
Another thing that changed is since I'm not updating the various clouds, the live graphs I have scattered in earlier posts may not work any more. I replaced some of them with pictures to keep it somewhat coherent, but gave up after a few. Also, the architecture of the overall house control and monitoring is a bit different. I guess I'll delete the page on that and try to create a new picture and description soon.
I'm keeping my legacy feeds on Xively running for now since the majority of my data is there. I wouldn't mind losing that data, I'll make more, but I want to annoy them as long a possible with a personal feed. They've been hinting that they may remove the personal feeds for a while now, and I just plan on staying there until they kick me off.
But, this thing is so cool. It just works and accepts transactions without complaining at all. MySOL is way capable and can do things I hadn't even thought of.
So, Dave's place has Arduinos running things like the garage, pool and air conditioning that talk to a Raspberry Pi that saves it's status to a NAS. There's also a weather head that sends RF data to another Raspberry Pi that saves weather data cumulatively to the NAS as well.
The code for the new way of doing things is out on GitHub, if you want to grab something.
Unfortunately, this exercise has given me a bunch of ideas on things I can do to improve the operation. Now that I can essentially have unlimited Raspberry Pi devices doing things and coordinating activities of other devices, I can put the house configuration file on the NAS and read it from any Ethernet connected device, I can separate the web services from everything else and increase house Ethernet security. I can save data on each room using the new temperature sensor in various ways. I can ... I can go completely nuts,
If I haven't already.