Most of my xPL components support the hbeat schema. So I can identify them by sending a hbeat.request message. My perl xpl-sender command can do this:

bash$ xpl-sender --schema hbeat.request --wait 10

By default, the sender just sends the specified message and exits. But, with the --wait parameter, it prepares to receive replies, sends the message and waits the specified number of seconds for any replies. The output from the above command is a summary of each received hbeat.app reply:

xpl-stat/hbeat.app: bnz-apcups.apc -> * 5/53556/192.168.32.1
xpl-stat/hbeat.app: bnz-blue.slave -> * 5/51540/192.168.32.1
xpl-stat/hbeat.app: bnz-ccost.slave -> * 5/58435/192.168.32.1
xpl-stat/hbeat.app: bnz-dmx.slave -> * 5/44072/192.168.32.1
xpl-stat/hbeat.app: bnz-gpower.slave -> * 5/59772/192.168.32.1
xpl-stat/hbeat.app: bnz-heyu.slave -> * 5/56009/192.168.32.1
xpl-stat/hbeat.app: bnz-lcdproc.slave -> * 5/41722/192.168.32.1
xpl-stat/hbeat.app: bnz-linux.slave -> * 5/46917/192.168.32.1
xpl-stat/hbeat.app: bnz-lirc.slave -> * 5/33992/192.168.32.1
xpl-stat/hbeat.app: bnz-mpd.slave -> * 5/32798/192.168.32.1
xpl-stat/hbeat.app: bnz-ownet.slave -> * 5/46631/192.168.32.1
xpl-stat/hbeat.app: bnz-rfxcomrx.slave -> * 5/40448/192.168.32.1
xpl-stat/hbeat.app: bnz-rfxcomtx.slave -> * 5/48635/192.168.32.1
xpl-stat/hbeat.app: bnz-sendsms.slave -> * 5/47592/192.168.32.1
xpl-stat/hbeat.app: bnz-smart.slave -> * 5/43472/192.168.32.1
xpl-stat/hbeat.app: bnz-udin.slave1 -> * 5/50553/192.168.32.1
xpl-stat/hbeat.app: bnz-udin.slave2 -> * 5/52187/192.168.32.1
xpl-stat/hbeat.app: bnz-wol.slave -> * 5/33981/192.168.32.1
xpl-stat/hbeat.app: bnz-zenah.slave -> * 5/39324/192.168.32.1
xpl-stat/hbeat.app: bnz-mythtv.vz -> * 5/34473/192.168.32.6
xpl-stat/hbeat.app: bnz-smart.vz -> * 5/34298/192.168.32.6
xpl-stat/hbeat.app: bnz-xosd.vz -> * 5/34487/192.168.32.6
xpl-stat/hbeat.app: bnz-xvkbd.vz -> * 5/49051/192.168.32.6

Each summary is of the form:

message_type/schema: source-identifier -> target interval/port/ip

and each source-identifier is of the form:

vendorid-deviceid.instanceid

My vendor id (developer id) is bnz, the device id is typically the type of the device being managed, and the instance id is typically the hostname of the machine running the client.

There are two machines in the list vz (which is my main mythtv box) and slave (which is the main home automation server). (The observant may notice that there is also apc but that is actually running on slave monitoring an APC UPS via USB but the instance id is changed as a convinience in order to distinguish between slave and the UPS as they both report power/battery information.) The clients on the mythtv box are:

• xpl-mythtv which reports the percentage utilisation of the video inputs/tuners,

• xpl-smart which reports disk temperatures so I can turn on fans to extract warm air from the server room,

• xpl-xosd which responds to osd.basic schema messages showing on-screen display text using xosd, and

• xpl-xvkbd which sends fake key presses to the active window (probably a security risk even though the keys always go to mythfrontend).

The clients on slave are:

• xpl-apcups which reports status of my APC UPS and sends events if when the UPS switches between mains and battery and vice versa,

• xpl-blue which monitors for the presence of various bluetooth devices so that my house "knows" when different people are home,

• xpl-ccost which monitors my mains power usage (or at least it did until it got confusing when I started exporting power) and the solar power generated by the PV panels on my roof,

• xpl-dmx which sends commands to a Milford Instruments DMX Transmitter to control a Pulsar ChromaZone 12 Controller which in turn drives a number of ChromaRange lamps,

• xpl-gpower which reports my power usage information into the Google PowerMeter API,

• xpl-heyu which uses the heyu X10 software control my mains appliances and lights,

• xpl-jabber which enables interaction via XMPP instant messages (such as Google Talk),

• xpl-lcdproc which responds to osd.basic schema messages showing text on a picoLCD-4x20-slideshow device using the [lcdproc][] protocol,

• xpl-linux which reports Linux battery status and events, system temperature, etc.,

• xpl-lirc which reports LIRC IR remote button presses (so that I can switch the kettle on or send wake-on-lan packets using a basic TV remote control),

• xpl-mpd which controls a Music Player Daemon to play music in several zones around the house,

• xpl-ownet which is an interface to an OWFS Daemon that reads from temperature/humidity sensors and writes to relay controllers on the one-wire network in my house,

• xpl-rfxcomrx which is an interface to an RFXCOM RF Receiver that reports RF messages received from various sensors, switches, etc.,

• xpl-rfxcomtx which sends RF messages via an RFXCOM RF Transmitter to control X10, HomeEasy, etc. devices,

• xpl-sendsms which sends SMS messages via any service supported by the SMS::Send Perl API (I use the service from Connection Software),

• xpl-smart which reports disk temperatures,

• two instances of xpl-udin to control two UDIN USB Relay Controllers to momentarily pulse the open/close inputs of several blind and curtains,

• xpl-wol which sends wake-on-lan packets (using the etherwake command) to wake up devices that are shutdown in order to save power (such as my mythtv box), and

• zenah which is the brains of my house - triggering actions based on timers and incoming xPL messages.

I am also running several other clients that run in stealth mode - that is only sending messages and not listening for or responding to hbeat.request messages. These "inputs" include:

• a web frontend,

• an experimental daemon supporting the lightswitch API so that I can use existing Android (or iPhone) applications to control my house, and

• some security-related inputs (that I wont be writing much about).

I'm currently trying to refactor the code to separate out the device-related code from the xPL code and to reduce the coupling between the web interface and the zenah "brains". This will probably involve a couple re-write of the zenah component and the web interface.

The primary issue was the need to perform filtering for every message on every client due to the use of broadcasts for messaging. One way to avoid this is to use a pub/sub messaging system instead. A number of people that I respect have been using the MQTT Protocol so I thought I'd take a better look at it.

I figured a good way to understand it would be to implement it. I've implemented several protocols from scratch without any documentation so implementing MQTT from a well-written specification was simple process. The basic implementation took less than a week and resulted in Net::MQTT and AnyEvent::MQTT on CPAN.

The protocol is lightweight and flexible. This is great. It works well in contexts where you are both publisher and subscriber because you can design the simplest, minimal messaging format between agents.

However, building communities that can write components that can interoperate will require some more structure. For example, using xPL, I have a client that monitors messages using the "sensor.basic" schema and creates/updates round robin database files for each sensor reading. This works for all xPL clients that support the "sensor.basic" schema regardless of developer/platform/etc. Another good example of this kind of protocol layering is XMPP and the XMPP Extension Protocols.

Essentially what is required to make this work is an ontology - a specification for topic usage and semantics. I have no idea how to produce such a suitable ontology but I plan to think about it some more. I'll probably draw ideas from various existing sources such as xPL Message Schema, xAP Message Schema, Pachube feed definitions, etc.

If anyone has any ideas (is it a good idea, how to proceed, etc?) then I'd love to hear about them.

The xPL Protocol has three types of messages: commands (xpl-cmnd), triggers (xpl-trig) and status (xpl-stat). They have the same simple format consisting of the message type, common "header" fields, the schema type and schema-specific "body" fields. For example:

xpl-cmnd
{
hop=1
source=vendor-device.host
target=*
}
x10.basic
{
command=on
device=c3
}

xPL doesn't use a server as its "message bus" is simply UDP port 3865 on the local network broadcast address. In order to avoid multiple devices having to listen on the same port on the same broadcast address, each host runs a hub that distributes messages to all local xPL clients. I think this is a poor solution to this problem. It introduces an unnecessary single point of failure (albeit one with very simple-to-test behaviour). To avoid this single point of failure, my xPL clients also support a hubless mode where they simply set the SO_REUSEADDR socket option on the listen socket so that all clients can coexist on the same port on the same broadcast address. However, if you need to interoperate with clients from other developers then you'll need to use the standard hub model.

Using UDP broadcasts to avoid any (unnecessary) single points of failure is a good idea. However, I have more than 20 xPL clients on my network and all clients process all messages they receive, discarding most of them and responding to relatively few. This means that all of the clients have to wake up and consume CPU resource for every single message. (This is one of the issues that is prompting me to re-evaluate my choice of protocol.)

I thought about avoiding this problem by combining several clients in to one more complex client but this seems like a bad idea as it will inevitably lead to more complexity and thus more complicated failure cases. (My xPL clients are structured in such a way that this is very easy to do and some users with lower-powered servers do this.)

For me, the xPL Message Schema are what make the protocol work so well. Essentially, the schema define the fields to expect in the body of an xPL message. This means I can write a client for a UDIN USB Relay Controller or a Phaedrus VIOM that supports the control.basic schema and someone else can write a client that uses the same schema to control relays and it will happily interoperate with my clients. (In fact, I realised this advantage when I migrated from using a VIOM to several more compact UDIN controllers. I only needed to change the names of the relays for open/close on my blinds/curtains after each set of wires were migrated to the new hardware.)

The schema, as described in the protocol documentation, support a type of inheritance. A message schema that inherits from another must include all of the mandatory fields of the other, in the same order, so clients can support the basic function even if they don't understand the additional sub-schema fields. However, since optional vendor-specific fields are also allowed without inheritance (giving a simple duck typing approach), the inheritance mechanism doesn't really give much benefit.

Take hbeat messages, there are two types, basic:

xpl-stat
{
hop=1
source=vendor-device.host
target=*
}
hbeat.basic
{
interval=5
}

and, inherited from basic, app:

xpl-stat
{
hop=1
source=vendor-device.host
target=*
}
hbeat.app
{
interval=5
port=43438
remote-ip=198.51.100.2
}

Why do the messages need the .basic or .app? So that a client knows to expect the port and remote-ip fields? Why does a client need to know to expect them? A client can just parse the message and if the fields are present - i.e. no } following the interval=5 line - then it can treat the message like an .app message and like a .basic message if not. It is simply redundant.

The only obviously valid use of the .suffix is .confirm suffix but this would probably be better implemented as a fourth message type such as xpl-ack or xpl-conf.

The wildcard target=* line is also redundant since a lack of explicit target=vendor-device.host could just as easily be used to distinguish the wildcard case.

These superfluous elements mean that the "Lite on the wire, by design" subtitle on the xPL Protocol specification isn't really as "lite" as it could be (and IMHO should be).

Another confusing aspect of the xPL Protocol is the notion of a device. In the protocol spec, for instance in the Device Groups section, it is clear that the target field is addressing an individual device - a controller for a single curtain/drape. This is fine in theory, but in practice, individual devices are general addressed using fields in the body. For instance, control.basic, remote.basic, sensor.basic, x10.basic, and x10.security use device, homeasy.basic uses the combination of address and unit, zwave.basic uses node, etc. This is because xPL clients are typically controllers/gateways for several devices. For instance, I have a 1-wire sensor/relay network with dozens of devices attached to it but one xPL client opens the 1-wire USB interface and manages all of these devices based on the device field in control.basic and sensor.basic messages.

The protocol would be simpler to use if addressing of devices was more consistent. In fact, devices (rather than clients) should be first-class citizens on the "network". That is, they should be discoverable (like clients are today with hbeat requests) and probably should be aliasable (so that they can be given memorable names).

I like the xPL Protocol; it has been my protocol of choice for more than 5 years and most of these issues are not a big deal. The broadcast message bus is simple (if you ignore hubs) and reliable but the inefficiency of waking all clients on every message is starting to become an issue for me. So, in the next few posts, I'll write about some of the alternatives I am considering.

Although all our lights have conventional switches (well, momentary switches in conventional locations), we use a lot of X10 RF Remotes and X10 Motion Sensors to trigger lights and other devices. Initially, we used a TM13U RF Transceiver Module that receives the X10 RF signals and converts them in to X10 powerline signals. If we stopped here and let the powerline signals trigger the X10 devices directly this solution would probably be okay. However, we wanted computer control so we have another level of indirection. The powerline signals are then received by the X10 CM12U Transceiver and sent to the house computer. The computer then figures out how to react to this signal and, if the action involves an X10 device, sends another signal back to the CM12U where it is sent on the powerline.

The problem with this setup is that the X10 powerline is slow, so converting the (fast) RF signal to a slow X10 signal to get it to the computer is not very smart.

Additionally, RF devices repeat their signals which in turn get repeated on the powerline. While the repeats are going on, the computer, responding to the first received signal, is instructing the X10 Transceiver to transmit a command on the powerline. This results in frequent collisions leading to noticeable delays (and X10 powerline signalling isn't fast to begin with).

So, we purchased a W800 X10 RF Receiver and gave up on the TM13U. Using the W800 serial port receiver, the X10 RF could be read directly by the house computer. This avoids the delay and the collisions of the X10 powerline signals for the trigger and any triggered X10 commands.

The W800 was a solid piece of kit and worked very reliably. However, in the end, I replaced it with an RFXCOM RF Receiver as that was able to receive additional protocols. I expect I'll rave about RFXCOM products in future posts.

We had three significant problems. The first was noisy devices preventing the X10 powerline signals reaching some parts of our house.

Initially, we attempted to resolve the noise problems by using X10 FM10 Plug in Filter Modules or X10 FD10 DIN-mounted Filter Modules. We still use them on a number of appliances - things like TVs, fridges, laptop power supplies, UPSs, etc. They definitely help but there were still sockets that didn't get any signal. (We know because we used an X10 Signal Tester to check the sockets.)

Eventually, I came across Jeff Volp's excellent explanation of X10 problems and ordered an XTB signal booster. This device takes the X10 signal from the X10 CM12U Transceiver computer interface and boosts it by a factor of 10. This fixed our X10 powerline signal problems completely. We still occasionally plug in a new, exceptionally noisy device that needs a filter but this is very unusual.

The second problem was with X10 RF signals. When the batteries in an X10 Motion Sensor run out, they often end up transmitting excessively. This effectively jams any RF signals (including my car key buttons). Unlike most of the other RF devices we use, these devices don't report battery status so I've solved this problem by using an excellent charger, good quality rechargeable batteries and reminders to change the batteries every 6 months.

The third problem, with latency, is a bit more complicated so I'll cover that in another post.

Although I am a morning person, I still need a little help waking up in the morning. My lovely children help a bit but being able to have the lights gradually brighten to full brightness when it is time to get up helps a lot, particularly in the winter. It also means that the children wake up at the same time every day, which in turn means they tend to go to sleep pretty reliably in the evening. In the event that they wake up too early, they know it's too early as the lights aren't on yet so generally don't wake us.

We also have X10 Motion Sensors so can automatically trigger lights in rooms with no natural light. Our utility room, where we change nappies, is a good example as this behaviour is particularly useful when you have your hands full carrying a reluctant toddler.

The children rather like the low-level stick-on X10 RF Remotes that enable them to turn on their lights (and I like the fact that they don't have to climb on the furniture to reach the "normal" switches). (They also like having the remotes trigger wolf howls, T-Rex roars and Michael Rosen from the ceiling speakers.)

One disadvantage of the dimmable lights is that most dimmers have a minimum rating. The LD11 X10 DIN-mounted Lamp Modules that we use require a minimum load of 60W. This is a problem because it is becoming increasingly difficult to buy bulbs that meet this requirement. At the moment, we typically use Energy Saver GLS Lamps which look great and use just enough power to keep the dimmers happy.

I'd be very interersted to know how others with X10 solve this problem as I'd love a solution that used less power but which was still dimmable and produced a pleasant bright light.

The most important device we've automated is the kettle. We "prime" the kettle after making a cup of tea by filling it, switching it on but turning it off at the mains (with an X10 Appliance module). We can then turn it on using either an X10 RF Remote, a web interface, timers or via a number of other mechanisms. This is a real time saver. I can "C-c k" from emacs and only go to the kitchen when the kettle has boiled rather than going to switch it on and wait while it boils.

Tracy says that this is just laziness. I probably used to empty/load the dishwasher while waiting for the kettle, so perhaps she is right. She prefers the fact that you can hit a remote button by the front door when you come home on a cold day, or from upstairs before you come down in the morning, and have the kettle waiting when you get to the kitchen.

The house computer also monitors the mains current and attempts to turn the kettle off for you if it sees a drop in current usage equivalent to the kettle usage. At the moment, it's working off the whole house current usage data so it doesn't always work but it still often manages to recognize the change and switch off the kettle at the main while you are filling the kettle.

When we got our boiler replaced, we asked the heating engineers to put in an extra room thermostat under the stairs in the future node 0. (Of course, there was nothing under there at that time and they thought we were mad.) However, they did install it and so we had two thermostats, either of which could "call for heat".

The plan for control is to turn down the thermostat that is in a sensible position, so that it'll only call for heat if things go horribly wrong. Then we're going to replace the other thermostat with an AD10 X10 din rail switch. When you take the front off, the room stat has three terminals Neutral, Live, and Switched-Live so it's a simple matter (cough I only blew the fuse on the heating once!) of connecting these to the Live In, Neutral and Live Out on the AD10.

Our brief tests show that this works. Now we just need to persuade our electrician that it's a good idea.

It's still a little odd whenever I catch sight of the previously dingy room out of the corner of my eye. As you can see there's still a bit of work to be done before the en suite is ready, so I have a little bit of time to get over it before that room is functional.

(It looked even better when I remembered to wipe off the biro I'd used to mark out the features. ;-) We used an X10 RF signal to trigger a short chase effect and play an appropriate sound. That was even better so I created a small movie of the test.

We waited impatiently until it finally got dark and we could install it overlooking the driveway.

I'm going to be most disappointed if we don't scare a few kids with it on Monday.