Updating Eclipse Ditto

First, kill all docker containers. Then remove them, and remove their related images:

docker kill $(docker ps -q) && docker rm $(docker ps -a -q) && docker rmi $(docker images -q)

Then we get the latest version of the docker-compose from GitHub:

wget https://github.com/eclipse/ditto/archive/refs/heads/master.zip
unzip master.zip

Unzip it, browse to the right directory, and start it:

cd master/deployment/docker
docker-compose up -d

Don’t forget to adjust the password:

openssl passwd -quiet
 Password: <enter password>
 Verifying - Password: <enter password>

Append the printed hash in the nginx.httpasswd (in the same folder as docker-compose.yml) file placing the username who shall receive this password in front like this:

ditto:A6BgmB8IEtPTs

Configuring Juniper SRX (100) as a DHCP server

Create a vlan interface:

set interfaces vlan unit 21 family inet address 192.168.x.2/24

Assign the vlan interface to a vlan :

set vlans vlan_name vlan-id 21
set vlans vlan_name l3-interface vlan.21

Assign the vlan to a physical interface:

set interfaces fe-0/0/0 unit 0 family ethernet-switching vlan members vlan_name

Assign the dhcp-local-server service to the vlan interface:

set system services dhcp-local-server group IoT interface vlan.21

Create the DHCP pool:

set access address-assignment pool Verhaeg_IoT family inet network 192.168.x.0/24
set access address-assignment pool Verhaeg_IoT family inet range r1 low 192.168.x.200
set access address-assignment pool Verhaeg_IoT family inet range r1 high 192.168.x.250
set access address-assignment pool Verhaeg_IoT family inet dhcp-attributes name-server 192.168.x.1
set access address-assignment pool Verhaeg_IoT family inet dhcp-attributes name-server 8.8.4.4
set access address-assignment pool Verhaeg_IoT family inet dhcp-attributes name-server 8.8.8.8
set access address-assignment pool Verhaeg_IoT family inet dhcp-attributes router 192.168.x.1

Don’t forget your security zone to allow dhcp traffic:

set security-zone x interfaces vlan.20 host-inbound-traffic system-services dhcp

Validate clients have received an IP address from the DHCP server:

root@ROU-02> show dhcp server binding

IP address        Session Id  Hardware address   Expires     State      Interface
192.168.x.201    2           b6:54:ca:26:51:ae  70785       BOUND      vlan.21
192.168.x.202    3           c8:34:8e:5f:a4:2d  85932       BOUND      vlan.21

Determine washer program start-time based on predicted PV energy production.

When we decided we wanted to build a new house I wanted to invest in both passive and active technology to reduce our energy consumption as much as possible. My goal is to reduce our dependency from external energy sources, without installing batteries. This means I need to match our energy consumption with its (local) availability (or simply said: solar-powered production).

Impression of the design of our house.

Use case

The washer, dishwasher, and dryer are energy-consuming devices. We are used to start these devices at night: the energy was cheaper, there was no noise pollution in the living room, and it made sense at the end of the day.

At the same time, it doesn’t really matter when these devices finish their work. Typically you want them finished within the next 12 hours or so. Therefore, a smart system could nicely plan their consumption based on the next available solar-power production peak, which typically happens around lunch-time anyway. This would increase the energy we consume directly from the solar panels and reduce the energy we need to consume from the grid.

To make things a bit easier I decided to buy devices that support the B/S/H Home Connect system. As we are still building the house I don’t have a dishwasher and at the moment we don’t use a dryer, so I started with the washer.

Challenges

I need a lot of information to make this work. Luckily there are some public (free-of-charge) cloud-services available that helped me a bit here and there. All of the selected services have well documented APIs that I could implement with ease. My biggest challenge was to get the OAuth 2.0 Device Authorization flow up and running for Home Connect.

In our current apartment I don’t have solar-panels. Therefore I’m using data from another (live) solar production site to simulate the behaviour of the concept.

System architecture

To manage expectations: I’m not a professional software engineer. I’m not planning to productise this, and just want to be able to maintain everything myself. I might publish some of the related projects on Github, but don’t expect a lot of documentation on it.

System architecture overview

The image below shows a high-level system architecture. I’m using Ditto as a local digital twin for capturing the current state of all the entities in the system. Additionally, I’m using InfluxDB to store the historical states of the digital twins and Grafana to visualise the historical states. The green services are “sensors”: they retrieve data from their sources and update the digital twin. The blue services are “controllers”: they control the devices based on the status of the digital twin. The orange services are “processors”: they transform data or decide to start actions. The red services are “communicators”: they communicate with the user of the specific use-case.

How to use it?

Well, that’s the good news. Instead of pressing the “start” button on the washer, we press the “remote start enabled” (or “app”) button. The HomeConnect.Sensor captures this event and updates the digital twin. The HomeConnect.Planner is subscribed to this update and starts calculating the best possible timeslot in the next 16 hours to start the program. It considers the predicted solar-panel production, the history washer program consumption registered by the Shelly Plug, and the Epex spot pricing. Once it has calculated the ideal start time updates the “scheduledprogram” digital twin. The HomeConnect.Controller is subscribed on this update and sends the start command to the washer based on the time HomeConnect.Planner has defined. Slack (using Slack.Messenger) keeps the user up-to-date on what is happening, for example, when the program is planned, started, finished, or cancelled.

Installing and configuring Mosquitto on Debian

Eclipse Mosquitto is an open source (EPL/EDL licensed) message broker that implements the MQTT protocol versions 5.0, 3.1.1 and 3.1. Mosquitto is lightweight and is suitable for use on all devices from low power single board computers to full servers. The MQTT protocol provides a lightweight method of carrying out messaging using a publish/subscribe model. This makes it suitable for Internet of Things messaging such as with low power sensors or mobile devices such as phones, embedded computers or microcontrollers. The Mosquitto project also provides a C library for implementing MQTT clients, and the very popular mosquitto_pub and mosquitto_sub command line MQTT clients. Mosquitto is part of the Eclipse Foundation, is an iot.eclipse.org project and is sponsored by cedalo.com.

https://mosquitto.org/

Installation

wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
sudo apt-key add mosquitto-repo.gpg.key
cd /etc/apt/sources.list.d/
sudo wget http://repo.mosquitto.org/debian/mosquitto-buster.list
apt-get update
apt-get install mosquitto

Configuration

Create file /etc/mosquitto/conf.d/listener.conf:

bind_address x.x.x.x

Create file /etc/mosquitto/conf.d/authorisation.conf:

listener 1883 x.x.x.x

Create a user and password:

mosquitto_passwd -c <password file> <username>
kill -HUP <process id of mosquitto>

Result in /var/log/mosquitto/mosquitto.log:

1636747458: New connection from x.x.x.x:59620 on port 1883.
1636747458: New client connected from x.x.x.x:59620 as y (p2, c1, k60, u'client').
1636747843: New connection from x.x.x.x:5709 on port 1883.
1636747843: New client connected from x.x.x.x:5709 as z (p2, c0, k60, u'client').

Validation

Backup and restore ArangoDB data

You can easily back-up and restore the ArangoDB database from Windows. All you need is WinSCP, Putty, and some diskspace. I’m running the batch script below every time I login to my workstation automatically to backup the configuration database:

plink -batch -pw xxx -t root@1.1.1.1 "systemctl stop arangodb3"
plink -batch -pw xxx -t root@1.1.1.1 "cd /data/backup && zip -r arangodb.zip /data/arangodb/data"
plink -batch -pw xxx -t root@1.1.1.1 "systemctl start arangodb3"
pscp -pw xxx -v -r root@1.1.1.1:/data/backup/arangodb.zip  G:\IoT\ArangoDB\arangodb.zip
plink -batch -pw xxx -t root@1.1.1.1 "rm /data/backup/arangodb.zip"

You can restore the database by installing ArangoDB on another system and copying the data back into the correct directory. Make sure you stop the ArangoDB service before you restore the data.

When the ArangoDB version on the target system is newer then the ArangoDB version on the source system you need to upgrade the database first. Stop the ArangoDB service and start it in the console with the –database.auto-upgrade parameter:

systemctl stop arangodb
/usr/sbin/arangod --uid arangodb --gid arangodb --pid-file /var/run/arangodb3/arangod.pid --temp.path /var/tmp/arangodb3 --log.foreground-tty true --database.auto-upgrade

Install ArangoDB on Debian

Installation

curl -OL https://download.arangodb.com/arangodb37/DEBIAN/Release.key 
sudo apt-key add - < Release.key 
echo 'deb https://download.arangodb.com/arangodb38/DEBIAN/ /' | sudo tee /etc/apt/sources.list.d/arangodb.list
sudo apt-get update 
sudo apt-get install arangodb3

Change database directory

Change the database directory in: /etc/arangodb3/arangod.conf

[database] 
directory = /data/arangodb/data

Ensure the arangodb has access rights to this directory: /etc/systemd/system/arangodb3.service

chown -R arangodb:arangodb /data/arangodb
chmod 700 /data/arangodb

Change endpoint and secure connection

ArangoDB configuration file location: /etc/arangodb3/arangod.conf

endpoint = ssl://0.0.0.0:8530 
[ssl] 
keyfile = /var/lib/arangodb3/server.pem

Generate the server certificate:

# create private key in file "server.key" 
openssl genpkey -out /var/lib/arangodb3/server.key -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -aes-128-cbc 

# create certificate signing request (csr) in file "server.csr" 
openssl req -new -key /var/lib/arangodb3/server.key -out /var/lib/arangodb3/server.csr 

# copy away original private key to "server.key.org" 
cp /var/lib/arangodb3/server.key /var/lib/arangodb3/server.key.org 

# remove passphrase from the private key 
openssl rsa -in /var/lib/arangodb3/server.key.org -out /var/lib/arangodb3/server.key 

# sign the csr with the key, creates certificate PEM file "server.crt" 
openssl x509 -req -days 10000 -in /var/lib/arangodb3/server.csr -signkey /var/lib/arangodb3/server.key -out /var/lib/arangodb3/server.crt 

# combine certificate and key into single PEM file "server.pem" 
cat /var/lib/arangodb3/server.crt /var/lib/arangodb3/server.key > /var/lib/arangodb3/server.pem

Webinterface

https://x.x.x.x:8530/

Login with the specified root password.