Added MQTT support

vk2him

First I would download mqtt-spy. This utility allows you to connect to the mqtt server and validate that mosquitto is properly configured and up and running.

You can find it here https://github.com/eclipse/paho.mqtt-spy/wiki/Downloads

Once started go to Connections → Manage connections to create a new one for your environment. With a new connection created, click on “Control Panel”. You should now see a button for your server appear in the middle section. Click on it. If it turns green then it means that your mqtt server is up and running.

Next check your WD setting:

I am attaching a screenshot of my setup. Please noe:

  1. I am using a non-standard port (i.e. 30925 rather than the default1883)
  2. make sure that the "send customclientraw is NOT checked since the version of teh dashboard I shared with you was created using clientraw.

Let me know if this works for you.


Thanks for the help Renato - I think I see my problem. I’m assuming your ip address 192.168.1.50 you have setup in WD is your mqtt server?

I didn’t realise I needed a mqtt server - I have a Solar/Power meter that directly outputs mqtt and I’ve been able to connect and receive data using Node-Red running on my Synology directly connected to the solar meter without needing a mqtt server. I (wrongly) assumed WD was also able to also directly connect without a mqtt server.

So now I’ll investigate mqtt servers - it may be better to run the mqqt server on my Synology than on the Windows10 laptop that WD is currently running.
I’ll try again tonight
Regards
Ian

Might be worth to check out CloudMQTT.com, they have a free option.

Ian

If this is an option for you, I would probably invest in a Raspberry Pi 3B+ and deploy Mosquitto there. Later on, you may also want to install there MySQL/MariaDB etc.

Personally I like to “leave the NAS” alone.

My 2c

Renato

Thanks Renato - I have mariaDb and mysql working successfully on my Synology - it gathers data from my Blitzortung lightning detector:

http://www.theshackbythebeach.com:443/mybo/index.php?bo_page=map

I’ll investigate teh options tonight - thansk for the suggestions - cheers

Hi Renato - It’s taken me a few days to do testing etc

Partial success - long story short - for unknown reasons running Node-red on my Synology won’t allow me to connect to the Mqtt broker thats also running on the Synology. WD is able to publish to the broker fine and I can subscribe to the broker using a test mqtt.fx app. I can connect from Node-red on the synology to another mqtt device on a different IP and it works fine, I even created some new gauges to display the other mqtt data. I then tried installing Node-red on my windows laptop and I could connect to the Synology broker and display the WD gauges.

So something is broken on the Synology that doesn’t allow Node-red to connect - I tried localhost and 127.0.0.1 as well as the IP of the Synology without luck. I even tried a different Node-red docker container and it had the same problem.

I’ll keep checking … thanks anyway
Cheers

Brian/Ian

Here is an example of the “art of possible” I came across last night.

https://pysselilivet.blogspot.com/2018/09/weatherlink-weather-template_26.html

To implement this, all is needed is to replace the WeatherLink source (i.e. the http request object) with the message coming from WD. Also, some of the Function objects would have to be modified as well to account for available datapoints.

Next

  1. Currently working on a NR script that would persist the message in a MySQL database. Trying to decide if storing the message as a single JSON datatype or individual columns…
  2. Persist the data in a timeseries database (i.e. InfluxDB) for later rendering on a dashboard (i.e. Grafana)

Next we could look at replacing the FTP process all together by having a script run on the web server that would subscribe to the message and convert it into a text file to b e stored locally (i.e. pull vs push).

Will keep you posted on the progress

Renato

And there you have it.

Data is being persisted in an InfluxDB TimeSeries database and the data is rendered on the Grafana dashboard.

For this to be possible, I had to:

  1. Strip the “Client ID” from the message, since its presence would result in an incorrect JSON structure

  2. All the values in the customomclientraw file are entered in double quotes. This means that every datatype would be created as strings. I went thru the entire structure of the file and removed the double quotes on those fields that would generate float values (i.e temp, tempTL, tempTH, intemp etc)

  3. Brian, I noticed that the 2 values for presstrendval and temptrend should be of type float but they are actually being generated as strings (since values greater than 0 are created with a leading “+” sign). Is this by design?


re your Q 2, is this for the customclientraw, or?

also have you come across the MQTT skin
Belchertown

Brian

Yes, I was referring to the customclientraw.

Also, do you think you could make the Client ID optional? Or better yet, make it part of the perhaps part of the customclientraw file. Shirt term, I am going to hard code the value “ClientID”:“Southampton” at the front of teh file… As I mentioned, I had to remove it from the message since it is creating an invalid JSON array.

As far as the MQTT skin Belchertown, yes I have. Will look into that next. As for its ability to “auto refresh” the Belchertown page, it relies on the mqtt websocket service. To test that, I will need to modify my mosquitto/mqtt deployment since at present I am not exposing the service.

Thanks
Renato

what if you set the clientid as blank?

tried that, it populated a system generated value.

Renato

ah
that is by design, sorry
but you could set it as a space ?

Brian

Tried that, It is still a problem, since now the text preceding the JSON record is a space followed by an exclamation point. For the data to be recognized as a valid JSON array, I still need to strip everything to the left of the “{”

Renato

try
http://www.weather-display.com/downloadfiles/cronwdmqtt.zip
and with it set to nothing

I did, I have cleared the Client ID field on the form but I am still getting this…

w1nbPUexvuStwFFlP4TQ9aa!{…

Renato

then its not doable
as the component when it sees the clientid set to blank, will then set an auto generated one

Brian

I have added the ability to store the message into a MYSQL database.

To optimize the design, the flow inserts into 2 tables:

MQTT holds the entire message array
MQTT_Summary gets populated by the “UPSERT” block of the flow. AT any given time, this table will only contain 1 record, the most current one. This will improve the performance when trying to display “t0” data.

Lastly, on the database side, I have created an event that will purge records on the MQTT table as records become older than X days.

Let me know what would be the best way to share the scripts with those users who may be interested in replicating the approach.

Renato


Hi Brian,

I ended up setting up a mqtt broker via CloudMqtt and WD is successfully able to update it.

YEsterday though the CloudMqtt server I connect to had a harware failure and was offline for a few hours - the WD mqtt connection program obviously disconnected. When the server came back up, I had to manually re-connect via the Connect button in the WD mqtt window.

Is it possible for an auto connection retry to occur, say once every 10 mins or so?

Thanks
ian

I guess a auto re connect it a good idea for when you have set for updates fast where the program does not restart (and so re connect)