Added MQTT support

Brian

Tried that, It is still a problem, since now the text preceding the JSON record is a space followed by an exclamation point. For the data to be recognized as a valid JSON array, I still need to strip everything to the left of the “{”

Renato

try
http://www.weather-display.com/downloadfiles/cronwdmqtt.zip
and with it set to nothing

I did, I have cleared the Client ID field on the form but I am still getting this…

w1nbPUexvuStwFFlP4TQ9aa!{…

Renato

then its not doable
as the component when it sees the clientid set to blank, will then set an auto generated one

Brian

I have added the ability to store the message into a MYSQL database.

To optimize the design, the flow inserts into 2 tables:

MQTT holds the entire message array
MQTT_Summary gets populated by the “UPSERT” block of the flow. AT any given time, this table will only contain 1 record, the most current one. This will improve the performance when trying to display “t0” data.

Lastly, on the database side, I have created an event that will purge records on the MQTT table as records become older than X days.

Let me know what would be the best way to share the scripts with those users who may be interested in replicating the approach.

Renato


Hi Brian,

I ended up setting up a mqtt broker via CloudMqtt and WD is successfully able to update it.

YEsterday though the CloudMqtt server I connect to had a harware failure and was offline for a few hours - the WD mqtt connection program obviously disconnected. When the server came back up, I had to manually re-connect via the Connect button in the WD mqtt window.

Is it possible for an auto connection retry to occur, say once every 10 mins or so?

Thanks
ian

I guess a auto re connect it a good idea for when you have set for updates fast where the program does not restart (and so re connect)

Thanks Brian - remind me what the frequency threshold is above which the the process doesn’t terminate?

I set in the component to stay alive/auto reconnect
seems to work
(i.e if I disconnect my internet it will re connect when I reconnect that)
if the update rate is less than 60 seconds it will stay running

try this new update
http://www.weather-display.com/downloadfiles/cronwdmqtt.zip
vers 1.8

Thanks - new version works fine with my local testing - I’ll monitor with the connection to CloudMqtt - Regards Ian

Has anyone else besides myself experimented with Weather Display’s new MQTT support and Home Assistant? I posted a separate message on my configuration on how I pulled in the temp and humidity via WD->MQTT->Home Assistant.

Not sure if this would help other MQTT related projects, but having each “custom tag” sent as it’s own topic (using the tag’s name) would be extremely helpful in making this far easier to pull in to Home Assistant. That is due to the fact that you wouldn’t have to parse the line of text, but rather just point to the different topics by themselves. Much more straight-forward and would work for even the most challenging text - like current conditions, etc.


Mark

I have to respectfully disagree. Having a single message gives you the greatest flexibility from a programming standpoint while allowing for “3rd-party” components to create bolt-ons and pool whatever information is needed.

Having said that, as far as Home Assistant is concerned, here is how I was able to pull the temp and humidity:

sensor:
  - platform: mqtt
    name: "MQTT Temp"
    state_topic: "Southampton/WD/newjson"
    unit_of_measurement: "

Here is how I am pulling the data, which is basically parsing the clientraw line of custom tags (temp and humidy) for testing. Since parsing is really not perfect when it comes to tags like current conditions (which is multiple words, etc), this logic won’t work properly.

I’m using EMQ-X as the MQTT broker with Home Assistant. Not familiar with the json setup, and couldn’t get that to work. Very possible could be me not calling the json functionality properly.

My thought of doing each custom tag as a different topic would be as simple as just having different “state_topics” for each sensor or whatever you are calling. No parsing needed.

Here is how I have mine working now:

sensor:

  • platform: mqtt
    name: “Outdoor Temperature”
    state_topic: “homeassistant”
    unit_of_measurement: ’

Not sure what is supposed to do the JSON conversion from straight text for the MQTT topic. I am using MQTT test clients and my Weather Display data is arriving in straight text format only, not JSON. I can’t find info on how to receive this topic via JSON - like you have yours setup. That is why I had to do all the crazy text parsing to get mine to work.

I’m using EMQ as the broker with Weather Display and Home Assistant.

Thanks!

  • Mark

I have been trying to figure out how to convert the text string that Weather Display sends via MQTT to JSON. That is still an extra step that wouldn’t be needed if all the tags were separate topics. Here is an example of how easy it would be for me to pull in WD tags to Home Assistant:

sensor:

  • platform: mqtt
    name: “Outdoor Temperature”
    state_topic: “WeatherDisplay/Temp”
    unit_of_measurement: ’

After re-reading this entire thread, understand that Node-Red is doing the text->JSON converting (after stripping the clientid) to form valid JSON which you are then using in HA. It is impressive the manipulations, converting, etc, you can do with Node-Red. I have never used it, but it looks very interesting.

I’m basically doing WD->MQTT-EMQ-X broker->Home Assistant (not using Node-Red in the middle). That is why for my setup, having each tag as a topic would be much easier when it comes to parsing.

If it isn’t a huge effort and can be an optional box to send each as their own topic, that would be very helpful in my case. I’m just pulling “temp & hum” at the moment, but want to pull lots more… :smiley:

Neat things could be done with WD’s data, like turning lights on if WD is receiving precip - or announce “It is raining” on my Google Mini speakers…

I should be able to add that as an option
when I find the spare time
(this week I do not have much as I am planting 1000’s of native trees I have grown 8O )

Mark

Obviously there are multiple ways to look at the problem. Each one has its own advantages and disadvantages.

I personally am of the opinion that distributing the workload minimizes the overhead on the machine where WD is running. It also gives you the flexibility of leveraging other HW and OS.

That’s why I believe that by dumping the entire structure and then have node-red become the 'traffic cop" of where a given workload should run is a more “scalable” solution. As of today, I have offloaded the mysql and influxdb population to a different server (in my case running Centos) while having the 2 repositories running on a 4 node Kubernetes cluster. In the case of mysql, I have 2 repositories (tables), one which collects ALL the data (data older than X days gets purged by an event) and another repository (i.e. another table) which at any given time only has 1 record. The most current one. This results in great performance advantages while rendering the current conditions on a dashboard.

As for the K8 cluster, today it manages the following workloads:

  1. Grafana (the dashboard) pulling data from mysql and/or influxdb
  2. Node-red
  3. InfluxDB (the timeseries database)
  4. Mysql
  5. phpMyAdmin
  6. Mosquitto

I am not suggesting that your approach is invalid, but like everything in consulting, the answer is always “it depends”. And it truly does, since it hinges on what one is ultimately trying to do, now and in the future.

Perhaps we could find a common ground.

What I am thinking would be the following:

  1. have the user create a new custom file with a collection of tabs that only he/she would require.
  2. WD would then read the file to determine which key/value combination to publish.

The file could look like this:

message_name1:{“key1”:%tag1%, “key2”:“%tag2%”…“keyn”:%tagn%}
message_name2:{“key1”:%tag1%}
message_name3:{“key2”:%tag2%|

So, in my case, I would only have the first row, and WD would generate a single message made up of the defined payload. This would also allow me to convert customclientraw.txt back to its original configuration since I had to remove the quotes from several fields. They were incorrectly being handled as strings rather than float.

In your case, you use use one row (i.e. message_name2:{“key1”:%tag1%}) for each individual key/value combination you are interested in subscribing to.

So if you are interested in temp, hum and pressure, then you would have 3 rows. Each one with its own message.

Could this work?

Assuming it does, ultimately, it is Brian’s call.

Thanks for the consideration

Renato

Mark

I forgot to mention that another use case would be to leverage an ESP8266 with a display. Each ESP8266 would then subscribe to the message(s) and display it on the screen.

Ultimately you will have built a very inexpensive weather console. With what you would have to pay for a new Davis console, you could easily built 5/10 of these for the same price!!!

Renato

1) have the user create a new custom file with a collection of tabs that only he/she would require. 2) WD would then read the file to determine which key/value combination to publish.

The file could look like this:

message_name1:{“key1”:%tag1%, “key2”:“%tag2%”…“keyn”:%tagn%}
message_name2:{“key1”:%tag1%}
message_name3:{“key2”:%tag2%|

I think this could work OK
so example the input file could be
temp/hum:{“temp”:%temp%,“hum”:%hum%}

which WD converts the custom tags and sends e.g
“temp”:14.5,“hum”:55 as the topic temp/hum

yes?