cloudy cloudy

Author Topic: HWS-template (WD34 / weather34) April release  (Read 6118 times)

0 Members and 1 Guest are viewing this topic.

Offline GW400

  • Posts: 81
  • Klepp, Norway
  • OS/Browser:
  • Win NT 10.0
  • Chrome 73.0.3683.103
Re: HWS-template (WD34 / weather34) April release
« Reply #30 on: April 25, 2019, 05:15:13 AM »
Who to use to run the cron-job
As the cron-job runs on your website, the website itself can not start the cron-job.
Your website provider probably has a cron-job facility and there are other free or paid-for cron-job-providers
But WD is already running 24/7 and has al cron-job services built in.  No need to use another way.

How does it work:
WD will start the cron-job with a http call to your website => that is the only thing WD has to do
The "cron-job" will be run on your webserver as it is a normal PHP script
The "cron-job" will load the  server-side "load_files" php-script which re-loads all external data files if they are to old, but this time with 80% of the allowed age

Some documentation
http://wd34.weather-template.com/ => Documentation => http://wd34.weather-template.com/documentation/WDapi-stationcron-2019-01.pdf

PLEASE ALWAYS SUPPLY THE URL TO YOUR WEBSITE
Sometimes a small problem or a provider-setting inhibits loading the external data.
Only with the link  the poeple who try to help you, can check if there is another problem

Wim
I'm using WeeWx and can't use WD to run the cron jobs.
Is it then recommended to manually define cron jobs running the same .py scripts as when using WD?
My website is at weather.meisal.com and I have created three jobs (os i Ubuntu 18.04) running the mentioned scripts. ...stationcron is running fine as scheduled (every 5 min), but the ..empty and ...addtoyear jobs does not work.
G400
Regards G400

Online wvdkuil

  • Wim van der Kuil
  • Posts: 1,204
  • Leuven Belgium CET
  • OS/Browser:
  • Mac OS X 10.13.6
  • Chrome 73.0.3683.103
    • My PWS at Leuven Belgium Europe
Re: HWS-template (WD34 / weather34) April release
« Reply #31 on: April 25, 2019, 10:54:30 AM »
I'm using WeeWx and can't use WD to run the cron jobs.
Is it then recommended to manually define cron jobs running the same .py scripts as when using WD?
My website is at weather.meisal.com and I have created three jobs (os i Ubuntu 18.04) running the mentioned scripts. ...stationcron is running fine as scheduled (every 5 min), but the ..empty and ...addtoyear jobs does not work.
G400
Running the station-cron as test in the browser returns:
Quote
w34_cron_stationcron.php (28) version =>w34_cron_stationcron.php|00|2018-12-29|
. . . shorted all load data messages . . .
w34_load_files.php (129) purplair sensor not defined
w34_load_files.php (236) File not old enough (910/2880 seconds)https://api.waqi.info/feed/geo:59.014728;6.902158/?token=_API_SETTING_

No loading of DB needed
. . . shortened , removed trace . . . 
success files loaded +to day file

Running the add-to-year-cron http://weather.meisal.com/_test.php?test=w34_cron_addtoyear.php as test in the browser returns:
Quote
No database usage allowed according to the settings

Please check your settings, you do not allow a database to be used.
When you are uploading to WeatherUnderground this is the correct setting.

When not uploading to WU you should let the stationcron upload to an sql table.
Then the two other cron-jobs can convert the sql-data to month and year files

Please replace current version of station-cron with attached one so that I know which version you use.

Wim

Offline GW400

  • Posts: 81
  • Klepp, Norway
  • OS/Browser:
  • Win NT 10.0
  • Chrome 73.0.3683.103
Re: HWS-template (WD34 / weather34) April release
« Reply #32 on: April 25, 2019, 09:23:09 PM »
Running the station-cron as test in the browser returns:
Running the add-to-year-cron http://weather.meisal.com/_test.php?test=w34_cron_addtoyear.php as test in the browser returns:
Please check your settings, you do not allow a database to be used.
When you are uploading to WeatherUnderground this is the correct setting.

When not uploading to WU you should let the stationcron upload to an sql table.
Then the two other cron-jobs can convert the sql-data to month and year files

Please replace current version of station-cron with attached one so that I know which version you use.

Wim

Inserted w34_cron_stationcron.php you attached to last message and also enabled the option "Database uses standard PHP connection..." in settings.

Manually executed all cron jobs:

http://weather.meisal.com/_test.php?test=w34_cron_stationcron.php
w34_cron_stationcron.php (28) version =>w34_cron_stationcron.php|00|2019-04-25|
w34_load_files.php(23) loaded  =>w34_load_files.php|00|2019-04-10|
....
w34_settings.php (218) $units =scandinavia $windunit=m/s $tempunit=C $rainunit=mm $pressureunit=hPa $distanceunit=km $wu_unit=m
w34_load_files.php (32) include  =>w34_common.php
w34_common.php (26) version =>w34_common.php|00|2019-01-03|
......
w34_cron_stationcron.php (80) data correctly loaded to database
w34_cron_stationcron.php (89) include_once =>w34_livedata.php
w34_cron_stationcron.php (99) Data saved to chartsmysql/result.csv
success files loaded +to db +to day file

http://weather.meisal.com/_test.php?test=w34_cron_addtoyear.php
object(mysqli_result)#2 (5) {
  ["current_field"]=>
  int(0)
  ["field_count"]=>
  int(12)
  ["lengths"]=>
  NULL
  ["num_rows"]=>
  int(1)
  ["type"]=>
  int(0)
}
string(170) "MAX_timestamp,MAX_outsideTemp,MIN_outsideTemp,MAX_dewpoint,MIN_dewpoint,MAX_raintoday,MAX_windgust,MAX_windSpeed,MAX_radiation,MAX_barometer,MIN_barometer,SUM_lightning,
"
string(74) "2019-04-25 22:31:33,9.3,0.0,5.80,-1.80,0.00,64.4,51.5,0.00,1015.10,0.00,0,"
success
w34_cron_addtoyear.php (28) version =>w34_cron_addtoyear.php|00|2018-12-29|
w34_cron_addtoyear.php (31) include_once =>w34_settings.php
w34_settings.php (25) version =>w34_settings.php|00|2019-04-10|
w34_settings.php (28) include =>w34_settings1.php
w34_settings1.php (4) version =>w34_settings1.php|00|2019-04-25|
w34_settings.php (218) $units =scandinavia $windunit=m/s $tempunit=C $rainunit=mm $pressureunit=hPa $distanceunit=km $wu_unit=m

http://weather.meisal.com/_test.php?test=w34_cron_empty.php
success 1

bool(true)
-2

All three script jobs seems to work when run manually.

Added to the following jobs within the context of the user running WD34:
su <user>
crontab -e

Then added the following:
0,5,10,15,20,25,30,35,40,45,50,55 * * * * php -q /path/w34_cron_stationcron.php
58 23 * * * php -q /path/w34_cron_addtoyear.php
0 0 * * *   php -q /path/w34_cron_empty.php

Note: "path" is not the real path

I will let it run until tomorrow and check again, but to me it seems unlikely it will not work when run as cron jobs when executed manually through browser works
Regards G400

Offline bitsostring

  • Posts: 845
  • Oregon Scientific WMR200
  • Edinburgh, U.K.
  • OS/Browser:
  • Win NT 10.0
  • Chrome 64.0.3282.140
    • Home Weather Station
Re: HWS-template (WD34 / weather34) April release
« Reply #33 on: April 25, 2019, 09:46:04 PM »
Quick question: where do I find the meteoalarm.eu "Europe: code for your area, example BE004" mentioned in easyweathersetup?

I have only entered "UK" so far. . .
It's meant to be fun. . .

Offline hcorrin

  • Posts: 1,245
  • Ballaugh Isle of Man GB
  • OS/Browser:
  • Win NT 10.0
  • Mozilla compatible
    • Ballaugh Weather
Re: HWS-template (WD34 / weather34) April release
« Reply #34 on: April 25, 2019, 10:04:54 PM »
Try UK004

Offline GW400

  • Posts: 81
  • Klepp, Norway
  • OS/Browser:
  • Win NT 10.0
  • Chrome 73.0.3683.103
Re: HWS-template (WD34 / weather34) April release
« Reply #35 on: April 26, 2019, 06:17:02 AM »
Inserted w34_cron_stationcron.php you attached to last message and also enabled the option "Database uses standard PHP connection..." in settings.

Manually executed all cron jobs:

http://weather.meisal.com/_test.php?test=w34_cron_stationcron.php
w34_cron_stationcron.php (28) version =>w34_cron_stationcron.php|00|2019-04-25|
w34_load_files.php(23) loaded  =>w34_load_files.php|00|2019-04-10|
... the rest in previous port...
I will let it run until tomorrow and check again, but to me it seems unlikely it will not work when run as cron jobs when executed manually through browser works

w34_cron_addtoyear.php is failing when run as cron job, but running in browser did work:
Error message when run as cron job: Couldn not write headers to chartsmysql/2019_04.csv
Regards G400

Online wvdkuil

  • Wim van der Kuil
  • Posts: 1,204
  • Leuven Belgium CET
  • OS/Browser:
  • Mac OS X 10.13.6
  • Chrome 73.0.3683.103
    • My PWS at Leuven Belgium Europe
Re: HWS-template (WD34 / weather34) April release
« Reply #36 on: April 26, 2019, 07:08:21 AM »
w34_cron_addtoyear.php is failing when run as cron job, but running in browser did work:
Error message when run as cron job: Couldn not write headers to chartsmysql/2019_04.csv

The error message tells us that the cron-job when run form within the server is not allowed to write to the file and or folder.

So set the permissions (using your FTP-program or cpanel) to either 755 or 777 for the folder and the folder-contents.


Online wvdkuil

  • Wim van der Kuil
  • Posts: 1,204
  • Leuven Belgium CET
  • OS/Browser:
  • Mac OS X 10.13.6
  • Chrome 73.0.3683.103
    • My PWS at Leuven Belgium Europe
Re: HWS-template (WD34 / weather34) April release
« Reply #37 on: April 26, 2019, 07:20:05 AM »
Quick question: where do I find the meteoalarm.eu "Europe: code for your area, example BE004" mentioned in easyweathersetup?

I have only entered "UK" so far. . .

@all
Visit http://www.meteoalarm.eu/en_US/0/0/EU-Europe.html
On the map on the left, click on your country, UK for this example http://www.meteoalarm.eu/en_US/0/0/UK-United%20Kingdom.html
On the country-map ,click on the area for your location

Now the addresline / URL changes to something like
Code: [Select]
http://www.meteoalarm.eu/en_US/0/0/UK004-Lothian%20&%20Borders.htmlThe meteoalarm code we need is UK004, it is always something like AA1111, 2 letters for your country,  3 digits for the area.

Wim

Offline richardcr

  • Posts: 4
  • OS/Browser:
  • Win NT 10.0
  • Chrome 74.0.3729.108
    • Collapit Weather
Re: HWS-template (WD34 / weather34) April release
« Reply #38 on: April 26, 2019, 09:09:51 AM »
Super job, Wim
I just have one niggle. I'm not getting any monthly or annual rainfall totals either at the top of the page or in the almanac. Curiously, the rainfall graphs show historical data which tends to bear out that I've correctly entered my Weather Underground station ID. My weather data is downloaded from the WU API as my PWS uploads direct to WU without a computer. My site is at www.collapit.org.uk/pwsWD/index.php
Have I missed an essential step?
Richard

Online wvdkuil

  • Wim van der Kuil
  • Posts: 1,204
  • Leuven Belgium CET
  • OS/Browser:
  • Mac OS X 10.13.6
  • Chrome 73.0.3683.103
    • My PWS at Leuven Belgium Europe
Re: HWS-template (WD34 / weather34) April release
« Reply #39 on: April 26, 2019, 09:27:21 AM »
Super job, Wim
I just have one niggle. I'm not getting any monthly or annual rainfall totals either at the top of the page or in the almanac. Curiously, the rainfall graphs show historical data which tends to bear out that I've correctly entered my Weather Underground station ID. My weather data is downloaded from the WU API as my PWS uploads direct to WU without a computer. My site is at www.collapit.org.uk/pwsWD/index.php
Have I missed an essential step?
Richard
Thank you for testing with the WU-version.
Not all  rain values are available with the new API / json files.
Yes, there are values in the .csv but we then have to calculate those values ourselves.

More data-sources have some values missing.
I am working on the next release and those  "not always available" fields will become optional and will not be shown.
I will also add calculations for the year / month values to the cron-job scripts in the near future,

Wim

Offline richardcr

  • Posts: 4
  • OS/Browser:
  • Win NT 10.0
  • Chrome 74.0.3729.108
    • Collapit Weather
Re: HWS-template (WD34 / weather34) April release
« Reply #40 on: April 26, 2019, 10:25:52 AM »
Thanks for the prompt reply, Wim. I've also just noticed that the Almanac figure for yesterday's rainfall total is in fact today's but I guess that's part of the same issue. No need to reply to this.

Offline bitsostring

  • Posts: 845
  • Oregon Scientific WMR200
  • Edinburgh, U.K.
  • OS/Browser:
  • Win NT 10.0
  • Chrome 64.0.3282.140
    • Home Weather Station
Re: HWS-template (WD34 / weather34) April release
« Reply #41 on: April 26, 2019, 10:45:50 AM »
Try UK004

Thanks, Harold.

The meteoalarm code we need is UK004

Thanks, Wim. I have crawled all over their website and never noticed the code change in URL  :oops:

But just as a test I entered UK001 (South-West), where a yellow warning for high winds has been in force for over 24 hours, and I still get "No Current Weather Alerts": see http://swanston.maui.co.uk/pwsnew/

Does it know I'm telling lies?  :)
It's meant to be fun. . .

Online wvdkuil

  • Wim van der Kuil
  • Posts: 1,204
  • Leuven Belgium CET
  • OS/Browser:
  • Mac OS X 10.13.6
  • Chrome 73.0.3683.103
    • My PWS at Leuven Belgium Europe
Re: HWS-template (WD34 / weather34) April release
« Reply #42 on: April 26, 2019, 12:17:20 PM »
. . . shortened . . .
But just as a test I entered UK001 (South-West), where a yellow warning for high winds has been in force for over 24 hours, and I still get "No Current Weather Alerts": see http://swanston.maui.co.uk/pwsnew/
. . .

These are the error-messages from the script
Code: [Select]
<!-- $wrn_lang=en  -->
<!-- module wrnWarningEU.php = version: 0.00 2019-01-22 -->
<!-- warnings and other info goes here  -->
<!-- Severe weatherdata (./jsondata/warningUK001) not found in cache -->
There is no cached file for UK001 => that is correct as first it should load the warnings and cache them
Code: [Select]
<!-- Severe weatherdata loaded from http://www.meteoalarm.eu/en_UK/0/0/UK001.html  -->
<!-- ERROR Severe weatherdata empty,(http://www.meteoalarm.eu/en_UK/0/0/UK001.html)  could not be loaded  -->
This should not happen ! Wwhen I test http://www.meteoalarm.eu/en_UK/0/0/UK001.html  in the browse and in some test-sites correct data is loaded.
Code: [Select]
<!-- upped cachetime from 900 to 1800 as no valid data was retrieved -->
When there is a temporary problem there is a validity check of 30 minutes for the cached data
Code: [Select]
<!-- Severe weatherdata (./jsondata/warningUK001) not found in cache -->
<!-- ERROR no warnings  retrieved, script ends -->
<!-- ERROR no warnings  retrieved, script ends -->

So I think the script is not allowed to access the outside world?
The CURL statements are at line 711 and further in wrnWarningEU.php

Success, Wim



Offline bitsostring

  • Posts: 845
  • Oregon Scientific WMR200
  • Edinburgh, U.K.
  • OS/Browser:
  • Win NT 10.0
  • Chrome 64.0.3282.140
    • Home Weather Station
Re: HWS-template (WD34 / weather34) April release
« Reply #43 on: April 26, 2019, 01:07:53 PM »
So I think the script is not allowed to access the outside world?
The CURL statements are at line 711 and further in wrnWarningEU.php

Ah! If you remember, I had to add a curlopt_proxy line to w34_load_files.php to get those scripts to access the outside world (Reply #9 above). I'll check wrnWarningEU.php now: any more CURLs I have to worry about?  :)

EDIT: That's it fixed now! Test over, now I know it's working I'll go back to UK004.

Many thanks, Wim.
« Last Edit: April 26, 2019, 01:36:22 PM by bitsostring »
It's meant to be fun. . .

Online wvdkuil

  • Wim van der Kuil
  • Posts: 1,204
  • Leuven Belgium CET
  • OS/Browser:
  • Mac OS X 10.13.6
  • Chrome 73.0.3683.103
    • My PWS at Leuven Belgium Europe
Re: HWS-template (WD34 / weather34) April release
« Reply #44 on: April 26, 2019, 01:57:30 PM »
. . .  shortened . . .
 any more CURLs I have to worry about?  :)
. . .
As long as you do not move to Canada or the USA, there are no other CURLs.

All weather34 CURL-requests are in  w34_load_files.php
But the three warning-scripts are external / other authors scripts and better maintained by not changing to much.

Wim

 

cumulus