cloudy cloudy

Author Topic: Problem with Saratoga WU-forecast.php  (Read 586 times)

0 Members and 1 Guest are viewing this topic.

Offline vk2him

  • Posts: 659
  • Gerringong, NSW, Australia
  • OS/Browser:
  • Win 7/Srvr 2008R2
  • Chrome 67.0.3396.62
    • The Shack by the Beach
Problem with Saratoga WU-forecast.php
« on: June 05, 2018, 11:02:52 PM »
Hi Ken,

I've noticed over the last few months the WU-forecast.php program won't always retrieve the latest forecast from Weather Underground and has failed with various php errors.

I haven't had a chance to look into it until yesterday when I found I was using the older version 3.03 so I updated to v3.04

With v3.04 I'm still getting an occasional error but this time the errors are different - In my dashboard I sometimes now get the following at the top and the forecast is blank :

Code: [Select]
Warning: Invalid argument supplied for foreach() in /Applications/XAMPP/xamppfiles/htdocs/wx/WU-forecast.php on line 550

Warning: Invalid argument supplied for foreach() in /Applications/XAMPP/xamppfiles/htdocs/wx/WU-forecast.php on line 564

Warning: Division by zero in /Applications/XAMPP/xamppfiles/htdocs/wx/WU-forecast.php on line 632

Note that I've inserted one extra line into my WU-forecast.php at 174 to hard-code the $SITE['fcsturlWU'] compared to yours so I can run WU-forecast.php standalone for testing, so the above line numbers will be one higher than your standard code.

If I run WU-forecast.php standalone when those errors appear, the forecast is blank, including the town name which is a period ".", if I then click to force update it generally works.

If it helps, $SITE['fcsturlWU']  is set to

Code: [Select]
$SITE['fcsturlWU']  = 'http://www.wunderground.com/cgi-bin/findweather/getForecast?query=Gerringong%2C+Australia'; // Gerringong
Thanks
Ian
« Last Edit: June 06, 2018, 02:39:35 PM by saratogaWX »

Offline saratogaWX

  • Global Moderator
  • Posts: 5,785
  • Ken True
  • Saratoga, CA, USA 37:16:28N, 122:01:23W - Elev: 374ft.
  • OS/Browser:
  • Win NT 10.0
  • Firefox 60.0
    • Saratoga Weather
Re: Problem with Saratoga WU-forecast.php
« Reply #1 on: June 06, 2018, 02:41:52 PM »
Hi Ian,

This is likely caused by a transient connection issue from your webserver to wunderground.com resulting in a 'null' forecast being seen.

I ran your wxforecast.php page with ?force=1 and in the view-source of the page it showed
Quote
<!-- WU-forecast.php (ML) Version 3.04 - 27-Feb-2018 on PHP 5.6.32 -->
<!-- temps in C -->
<!-- WU_API Raw URL='http://www.wunderground.com/cgi-bin/findweather/getForecast?query=Gerringong%2C+Australia' -->
<!-- WU API New URL='http://api.wunderground.com/api/42cac6f726e5ac53/forecast10day/geolookup/lang:EN/q/Australia/Gerringong.json' -->
<!-- loading from https://api.wunderground.com/api/42cac6f726e5ac53/forecast10day/geolookup/lang:EN/q/Australia/Gerringong.json. -->
<!-- curl fetching 'https://api.wunderground.com/api/42cac6f726e5ac53/forecast10day/geolookup/lang:EN/q/Australia/Gerringong.json' -->
<!-- HTTP stats:  RC=200 dest=104.116.123.15 port=443 (from sce=192.168.1.79)
      Times: dns=0.156 conn=0.166 pxfer=0.363 get=0.192 total=0.555 secs -->
<!-- RC=200 OK, bytes=24275 -->
<!-- saved cache to ./WU-forecast-json-0-en.txt (24275 bytes) -->
<!-- using charsetInput='UTF-8' charsetOutput='ISO-8859-1' doIconv='1' doRTL='' -->
<!-- processing JSON entries for forecast -->
<!-- rawJSON size is 23817 bytes -->
<!-- WU_prepareJSON: Success. Valid UTF-8. -->
<!-- json_decode returns  - No errors -->
<!-- replace icon 'partlycloudy.gif' with 'sct10.jpg' pop=10 -->
<!-- replace icon 'nt_clear.gif' with 'nskc.jpg' pop=10 -->
<!-- replaced forecastcond[2]='Chance rain' -->
<!-- replace icon 'chancerain.gif' with 'hi_shwrs50.jpg' pop=50 -->
<!-- replaced forecastcond[3]='Chance rain' -->
<!-- replace icon 'nt_chancerain.gif' with 'hi_nshwrs80.jpg' pop=80 -->
<!-- replaced forecastcond[4]='Chance rain' -->
<!-- replace icon 'chancerain.gif' with 'hi_shwrs50.jpg' pop=50 -->
<!-- replaced forecastcond[5]='Chance rain' -->
<!-- replace icon 'nt_chancerain.gif' with 'hi_nshwrs40.jpg' pop=40 -->
<!-- replaced forecastcond[6]='Chance rain' -->
<!-- replace icon 'chancerain.gif' with 'hi_shwrs40.jpg' pop=40 -->
<!-- replaced forecastcond[7]='Chance rain' -->
<!-- replace icon 'nt_chancerain.gif' with 'hi_nshwrs30.jpg' pop=30 -->
<!-- replace icon 'clear.gif' with 'skc.jpg' pop=20 -->
<!-- replace icon 'nt_clear.gif' with 'nskc.jpg' pop=20 -->
<!-- replace icon 'partlycloudy.gif' with 'sct20.jpg' pop=20 -->
<!-- replace icon 'nt_partlycloudy.gif' with 'nsct20.jpg' pop=20 -->
<!-- replace icon 'clear.gif' with 'skc.jpg' pop=20 -->
<!-- replace icon 'nt_clear.gif' with 'nskc.jpg' pop=20 -->
which is (now) a fairly normal response from the access to wunderground.
Ken True
Saratoga Weather
CWOP: CW1792
WeatherUnderground: KCASARAT1
Free weather website PHP scripts and WD website AJAX templates

Offline vk2him

  • Posts: 659
  • Gerringong, NSW, Australia
  • OS/Browser:
  • Win 7/Srvr 2008R2
  • Chrome 67.0.3396.79
    • The Shack by the Beach
Re: Problem with Saratoga WU-forecast.php
« Reply #2 on: June 07, 2018, 03:55:12 AM »
Thanks for the tips Ken,

When I get the error this is in the page source:

Code: [Select]
<br />
<b>Warning</b>:  Invalid argument supplied for foreach() in <b>/Applications/XAMPP/xamppfiles/htdocs/wx/wxforecast.php</b> on line <b>713</b><br />
<br />
<b>Warning</b>:  Invalid argument supplied for foreach() in <b>/Applications/XAMPP/xamppfiles/htdocs/wx/wxforecast.php</b> on line <b>727</b><br />
<br />
<b>Notice</b>:  Undefined variable: WUforecasticons in <b>/Applications/XAMPP/xamppfiles/htdocs/wx/wxforecast.php</b> on line <b>795</b><br />
<br />
<b>Warning</b>:  Division by zero in <b>/Applications/XAMPP/xamppfiles/htdocs/wx/wxforecast.php</b> on line <b>795</b><br />
<br />
<b>Notice</b>:  Undefined variable: WUforecasticon in <b>/Applications/XAMPP/xamppfiles/htdocs/wx/wxforecast.php</b> on line <b>796</b><br />
<br />
<b>Notice</b>:  Undefined variable: WUforecasticons in <b>/Applications/XAMPP/xamppfiles/htdocs/wx/wxforecast.php</b> on line <b>801</b><br />
<br />
<b>Notice</b>:  Undefined variable: WUforecasticons in <b>/Applications/XAMPP/xamppfiles/htdocs/wx/wxforecast.php</b> on line <b>801</b><br />
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>WeatherUnderground 0-day Forecast - </title>
    <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" />
</head>
<body style="font-family:Verdana, Arial, Helvetica, sans-serif; font-size:12px; background-color:#FFFFFF">

<!-- WU-forecast.php (ML) Version 3.04 - 27-Feb-2018 on PHP 5.6.32 -->
<!-- temps in C -->
<!-- WU_API Raw URL='http://www.wunderground.com/cgi-bin/findweather/getForecast?query=Gerringong%2C+Australia' -->
<!-- WU API New URL='http://api.wunderground.com/api/42cac6f726e5ac53/forecast10day/geolookup/lang:EN/q/Australia/Gerringong.json' -->
<!-- loading from ./WU-forecast-json-0-en.txt (0 bytes) -->
<!-- using charsetInput='UTF-8' charsetOutput='ISO-8859-1' doIconv='1' doRTL='' -->
<!-- processing JSON entries for forecast -->
<!-- rawJSON size is 0 bytes -->
<!-- WU_prepareJSON: Success. Valid UTF-8. -->
<!-- json_decode returns  - No errors -->
<br />
<b>Notice</b>:  Undefined variable: WUforecasttext in <b>/Applications/XAMPP/xamppfiles/htdocs/wx/wxforecast.php</b> on line <b>873</b><br />
<br />
<b>Notice</b>:  Undefined variable: PHP_SELF in <b>/Applications/XAMPP/xamppfiles/htdocs/wx/wxforecast.php</b> on line <b>875</b><br />
<br/><br/>Forecast blank? <a href="?force=1">Force Update</a><br/><br/>  <table width="640px" style="border: none;" class="WUforecast">
          <tr align="center" style="background-color: #FFFFFF;">
      <td><b>WeatherUnderground 0-day Forecast: </b><span style="color: green;">
   </span>
      </td>
    </tr>
    <tr>
      <td align="center">
    <table width="100%" border="0" cellpadding="0" cellspacing="0"> 
<!-- lengths title=0 cond=0 temps=0 -->
        </table><!-- end icon table -->
     </td>
   </tr><!-- end print icons -->
    </table>
  <p>&nbsp;</p>

<table style="border: 0" width="640px" class="WUforecast">
<br />
<b>Notice</b>:  Undefined variable: WUforecasttitles in <b>/Applications/XAMPP/xamppfiles/htdocs/wx/wxforecast.php</b> on line <b>1019</b><br />
   </table>
<p>&nbsp;</p>
<p>Forecast from <a href="http://www.wunderground.com/cgi-bin/findweather/getForecast?query=Gerringong%2C+Australia">WeatherUnderground</a>
for .
</p>
</body>
</html>

It would be "nice" if I could program it to retry a few times automatically on failure, particularly as my dashboard will show php error messages when it fails.

Thanks
Ian

Offline saratogaWX

  • Global Moderator
  • Posts: 5,785
  • Ken True
  • Saratoga, CA, USA 37:16:28N, 122:01:23W - Elev: 374ft.
  • OS/Browser:
  • Win NT 10.0
  • Firefox 60.0
    • Saratoga Weather
Re: Problem with Saratoga WU-forecast.php
« Reply #3 on: June 08, 2018, 01:47:47 AM »
One thing to try is to change the network timeout in the WU-forecast.php script
Code: [Select]
   $numberOfSeconds=4;   
to
Code: [Select]
   $numberOfSeconds=8;   
to give your webserver up to 8 seconds to successfully contact weatherunderground.com API.

A second thing to try is to avoid writing missing data to the cache (and reuse the cached version instead).

Change
Code: [Select]
      $fp = fopen($cacheName, "w");
  if (!$fp) {
    $Status .= "<!-- unable to open $cacheName for writing. -->\n";
  } else {
        $write = fputs($fp, $html);
        fclose($fp); 
$Status .= "<!-- saved cache to $cacheName (". strlen($html) . " bytes) -->\n";
  }
to
Code: [Select]
    if(strlen($html) > 20000) {
      $fp = fopen($cacheName, "w");
      if (!$fp) {
        $Status .= "<!-- unable to open $cacheName for writing. -->\n";
      } else {
        $write = fputs($fp, $html);
        fclose($fp); 
        $Status .= "<!-- saved cache to $cacheName (". strlen($html) . " bytes) -->\n";
      }
    } elseif (file_exists($cacheName)) {
      $html = implode('', file($cacheName));
      $Status .= "<!-- problemm loading from api .. reusing existing cache file -->\n";
      $Status .= "<!-- loading from $cacheName (" . strlen($html) . " bytes) -->\n";
    }
and that will only save the cache if >20K bytes are available, and if a cache file exists, load it instead.

Together, they may avoid the issue with slow/timeout/bad API returns by 'recycling' your last good forecast from them.
Ken True
Saratoga Weather
CWOP: CW1792
WeatherUnderground: KCASARAT1
Free weather website PHP scripts and WD website AJAX templates

Offline vk2him

  • Posts: 659
  • Gerringong, NSW, Australia
  • OS/Browser:
  • Win 7/Srvr 2008R2
  • Chrome 67.0.3396.79
    • The Shack by the Beach
Re: Problem with Saratoga WU-forecast.php
« Reply #4 on: June 08, 2018, 02:43:53 AM »
Excellent suggestions Ken - thanks for that - I'll make the changes and will monitor. Cheers

Offline vk2him

  • Posts: 659
  • Gerringong, NSW, Australia
  • OS/Browser:
  • Mac OS X 10.13.5
  • Chrome 67.0.3396.87
    • The Shack by the Beach
Re: Problem with Saratoga WU-forecast.php
« Reply #5 on: June 14, 2018, 02:24:29 AM »
Hi Ken,

For future reference in case others have this issue:- I had to slightly modify one of your suggestions to make it work properly. It seems that the string being > 20k wasn't quite large enough as I got the same errors again. I looked at the json size when it failed and founded to needed to be >22k.

So the modified code is now

Code: [Select]
if(strlen($html) > 22000) {
      $fp = fopen($cacheName, "w");
      if (!$fp) {
        $Status .= "<!-- unable to open $cacheName for writing. -->\n";
      } else {
        $write = fputs($fp, $html);
        fclose($fp); 
        $Status .= "<!-- saved cache to $cacheName (". strlen($html) . " bytes) -->\n";
      }
    } elseif (file_exists($cacheName)) {
      $html = implode('', file($cacheName));
      $Status .= "<!-- problemm loading from api .. reusing existing cache file -->\n";
      $Status .= "<!-- loading from $cacheName (" . strlen($html) . " bytes) -->\n";
    }

Has been quite stable now for days where previously I was getting the error a few times per day.
Cheers!!!!

Offline saratogaWX

  • Global Moderator
  • Posts: 5,785
  • Ken True
  • Saratoga, CA, USA 37:16:28N, 122:01:23W - Elev: 374ft.
  • OS/Browser:
  • Win NT 10.0
  • Firefox 60.0
    • Saratoga Weather
Re: Problem with Saratoga WU-forecast.php
« Reply #6 on: June 14, 2018, 03:48:32 AM »
Rather than size, I decided to look for an actual forecast in the contents of the JSON using
Code: [Select]
    if(preg_match('|fcttext|Uis',$html)) {
      $fp = fopen($cacheName, "w");
      if (!$fp) {
        $Status .= "<!-- unable to open $cacheName for writing. -->\n";
      } else {
        $write = fputs($fp, $html);
        fclose($fp); 
        $Status .= "<!-- saved cache to $cacheName (". strlen($html) . " bytes) -->\n";
      }
    } elseif (file_exists($cacheName)) {
      $html = implode('', file($cacheName));
      $Status .= "<!-- WU API return has no forecast .. reusing existing cache file -->\n";
      $Status .= "<!-- loading from $cacheName (" . strlen($html) . " bytes) -->\n";
    }
.. that's the one I'll distribute shortly.  I found a few forecast sites with <20000 characters in the return, but were still good forecasts.  This test allows all the JSON with forecast content and only saves the cache then..  If it's time to reload the cache, the API result is checked for forecast content and saved if present, otherwise the existing cache file is reused with the slightly older forecast.
Ken True
Saratoga Weather
CWOP: CW1792
WeatherUnderground: KCASARAT1
Free weather website PHP scripts and WD website AJAX templates

Offline saratogaWX

  • Global Moderator
  • Posts: 5,785
  • Ken True
  • Saratoga, CA, USA 37:16:28N, 122:01:23W - Elev: 374ft.
  • OS/Browser:
  • Win NT 10.0
  • Firefox 60.0
    • Saratoga Weather
Re: Problem with Saratoga WU-forecast.php
« Reply #7 on: June 14, 2018, 04:12:22 AM »
All distributed.

Standalone from https://saratoga-weather.org/scripts-WUforecast.php

Template users from https://saratoga-weather.org/wxtemplates/updates.php with a query of 13-Jun-2018, Base-World, Plugin-*


Ken True
Saratoga Weather
CWOP: CW1792
WeatherUnderground: KCASARAT1
Free weather website PHP scripts and WD website AJAX templates

Offline vk2him

  • Posts: 659
  • Gerringong, NSW, Australia
  • OS/Browser:
  • Mac OS X 10.13.5
  • Chrome 67.0.3396.87
    • The Shack by the Beach
Re: Problem with Saratoga WU-forecast.php
« Reply #8 on: June 14, 2018, 07:10:39 AM »
Thanks again Ken - I've now implemented your distribution,
Cheers
ian