cloudy cloudy

Author Topic: Various Problems Loading Old Data  (Read 478 times)

0 Members and 1 Guest are viewing this topic.

Offline 3DAdamsWeather

  • Posts: 10
  • Tualatin, OR USA
  • OS/Browser:
  • Mac OS X 10.14.3
  • Safari 12.0.3
    • 3DAdams Weather
Various Problems Loading Old Data
« on: January 02, 2019, 09:49:16 PM »
I am running Weather Display version 10.37s85 dated 12/28/2018.  I have upgraded my computer to an HP Compaq 8200 Elite SFF PC running an i5-2400 CPU @3.10 GHz.  It has 8.0 GB of RAM.  Weather Display is on a  WD 1 Terabyte SSD.  I am also logging to MySQL Version 8.0.13 that is installed at this time on the same computer (localhost) as Weather Display.

The problems are:
1.  When I goto Action/Import Log Files from Other Software/Davis .wlk log file import setup and check the checkbox for "It is a .wlk Monthly (binary) file (recommended method), Weather Display will crash about halfway through the month.  This happens on any of the files that I have.  I have data back to September 1998, so there are a lot of files.  I have included one of the files that I am having problems importing.

2.  When I import data into MySQL Version 8.0.13, the Import ends before the file completely imports.  Using the Log/Data File Viewer/Correct Data window, I see that the last two records match the last two records in the database.  In the file that I have attached (Month1201) you can see that the last two records are for 1/31/2001 at 21:20 and then 1/31/2001 at 23:00.  If I look at the log file (12001g), the file contains the full month of data.  I have tried to convert the log file to a graph file and that does not fix the file problem.  I have also verified that there are no duplicate records in the log file. 

3.  During normal logging of data to the MySQL database, I get a message "Data too long for column 'current_weather_desc' at row 1" message.  Some of my weather conditions are longer than the 20 characters the definition of the column.
   
As always, thanks for you help.

Don Adams
3DAdams Weather
Don Adams
dsadams@3dadams.org

Offline 3DAdamsWeather

  • Posts: 10
  • Tualatin, OR USA
  • OS/Browser:
  • Mac OS X 10.14.3
  • Safari 12.0.3
    • 3DAdams Weather
Re: Various Problems Loading Old Data
« Reply #1 on: January 03, 2019, 05:16:11 PM »
I knew that I would forget something.  I am using a Davis Vantage Pro 2 plus with fan. Not sure that is important, but at least you know.

Thanks again,
Don Adams
Don Adams
dsadams@3dadams.org

Offline NorCal Dan

  • Dan - KJ6RGX
  • Posts: 10,533
  • Davis Vue/Win10
  • Ocean Park, WA.
  • OS/Browser:
  • Mac OS X 10.10.5
  • Chrome 71.0.3578.98
    • Traveling RV Weather
Re: Various Problems Loading Old Data
« Reply #2 on: January 03, 2019, 07:11:19 PM »
Here is your logfile run through the checker...

http://www.stella-maris.org.uk/checklogs.php?site=http%3A%2F%2Ftravelingrvwx.com%2Fpublic%2F12001lg.txt&submit=Submit


You can ignore the record length notices, but you should try to fix the timestamp does not increase errors.  I did not look at the entire list so there may be other errors.

Some history...in the past, there were instances where WD saved "extra" entries in the log file.  By saving one entry per minute, there should be exactly 1440 entries per each day of the month.  Anything more or less could cause issues.

Offline Weather Display

  • Posts: 87,036
  • Davis VP2
  • New Zealand
  • OS/Browser:
  • Win NT 10.0
  • Chrome 71.0.3578.98
    • Weather Display
Re: Various Problems Loading Old Data
« Reply #3 on: January 03, 2019, 07:16:45 PM »
if the logfile has too many lines (more than  44640 lines) then it will not all get converted to data file (which is fixed to 44640 data points (i.e 1 minute interval for a month)

Offline NorCal Dan

  • Dan - KJ6RGX
  • Posts: 10,533
  • Davis Vue/Win10
  • Ocean Park, WA.
  • OS/Browser:
  • Mac OS X 10.10.5
  • Chrome 71.0.3578.98
    • Traveling RV Weather
Re: Various Problems Loading Old Data
« Reply #4 on: January 04, 2019, 07:48:36 PM »
See if you can import this (your 12001lg.txt)) corrected file (as a test).

Offline 3DAdamsWeather

  • Posts: 10
  • Tualatin, OR USA
  • OS/Browser:
  • Mac OS X 10.14.3
  • Safari 12.0.3
    • 3DAdams Weather
Re: Various Problems Loading Old Data
« Reply #5 on: January 04, 2019, 09:58:25 PM »
I tried to convert the logfile to a graph file, which worked on the second try.  Then I tried to import the datafile into the MySQL database and it stopped at the record with a datetime stamp of 2001/01/31 23:57:00.  The logfile has data to 2001/01/31 23:59:00, so that last two records did not import.  The progress bar still has blue in it, which I think indicates that it is not complete. 

I hope that you did not have to edit the file with a program editor.  That is tedious and time-consuming.  I had another problem yesterday unrelated to weather-display that caused me to have to power off the computer instead of close the applications and shutdown.  Weather-display did not like that.  When I started it again, the graphs had no data.  I decided that I would reload the data again (this is about the fifth time).  In that process, I noticed that I was getting a lot of bad data from Weather Underground.  Lots of negative numbers, etc.  I stopped loading really old data from them and am working on another solution to recover some of my old data, but some of it is not in my database.  I still need to be able to pull data out of the old WeatherLink files though and that still crashes before it reaches the end of the file.  Any ideas on that problem?

Does anyone have a program that can check a logfile and find if there are problems?  I was not sure if the link that you sent yesterday can do that, or if that was open for me to use?   

I really appreciate the effort that you have put into this.  I would like to get this thing running perfectly.

Thanks,

Don
Don Adams
dsadams@3dadams.org

Offline Weather Display

  • Posts: 87,036
  • Davis VP2
  • New Zealand
  • OS/Browser:
  • Win NT 10.0
  • Chrome 71.0.3578.98
    • Weather Display
Re: Various Problems Loading Old Data
« Reply #6 on: January 04, 2019, 10:12:22 PM »
yes there are online logfile checkers
try doing a search of the forum

Offline 3DAdamsWeather

  • Posts: 10
  • Tualatin, OR USA
  • OS/Browser:
  • Mac OS X 10.14.3
  • Safari 12.0.3
    • 3DAdams Weather
Re: Various Problems Loading Old Data
« Reply #7 on: January 05, 2019, 12:53:47 AM »
Dan,
I found two more errors in the file that you sent me, but I still cannot load it into the MySQL database.  Argh!

Thanks,

Don
Don Adams
dsadams@3dadams.org

Offline NorCal Dan

  • Dan - KJ6RGX
  • Posts: 10,533
  • Davis Vue/Win10
  • Ocean Park, WA.
  • OS/Browser:
  • Mac OS X 10.10.5
  • Chrome 71.0.3578.98
    • Traveling RV Weather
Re: Various Problems Loading Old Data
« Reply #8 on: January 05, 2019, 01:16:15 AM »
Dan,
I found two more errors in the file that you sent me, but I still cannot load it into the MySQL database.  Argh!

Thanks,

Don

Yea, it was way short on entries...I noticed a lot of errors occurred at the same time...03:02...double entry, all the other issues appeared to be failed imports where the old data was appended.  I believe this was an old issue that Brian fixed a long time ago, but your log files are from before the fix.

Unfortunately there is no easy way to actually repair the log files that I know of...

Offline 3DAdamsWeather

  • Posts: 10
  • Tualatin, OR USA
  • OS/Browser:
  • Mac OS X 10.14.3
  • Safari 12.0.3
    • 3DAdams Weather
Re: Various Problems Loading Old Data
« Reply #9 on: January 05, 2019, 04:25:41 AM »
I have imported data for a long time from VWS into a FileMaker DB.  Let me play with that some.  I think I can get all of the past solar data this way also.  Thanks for your time and help on the log file.  I still would like to see if I can import the really old data from WeatherLink though.  Maybe Brian can look to see why the program crashes when I do an import from a .wlk file.

Thanks,

Don
Don Adams
dsadams@3dadams.org

 

cumulus