Weather Display > WD Bug Reports
WDMysql - add past history problem
WeerinMontfort:
I have installed WD 10.37R23 with WDMysql 10.6 on a dedicated PC running Windows7-64 and am uploading past history from my Meteohub (ver 4.9g) data logger to a MYSQL database (ver 5.1.61-0) hosted by a web hosting provider.
This works perfect but with one strange glitch in this process:
The first month (for example 2008-2) that is extracted from the Meteohub is uploaded by WDMysql perfectly, but the next month (2008-3) i try to upload always starts uploading at day 28, finishes at the end of the month and then starts at day 1 to complete at day 31.
Looking at the 2008-3 data in the MYSQL database only data from day 28 to the end of the month are uploaded, the previous data is missing.
After I remove the data of 2008-3 from database, reset the WD logger dates in the WD control panel to start again at 2008-3-1 00:00 and when it finished extracting, upload the 2008-3 data again, all 2008-3 data is uploaded perfectly.
But then the next month 2008-4 becomes the month with exactly the same problem.
This workaround shows there is no problem with the Meteohub data, but it takes a lot of time, especially since I want to upload data starting from 2008.
Can anyone help me?
WeerinMontfort:
Disappointing, no response from anyone.
In the meanwhile i have followed my workaround and migrated all data to the MYSQL database. It did cost me more than two weeks.
Looking at the data in the MYSQL database i must say that the import is not perfect: comparing the imported historic data with the data that is live collected from the meteohub i see quite a lot of differences:
- several fields that are derived from direct sensor measurements that are present in the live collected data are empty in the imported historic data, for example the fields daily_rainfall, monthly_rainfall, rain_rate, current_weater_desc, barometer_trend_last_hour et cetera
- the field 'datetime' that is computed during the import has a bug in it: whereas for the live data the format is "20120601.....", the format of the imported data omits the trailing zero before the month: the format is "2012601". This will cause problems when you want to use this field to sort or select data.
I am trying to fix my data in the database using SQL commands.
#-o
Weather Display:
I will check on that missing leading zero problem
you are right in that some of the data is not computed for import compared to live
(as that data is not in the data file used)
I am a bit confused as how you were going from meteohub to wdmysql exactly?
WeerinMontfort:
Morning Brian, nice to hear from you and kind of you to look into the problem.
First of all your question re the connection between Meteohub and MySQL: reading what i wrote in my first post again I understand you are a little confused.
To be more precise: there is no direct upload from the Meteohub to MySQL: I am migrating the data from Meteohub to WD and WD is uploading to MySQL using Control Panel > MySQL Database / the WDMYSQL program (upload historic month data manually to upload past months and real-time 5 minutes update right now).
I am planning to decommission Meteohub and attach the weather station directly to WD as soon as all historic data has been migrated. Until that moment, the weather station remains connected to Meteohub and I have setup Meteohub in WD as a virtual weather station using a TCP-IP connection:
* Control Panel > Station type and settings > Stationless/WLK/Vortex/VWS > “Stationless”
* TCP/IP connect on to Davis VP IP/V VP/Meteohub =>added the IP-adres and port of the Meteohub and “Use withMeteohub” “enabled”
Starting (and resetting every month in my workaround) the WD data logger to extract data from the Meteohub is done by Control Panel > Data Logger > Manual Adjust Start from Time/Date.
From that point on extraction was automatic, but due to the glitches I had to restart the extractions and uploads many times.
Kind of you to look into the missing zero problem in de datetime field, maybe other users will be helped with a solution.
I however have already corrected the data in the MySQL database using the following procedure:
* I have created two temporary field 'Hour_temporary' and 'Datetime_temporary'
* SQL: update `wx_data` set `Hour_temporary` = RIGHT(datetime,8)
* SQL: update `wx_data` set `Datetime_temporary` = CONCAT(`year`,'0',`month`,`Hour_temporary`) where CHAR_LENGTH(`datetime`) = 13
(this skips records in which the length of 'datetime' = 14, these are the months October, November and December that do not have the missing zero problem)
* After checking the data in the field 'Datetime_temporary' (all data was valid):
SQL: update `wx_data` set `datetime` = `Datetime_temporary` where CHAR_LENGTH(`Datetime_temporary`) = 14
(the length of 'Datetime_temporary' in the months October, November and December = 0, these records are not updated)
* Finally i removed the two temporary fields from wx_dataHowever, a problem that I have not solved yet is how to build the missing fields in wx_data.
I was expecting the import module to calculate the day-, month- and year total fields in wx_data, for example for the amount of rain, but this was not the case.
Or otherwise WD to have some basic functionality to rebuild these values (importing data is not the only potential cause of invalid or incomplete data, an option to rebuild data would be an added value).
I am no SQL programmer (probably the procedure mentioned above also can be done in an easier way?), for me the rebuilding of total fields in SQL will be a hard problem to solve ...
Kind regards.
Labbs:
--- Quote from: WeerinMontfort on June 09, 2012, 04:34:08 AM ---- several fields that are derived from direct sensor measurements that are present in the live collected data are empty in the imported historic data, for example the fields daily_rainfall, monthly_rainfall, rain_rate, current_weater_desc, barometer_trend_last_hour et cetera
--- End quote ---
--- Quote from: WeerinMontfort on June 15, 2012, 05:54:01 AM ---However, a problem that I have not solved yet is how to build the missing fields in wx_data.
I was expecting the import module to calculate the day-, month- and year total fields in wx_data, for example for the amount of rain, but this was not the case.
Or otherwise WD to have some basic functionality to rebuild these values (importing data is not the only potential cause of invalid or incomplete data, an option to rebuild data would be an added value).
I am no SQL programmer (probably the procedure mentioned above also can be done in an easier way?), for me the rebuilding of total fields in SQL will be a hard problem to solve ...
--- End quote ---
Hi,
Did you and Brian come up with a solution regarding this?
I am also missing some data after importing monthxxxx.inf files and wonder if it is possible and how to import the missing previous data into mysql? It is hard to compare historic data with sql queries otherwise...
Thank you :)
Navigation
[0] Message Index
[#] Next page
Go to full version