Ok, not sure this is the right place for this topic but…
I don’t know about everyone else but I’ve had troubles with my station giving bogus reading once in awhile or because of a crash I get some bogus data in my system. Finding the bad lines in the log files is very difficult. I’ve been playing around with a quick little program to look through the logs and find possible errors (baro drops 1 full point in a min, wind gust over 70 miles per hours, etc) I have a working demo that I’ve used to check my logs and it helped me find several errors. I then manually correct the log file.
If anyone is interested in looking at the program, let me know. If there is interest I’ll maybe make it nicer and available to everyone. Thanks
LOL, forgot the file on my USB drive at work, I’ll send it to you on Monday. It doesn’t modify anything, it just finds errors based on a checking system I created. It then gives you the exact date and time to look at. From that I was easily able to correct every error in my log files!
Can it be adapted to look for dew point drops? My 1-wire system seems to generate these in my main WS data if it stops and starts… usually it doesn’t affect anything except for a humidity or DP low for the day…
Yes, very interested in the utility. Email is there. thanks
What a good idea. I’d love to try it. I’ve given up trying to correct my log files. Also, see the current relevant discussion in WD feature suggestions.
Thank you.
w0mbat
It looks for any drop or gain in all the measurements of a certain number of points, like for example a drop or gain of 5 degress from the previous minute in temp, it will flag it and show you a message to check the temp for date xx-xx-xxxx at xx:xx. It does the same for every other measurement, ±.1 for baro, ±8 I think for dew point. etc.
I ran it through 16 months worth of logs and it only pulled out stuff that was really an issue so I think it’s pretty good for USA numbers. I’m not sure how it will work with other formats yet so yes, I think maybe I will make it adjustable. Thanks for the idea. Now if I wouldn’t have forgot my USB drive at work! I’ll post it on monday for people to look at and test.
Oh yes - adjustable offset for degrees C and hPa different from inHg… Thanks - good things take time!! Does it check for out of order times? ie missed new day: 23:00 to 00:00 and for us Kiwis and Ozzies - 08:50 - 09:00 for our gentlemen’s morning change over (when we check for the frost and the rainfall etc at gentleme’s times!!) - well when they used to… HAHA!
Well if looking for sudden offsets, then a sudden 4 degree as a number woul be ok - just adjustable to suit the diferent metric or Imp units - same for the barometer drops, 1" of Hg drop would mean several hPa.
The other errors are key to having continuous data to analyze for whole days averages etc. The NOAA and climatedataout tables need to be able to define the end of an old day and the start of a new day. Which for most folks is midnight, whereas the Australians and NZdrs use an 8:59 end of day.
Some utilities also check for dupe times and dates, which can sometime happen with data recovery from data loggers. But one would assume you would only have to check say 100 lines down, but still that is quite a bit of work for every line to check ahead… I dunno…
Or perhaps even point out missing data lumps more than 10 lines or so… ie date/time jumps more than 10 mins.
Sorry - i am making more work for you!! When it sounds like an excellent basic tool to start with!
Actually, the current method finds out of order data entries real easy because there will be some sort of difference between the previous line read and a line that even 10-20 minutes out of sync. I found every instance of that in my logs with that method.
I don’t think it would be too hard to make a check to be sure a certain time had readings, I will be able to add that for sure. I’ll post it later today for people to try and then I can make changes to fit the needs as they arise (I hope)
I would be very interested in this. I seem to have a problem since I updated to the new version of WD. I get wind gust readings @ 255 mph for a few mins here and there. Not sure it is really WD, may just be a coincidence. I just changed the battery in my wireless anemometer to see if that helps, but since then it has happened 2 more times. It lasts for around 2-10 mins. It may be the anemometer itself, but not sure how I would check it. It is less than a year old. So your little program would be a great way of finding these little problems and correct them.
Thanks Scott, it was bad because if you skipped or missed readings over the day change period, then they have to be guessed at and manually entered so all the Average and Extreme values can be calculated for that day and you actually get stuff on lines in the NOAA or climatedataout tables, otherwise all you see is — right across the table for that day. Similarly for end of month…
