I’ve seen the following lately during the last two cold snaps: using autolearn along with WXSIM-Lite, mixed in at different percentages, I’m strangely getting higher nighttime lows with the recommended 60% than I’m getting with either 0 or 100%. In particular, tonight’s low with WXSIM alone gives 18 F. With pure 100% WXSIM-Lite, it’s 21, but with 60% WXSIM-Lite, it’s 24. Logic might suggest 19 or 20 (which will probably be about right). I can see that autolearn is not behaving as I’d like with the WXSIM-Lite in effect, at least in these fairly extreme conditions (OK, it’s not THAT extreme, but it is the coldest so far in a mild winter).
I am looking into this, but am curious whether anyone else has noticed this in cold weather. The highs are just about perfect, but the lows seem too high.
The bad news is something seems amiss with this “ideal” setup, but the good news is that means there’s room for improvement, so I expect to make it better!
I have had a few “cold” mornings in this latest arctic plunge with Wxsim forecasting much warmer than the NWS and the actual low. I even tried running a new forecast once I reached the Wxsim forecast low around 7 pm, the new forecast was just a degree lower than the current temperature. Wxim forecast 9 NWS 3 actual low 3, we had clear skies, the dewpoint was near zero or just below, the winds saved us from zero or below.
It seems to me the last cold spell Wxsim did an excellent job for me, hitting the low several days in advance while our local NWS office took the conservative approach and never forecast cold enough.
Overall, looking at the last 4 days Wxsim has done better than the NWS.
I am running autolearn, and lite at 70.
I don’t see that at all. I’m running with a mix of 60% and the min temps are good. We have cold weather now, colder than -10C. I enclose a table showing the temps for the last weeks (max and min). Yr is from Yr, Fakt is the real max and min and Runar is from WXSIM. he temps are in C.
You hopefully received an email from me earlier tonight with a few more queries.
Alas I am extremely snowed under with work at present and can not allocate much time to this, though owing to it’s obvious global affect I could not omit this very small and sorry not much context to the mix.
I have only had time to do a few screen dumps from my charts which basically look at the difference between (HIGH/LOW) Forecasts (5am and 11am) and also the difference between (5am Forecast v Actual) and (11am Forecast v Actual).
Not really a good analytical compare though does highlight issues.Note:I Have also omitted WU from the image/compares.
Hopefully the images are self explanatory.
Some further info if required as I may not get around to replying for a while - due to work load.
Note: WxSim - Engaged in Wret, Weight Recent More on the 24th Aug 2014 and changed, Autolearn to look at 90 Days Historic from previous setting of 45 Days Historic.
After issues reverted back to original (disengaged Weight Recent More on the 23rd Sept 2014 and changed Autolearn to look at 45 Days Historic). Then re-engaged 90 Days and Weight Recent More on 13th Nov 2014.
Note:We began collecting new data using WxSim-Lite (for Beta Testing) on 01/12/2014 and Inclusion of WxSim-Lite Bias Corrections engaged in Auto on 08/12/2014 (pre-Feed into WxSim-Forecasts).
Note: Still Running WxSim-Autolearn Bias Corrections). On the 20 April 2015 we reduced WxSim-Lite Bias inclusion percent to 55%. Started reducing slowly as of the 24th April 2015. Then (On Tom’s advise) on 19th April 2015 increased to 70%. More recent advise has suggested a mix of 60% which we engaged on 14th of Jan 2016.
Results of these changes from a MAE and % inclusion point can be viewed here http://hrvistaweather.com/correc.txt
Sorry about size: You may have to scroll to Right to see the more recent issues. just click the screen once and use your arrow keys.
The 2 additional attachments below are from the standard “Compare” script and depict last 465 Days (which in essence are rather good really) Note: the last 2 days are as a result of including the (Day+5 and Day+6) FCasts late last year - so are not truly representative against the previous.
Cough Cough as we are in our 3rd day of Smoke filled skies due to Bushfires in the North and West of Tassie and little wind to blow it away. 8O
Kind regards,
First, thanks, Tony, for all that info! Also, I’ll reply to those earlier questions you had. I did get them. Been a crazy week.
OK, I do have a general request for data to help me sleuth out the problem I’ve seen lately. I could use cor.txt, correc.txt, and details.txt for a few locations. I’ve got a lot of data of my own I’ll be looking at, but that’s just specific to my site. My suspicion is that there’s some kind of interaction between autolearn and WXSIM-Lite which, under special circumstances, causes the autolearn correction factors to drift to unreasonable values. I say “special”, because at first glance, it appears that the combination worked well for me for over a year. My first guess is that the freakishly warm December we had got things confused. I should know a lot more soon, and will be able to take corrective action. It seems it takes time to expose these sorts of things!
Here are mine as well. I have turned off both Autolearn & WXSIM-Lite as they are driving way to far into the ditch…per say. As you mentioned below, this seems to be a recent development.
Thanks for posting these. I’m making good progress investigating the issue. I’ve found that autolearn/wret/wxsim/wxsim-lite have been working as intended with regard to this, but that one particular bias correction factor - the “slope” or tendency towards normal - is more susceptible than I had realized to odd patterns in the weather it’s “learning” from. I’ve found that, even with 60-90 days to learn from, this correction factor is varying significantly from year to year (I’ve been tracking it since July, 2013). I knew it might vary seasonally, but now see how easily it can get knocked around and thereby give occasional bad results.
I’m not quite settled on a solution yet, but have some ideas and hope to have a fix ready soon. Fortunately, it’s not affecting most users significantly. I think it surfaced for me because of a return to normal winter weather after having “learned” from a highly unusual December.
Fantastic data you all have posted! This is a great example of “crowd-sourcing”, I guess you might say. I think I’ve discovered a type of bias in the GFS data (as processed through WXSIM-Lite). It shows clearly in the correc.txt files (which I’ve now converted to Excel files and done all kinds of analysis on). except for Tony’s (hrvista’s), which is pretty neutral in this regard. Basically, for the rest of these sites, the GFS-derived WXSIM-Lite data shows a tendency to get too extreme (gets too warm on warm days and too cold on cold days). It’s not a big error, but what’s happening is that autolearn/wret is detecting it and trying to “fix” it through the learned bias correction factors, but since WXSIM is having only about a 40% say in the matter, it’s having to try “really hard” and is making nearly (or sometimes completely) “out of range” slope corrections. That, combined with the variability/volatility in this factor (tncor, or slope) is causing sporadic poor forecasts of relatively “extreme” temperatures. It’s not always happening, and not for every user, but under certain conditions it’s surfacing. Two items of good news: (1) now that I know pretty much what’s going on, I can find a way to address it (though I’ve got to give it careful thought) and (2) in spite of this issue, all of the posted data show substantial improvement resulting from use of WXSIM-Lite (can’t say for sure for tbrasel, because he used WXSIM-Lite almost from the start). I was generally able to compare nearly a full year both before and after institution of WXSIM-Lite, and found percentage improvements: hrvista 11%, weatherbee 14%, mikeym 16%, and jmcmurry 23%! I believe upcoming changes will improve these even a bit more, maybe another couple or few percent. We’ll see!
Meanwhile, the data posted here has been from the eastern half of the U.S. and one from Australia. I could use a bigger geographic/climatological sample (i.e. from Europe, Southern Hemisphere, etc.). Data from anyone who started using autolearn in early 2014 or before, and who started using WXSIM-Lite pretty steadily at least 10 months ago would be great.
Here are mine from northern Sweden.
I seem to have had the best accuracy when the temps where close to 10C above normal in November and December. We’ve had a bit more normal temps in January and it seems WXSIM’s forecast errors are increasingly too warm with falling temperatures.
I noticed you found improvements for everyone providing data except for mine. I thought my forecasts had shown marked improvement. Did my data not show that?
