New program to help with large numbers of .wxf files

Hi Everybody,

I’ve been aware of how slow autolearn/wret (and unresponsive, if you try to interrupt) and WXSIM-Lite can be when you have a lot of archived forecasts, even if it eventually finishes (as is almost always the case). While the analysis jobs are best scheduled during late night/early morning hours anyway, these long jobs could overlap with planned forecasts and possibly cause errors. I’ve advised in the past to sequester old forecasts in another folder and not to make too many forecasts per day, but the former can be tedious to do manually, and you might really want to run WXSIM every hour.

I tried recently to add a progress bar to wret.exe, but it “froze”, too! Then I thought of something: a tool to “clean up” excessive files, reversibly, and according to various criteria. I’m going to make this tool, WRETMATE, a standard, included part of the distribution package and upgrades, at no cost (new WXSIM version coming soon!). Of course, I’d like to test it out more first!

If this sounds like it will help, please give it a try. It has no initialization files - it’s just a simple .exe. Help is included in the rpogram (there’s a button for that).

Here’s the link. Just put it in the folder with your WXSIM files (usually c:\wxsim):

www.wxsim.com/wretmate.exe

Let me know how it goes!

Tom

Hi,
Just tested this and got Run-time error’68’: Device unavailable direct after start.

Regards
Bo

Thanks for that report. Hmmm … the only thing I can think of it that it makes reference to c: and d: drives. Does your computer not have one or both of those drive names?

Tom

Tom,
Seemed to work fine for me.
Tom

I’ve posted another version which should avoid that error. I found there was a way to refresh the file list box (surprise … it’s called “Refresh”!) without changing drive letters. Should have known that. :wink:

Anyway, the new version is at

www.wxsim.com/wretmate.exe

I’d say the main thing to is to rename all the files older than 400 days, and only do that random cull thing if you make lots of forecasts per day (like over eight) and have found WXSIM-Lite and/or autolearn/wret to be problematically slow.

Let me know how it goes!

Tom

Tom - this worked okay for me although when renaming files more than 400 days old the program initially went to ‘not responding’ after renaming a few hundred forecasts, plus I had the revolving blue circle. After 10 minutes I shut the program down on the ‘X’ which was the only way I could do it.

I had to rerun the program 7 times like this before I had renamed from 3816 down to 2020 forecasts - I do 5 forecasts a day - and the program ran fine on the last run when it achieved renaming all the required files. I’m not running a particularly slow computer, dual processor with 8 GB of memory but I suppose the problem could be down to my computer?

Hi,
Now it

Just trying this… Identified 4417 files… Told it to rename everything over 400 days old… Does a few then appears to hang with not responding in title bar and spinning blue circle… Also brings the rest of the PC to its knees in this state… The PC is not a slouch at all with an i7-4771 quad core CPU and 16 GB of RAM.

I’ll let it run a bit and see if it eventually finishes them all.

It took me 25 minutes to complete, have to close down WeatherDisplay during this.

Bo

It seems to choke (but continue working) after renaming 50-70 forecasts…

I’ve been experimenting with a folder with 579 forecasts, and it does the apparent “freeze” and blue spinning circle thing, but proceeds to finish. In other words, it looks frozen, but it’s actually still working and always finishes. I don’t know why this happens, but it’s a lot like what wret.exe itself does. The difference is that once you do this and reduce the number, wret should run faster and also it should be easier to maintain the smaller number of forecasts (using wretmate) than it was doing the initial “clean-up”. Still, I’m trying to think of how to speed it up and avoid the “pseudo-freeze”. I’ll play around with some ideas and see if anything helps.

Thanks!

Tom

I found a way to move some code around and speed things up considerably (by a factor of 2 or even 3), and now I don’t seem to be getting the “pseudofreeze”. Give this a try:

www.wxsim.com/wretmate.exe

Also, I did slight updates of autolearn, wret, and wxsimlite to include buttons for opening wretmate conveniently:

www.wxsim.com/autolearn.exe
www.wxsim.com/wret.exe
www.wxsim.com/wxsimlite.exe

I’m curious to know how much faster this is, and if it avoid freezes, even with alrge numbers of files. You might first use the button to restore all.wxz to .wxf to give it a full-scale test.

I did just think of a possible problem: if you periodically use the random cull feature, there will be a relative lack of older files compared to newer files, because the older ones will have been culled repeatedly. One option is to require a restoration of all .wxz each time, but that’s slow. Maybe I need to make it restore only the files less old than the limit on the button for renaming old files (i.e. 400 days). Gotta give that some thought!

Let em know what you find! :slight_smile:

Tom

Tom I am a little puzzled by this need to cull or rename files. I had a look at my files yesterday and find I had 8000+ wxf files going back to 2013 yet my autolearn ran in around 20-25 minutes and wxsimlite seems to take around the same sort of time. ALL this on a mini PC with a quad core celeron. I have no removed all files to a separate folder so I have only those from March 2015 forwards. I’ve not re-tested autolearn etc to see if it made it any faster.

Stuart

Using the four latest program files and having renamed all my .wxz back to .wxf I then renamed those over 400 days to .wxz again. The whole process took under 40 seconds including a 15 second ‘not responding’ warning.

A thought - perhaps it would be possible to include an optional schedule for this - maybe once a week or month at a selectable time?

Just tried the new version… Significantly better… I have 5061 forecasts in the folder… It still does the hang thing after about 7 seconds in each scenario, but it completed each in a reasonable amount of time:

  1. Restore 3066 .wxz files to .wxf – 8 seconds
  2. Rename all .wxf older than 400 days – 7 seconds
  3. Randomly rename 50% of remaining .wxf to .wxz – 8 seconds

Rick

I realized this yesterday with the old version when I kept killing the “Not Responding” exe and starting it over.

With new version, just did a rename all to .wxf, and then did the other 2 steps.

Maybe the check for forecasts older than x days before random cull would be a good idea.

Quick question – does the random cull make sure it only renames proper date formatted files? Just found when I went to run autolearn, the latest.wxf was missing even though a forecast had run about 15 mins prior. Just ran the forecast again, and didnt run into that the 2nd time.

Glad it’s better! Stuart’s point is a good one: this may not be necessary; many of you weren’t having any problems anyway. I have had the impression that some people might be, and my old, slow computer I run the programs on does take quite a while.

By the way, I have a lot of variables dimensions to 9999, to when you get very close to 10,000 there could be issues (though I believe I have some warnings built in).

At the very least, it would be good to rename the 400+ day old files, as nothing older than that will be used, and you might as well speed it up.

I guess the main concern/complaint I’d had from people was the unresponsiveness (but NOT truly frozen) appearance of wret during the process. I tried to find a way around that and have not managed to. I think it just gets awfully busy and hogs processor and/or memory, which is a good reason to schedule it in the middle of the night!

I’ll give some thought to the question of whether the random cull is needed. I might leave it in there and label it as “only if needed”. I’m all ears! :slight_smile:

Thanks!

Tom

Knowing nothing about this I’m prompted to ask how wret is currently ignoring 400 day old data? Maybe checking the file date would be quicker than what it is doing now…

Niko the files in question have a filename containing the creation date and hour so a directory list will simply allow the correct files to be selected. You get a file for each forecast run so if you do 24 hourly forecasts you get 24 files per day of this specific type (in total you get 4 files per forecast but only one is a .wxf file).

Stuart