cloudy cloudy

Author Topic: Sharing XML (WDFULLDATA.XML)  (Read 8819 times)

0 Members and 1 Guest are viewing this topic.

IT Farmer

  • Guest
Sharing XML (WDFULLDATA.XML)
« on: December 30, 2005, 09:03:11 AM »
Not sure if there is a need or even a topic for this. But I was wondering, should we start a topic on sharing the wdfulldata.xml data so you can put different weather station data on your webpage. Might be cool for weatherstation that are in the same county or state?

Just a tought.

Anyway I am working on a wdfulldata.xml file that allows all the tags as they are in the site made by carterlake http://www.weather-watch.com/smf/index.php/topic,12233.0.html I been using his layout on my sites. Why? because they pretty damn clever that is why :-)

My wdfulldata.xml from my weatherstation can be found on http://www.dayboro.info/wdfulldata.xml
Some of the templates in action http://dayboro.theweatherzone.com

I will post them for download if there is a need, but my real question was about the need for sharing wdfulldata.xml stuff.

Have fun
H.


Offline administrator

  • Administrator
  • Posts: 10,882
  • Get answers to your questions - use Forum SEARCH!
  • Blackpool, UK
    • Weather-Watch.com
Re: Sharing XML (WDFULLDATA.XML)
« Reply #1 on: December 30, 2005, 01:32:59 PM »
What's needed is a standard (formal or de facto) XML schema (format) for weather data. If we can agree one of those the sharing of data is possible. I've been thinking about providing data from the MML server in XML format and had started to look at possible schemas, but haven't made much progress yet.
Chris
Weather Forum Administrator

IT Farmer

  • Guest
Re: Sharing XML (WDFULLDATA.XML)
« Reply #2 on: December 30, 2005, 08:37:32 PM »
The schema in wdfulldata.xml would that not do for a starter? Maybee setup a dif tree structure for different weather stations. At the moment it is only:

-wxdata
--param

What if we can do something like:
--wddata (data for weather display)
---vpst (data for vp weather station)
---dalas (dalas weather station
--mmf
---etc.
---etc.

Setting up the xml and schema should be the easiest part would it not? Anyway lets see where otherpeople come back with.

H.

Offline Weather Display

  • Posts: 87,027
  • Davis VP2
  • New Zealand
    • Weather Display
Re: Sharing XML (WDFULLDATA.XML)
« Reply #3 on: December 30, 2005, 08:58:32 PM »
looks good in IE, but its not working in FF, well not for me anyway...hummmm
i see AJAX is making more news....Julian got that to work with WD on his site :)

Offline administrator

  • Administrator
  • Posts: 10,882
  • Get answers to your questions - use Forum SEARCH!
  • Blackpool, UK
    • Weather-Watch.com
Re: Sharing XML (WDFULLDATA.XML)
« Reply #4 on: December 30, 2005, 09:54:41 PM »
This is intended as constructive comment...NOT criticism...

The example wdfulldata.xml is an XML record that's appropriate to your station, but it's a very big record and structured the way it is simply because it's the set of tags that WD happens to output for your station/station type. That's probably not a good starting point for the design of an XML schema. If you're designing something for general use, probably by users of many station and software types, you need to make sure it's as station/software agnostic as possible, with station/software specific data defined as optional.

There needs to be discussion/agreement about a number of topics. We can't be sure what people will use the data for, so we need to define the fields/values/etc very well to ensure people don't use the data incorrectly, especially if trying to output it from some other software than WD or more so if they're using the data as a feed into some forcast modelling package! Some things that I think need discussion/agreement (and this isn't an exhaustive list) are:

- should it be one huge record, or should some normalisation of the structure be done, e.g. why not have a 'current conditions' schema and a 'historical conditions' schema or lite, standard and full schemas? Some people won't have any interest in historic data, so making them download what will end up (when fully defined) as at least a 30-40k data file to get the few current conditions details they want is probably unrealistic...both in terms of their own and your bandwidth usage. It might not be a problem to everyone, but if I started to use the schema as defined to publish data from MML in wdfulldata.xml format my bandwidth usage would multiply by a factor of at least 6-8 times!

- should fields be named after the WD tag names, or given a more appropriate name, e.g 'hum' v 'Rel_Humidity_%'

- date/time formats (12 v 24 hour, dd/mm/yy v mm/dd/yy v yyyy-mm-dd v dd-mmm-yyyy, etc).

- are times in UTC or local time, with or without DST? Is a DST indicator needed?

- whether fields are output with embedded units or with separate units fields or whether units are just assumed/part of the schema definition, e.g. what units apply to <visibility> ???

- whether fields are output in all possible units (pressure in mbar/hPa/in/???)

- numbers of significant digits/number of decimal places. How to handle very big/small numbers 3.142*10^6?

- how should missing data be denoted?

- whether to include all possible fields just because there are tags for them, or whether they're not really weather related so should be in another schema or even calculated locally, e.g. Easter dates.

- how do you define a month/day/other period? Is it the last 31 days, last calendar month, calendar month to date, etc? Does it start at midnight UTC, 0900 local time, or can it vary?

- what language to adopt, e.g. for descriptive fields...local or english or french or spanish or Esperanto, or include all possible languages

- should composite fields, e.g. <coldestnightonrecord>22.7 C  on: 28 Dec 2005</coldestnightonrecord> be broken down into constituent parts, e.g. temp, temp units and date field?

- what to do about WD tags (or other data) added after the schema is defined and what to do if a WD tag changes it's data/structure subsequent to a schema being agreed? wdFULLdata.xml is really only wdFULLdataon31December2005.xml because new tags seem to arrive every week or three.

Assuming people don't just adopt the WD tags version of wdfulldata.xml, then it will probably need Brian to add some new tags to support the new schema, so there's probably work required in WD to enable all this once it's been decided what data is needed.
Chris
Weather Forum Administrator

IT Farmer

  • Guest
Re: Sharing XML (WDFULLDATA.XML)
« Reply #5 on: December 30, 2005, 10:29:31 PM »
This is intended as constructive comment...NOT criticism...

Dude, you must be a keyboard wizard. Yep spot on. I am just a "kiddo" looking around the corner on this stuff (like windy pointed out, my site works in IE not in FF so i need to fix that).

I like using the XML file to reduce the traffic, that was its purpose. If we can reduce the traffic etc to update web pages would be great. I can only see what I can see and that is the output from wd. The tags sort of make sense to me but if we can come up with  a common naming convention that would be great but "why fix what ain't need fixing :-)". This is why I also start stripping the non VP tags.

If there is the ability to have multiple XML files for different purposes would be good, but my initial thought was not to upset the WD code to much but that is my oppinion.

Maybee something like:
general tags XML file
station specific tags XML file (or have all station writing to the same tags but that reduces the ability to have different stations hooked up, not sure if you actually can)
history specific tags XML file (more a search and retrieve from somewhere eg using the client*.txt files)

But good to get the discussion going. The question remains are you guys ready for something like this. I am just a bystander and actually do not do anything :-). But here is a commercial thing, if we can have multiple mods for different purpose it might be actually an add on $$ (Oops, shouldn't give Brian any ideas)

PS, criticism????? Puff I always assume other people know more than I do so I see it as a learning thing. I do not get easy offended.

Offline phldml3

  • Posts: 21
  • Millard, Nebraska
    • Millard Colonies Weather
Re: Sharing XML (WDFULLDATA.XML)
« Reply #6 on: January 06, 2006, 09:30:30 PM »
Couple of thoughts here:

Some of your schema questions could be answered by simply following WMO standards.  Like weather observations are typically reported in UTC.  Wind speeds are reported in KTS (I know everyone thinks that because the U.S has not converted but really it's the WMO standard).  Temps in Celsius.  So on and so forth.

Secondly, (and by applying this statement I'm not proporting favoritism to the U.S NWS) why not apply a schema similar to the U.S NWS XML "standard".  Or better yet, determine the national schemas and utilize them.  I've utilized the NWS so that when my personal weather station is off line, I can easily use the closest NWS site because the schema doesn't change.  Also, if all of the local personal weather stations did similiarly a "standard" XSL could be applied to all obs utilized by a website for use in 'plotting' obs.

Just my thoughts.
Paul

Offline phldml3

  • Posts: 21
  • Millard, Nebraska
    • Millard Colonies Weather
Re: Sharing XML (WDFULLDATA.XML)
« Reply #7 on: January 06, 2006, 09:37:49 PM »
Forgot to mention it, in a couple years this may all be for not anyway as WMO working out that all alpha-numeric data will be formatted into the digital format of BUFR.  We'll have to wait and see what an observation will look like when that comes out!?!

Paul

Offline administrator

  • Administrator
  • Posts: 10,882
  • Get answers to your questions - use Forum SEARCH!
  • Blackpool, UK
    • Weather-Watch.com
Re: Sharing XML (WDFULLDATA.XML)
« Reply #8 on: January 06, 2006, 09:41:27 PM »
I'm sure there are standards, e.g. WMO, that could be adopted, but you've got to get through the multi-national discussion first. For example, US users really won't like using Celcius and having to convert to get to the real temperature :wink: Perhaps a starting point is propose WMO units, allow a 4 week discussion/argument period then enforce the WMO units :wink:

NWS XML schema...is it published. Last time I looked all I could find was an example snippet. From memory the definition of temperature seemed to run into about 400 lines. Either I didn't look hard enough, or perhaps they've published the scheme since I went looking (about 4 months ago I think).


Sorry for dipping in and out of this discussion. I'm a little busy with various things at the moment, but I am really interested in the definition or adoption of a reasonable XML weather schema.
Chris
Weather Forum Administrator

Offline phldml3

  • Posts: 21
  • Millard, Nebraska
    • Millard Colonies Weather
Re: Sharing XML (WDFULLDATA.XML)
« Reply #9 on: January 06, 2006, 10:00:30 PM »
WMO is World Meteorological Organization.  That is the multinational body of the UN.  Thus I'm not proposing a US standard.  And US standard temp is celsius in a METAR observation because we use the WMO standard.  Any posting on the web of an 'official' US observation is already a conversion.  Last time I looked at it, only the Eastern Bloc nations reported outside the WMO standard and I believe the only difference there was that they used MPS rather than KTS.  Not sure why, but not like in this computer age we can't convert easily!  Besides I'm 'merican and like Celsius!  It's origins are more meteorologically sound and it is extremely easy to convert to F (notice I can't even spell our unit of measure) if need be.

F= ((C x 2) - 10%) + 32

Just my thoughts!
Paul

Offline Weather Display

  • Posts: 87,027
  • Davis VP2
  • New Zealand
    • Weather Display
Re: Sharing XML (WDFULLDATA.XML)
« Reply #10 on: January 06, 2006, 10:04:19 PM »
oC:= (oF - 32) /  1 . 8   :wink:

Offline phldml3

  • Posts: 21
  • Millard, Nebraska
    • Millard Colonies Weather
Re: Sharing XML (WDFULLDATA.XML)
« Reply #11 on: January 06, 2006, 11:24:15 PM »
Brian,
   You are absolutely correct.  I forgot to mention with my formula it was a close approximation that you can easily run in your head.  It was a formula given to me back in my observing days by a senior officer.

Paul

Offline administrator

  • Administrator
  • Posts: 10,882
  • Get answers to your questions - use Forum SEARCH!
  • Blackpool, UK
    • Weather-Watch.com
Re: Sharing XML (WDFULLDATA.XML)
« Reply #12 on: January 06, 2006, 11:56:42 PM »
Sorry...I know who WMO are and how they fit into the equation. My comment about US units issues is that the intended purpose for sharing XML files was to allow data sharing between web sites. When the subject of units comes up there is often a comment from US users that they don't want Celcius because no-one understands it. WD is littered with tags showing degF equivalents of the base degC data (and other units) mainly to suit the US market.

I have no problem with degC..although KTS is a bit alien to me. I tend to read KTS as being more or less MPH to avoid confusion!

Have you found the NWS XML schema definition?
Chris
Weather Forum Administrator

IT Farmer

  • Guest
Re: Sharing XML (WDFULLDATA.XML)
« Reply #13 on: January 07, 2006, 02:00:12 AM »

Offline phldml3

  • Posts: 21
  • Millard, Nebraska
    • Millard Colonies Weather
Re: Sharing XML (WDFULLDATA.XML)
« Reply #14 on: January 08, 2006, 02:25:52 AM »
Sorry all, I should have realized that my NWS comments would stir up US nonstandardization comments.  But what I'm trying to get at is what do your Met services use.  I'll break from any more comments here until I can contact my friend at UKMO and see what their XML standards are.  Sorry for intruding on your parade.

Paul

 

cumulus