cloudy cloudy

Author Topic: WS-1001 data direct to WD?  (Read 11540 times)

0 Members and 1 Guest are viewing this topic.

Offline Weather Display

  • Posts: 81,003
  • Davis VP2
  • New Zealand
  • OS/Browser:
  • Win NT 10.0
  • Chrome 56.0.2924.87
    • Weather Display
Re: WS-1001 data direct to WD?
« Reply #120 on: March 16, 2017, 10:15:52 PM »
so you can access the raw data on the hub via a IP address in a web browser?
can you post a screen shot of that?

Offline Ruben Perez

  • Posts: 28
  • OS/Browser:
  • Win NT 10.0
  • Chrome 51.0.2704.79
Re: WS-1001 data direct to WD?
« Reply #121 on: March 16, 2017, 10:24:44 PM »
Not yet... I still haven't received the unit. Next week probably I will receive.
I can tell you the advances when I receive it.

I have see a phyton program to decode livedata.htm page with the realtime data, but the user also tells that this method could damage the unit (I am not sure, he say that the unit is very basic and can't handle a petition every 15 seconds all time). I am not sure about it, or the refresh could be done slowly, maybe temperatura isue. I have to try when I receive the unit.

https://obrienlabs.net/code/observerip-weather-station-python-screen-scraping/
https://obrienlabs.net/code/receiving-data-weather-station/
And here, the author of above articles:
https://github.com/weewx/weewx/wiki/observer
Say:
I strongly recommend you do not advertise the scraping method. It is known to cause issues with the ObserverIP unit. The ObserverIP (black box with antenna on it) is an inexpensive unit that can barely survive an uptime of 30 days, let alone screen scraping every 15 seconds.

Ruben

Offline Weather Display

  • Posts: 81,003
  • Davis VP2
  • New Zealand
  • OS/Browser:
  • Win NT 10.0
  • Chrome 56.0.2924.87
    • Weather Display
Re: WS-1001 data direct to WD?
« Reply #122 on: March 17, 2017, 12:38:17 AM »
yes, that was my understanding, that the scraping of the data can cause the hub to lock up
you can set WD to get the data from the WU data

Offline Ruben Perez

  • Posts: 28
  • OS/Browser:
  • Win NT 10.0
  • Chrome 51.0.2704.79
Re: WS-1001 data direct to WD?
« Reply #123 on: March 17, 2017, 07:53:36 AM »
I can't understand why this has to happen... Maybe was a default unit, because weewx suport direct conection scraping web "livedata" page and nobody has reported anything like this. I want to try, because I don't want to depend of external WU or Internet Connection, or installing an additional web server to handle php... This is not necessary sure. If the unit is sending requests every 16 seconds to PHP server, there is no problem to send txt data of a webpage at same rate.

If you want, I can try to program some additional software (like 1-wire) to poll the data from the hub and pass to WD. 

Ruben

Offline Weather Display

  • Posts: 81,003
  • Davis VP2
  • New Zealand
  • OS/Browser:
  • Win NT 10.0
  • Chrome 56.0.2924.87
    • Weather Display
Re: WS-1001 data direct to WD?
« Reply #124 on: March 17, 2017, 07:03:11 PM »
your first link , it says clearly that the screen scaping:
Quote
This method may cause your ObserverIP to crash!

but its OK to instead set the hub to send the data to a web site php script (that creates the json file that WD can then read in)
I have already outlined that above

Offline Ruben Perez

  • Posts: 28
  • OS/Browser:
  • Win NT 10.0
  • Chrome 51.0.2704.79
Re: WS-1001 data direct to WD?
« Reply #125 on: March 17, 2017, 09:33:09 PM »
Well... I have a, maybe, very good idea, but I can't try it until I receive the unit.

If scraping is not a good idea, so I will try to implement a very simple "web server" socket based, listening the packets that IP Observer will send, and I think that I could parse and save this data to allow WD process it. So anybody hasn't to use a webserver to receive the data. When I can test this, I say you Brian, so you could, or implement it on WD or use as separate program to pass the data to WD as 1-wire reader do.

Ruben

Offline Weather Display

  • Posts: 81,003
  • Davis VP2
  • New Zealand
  • OS/Browser:
  • Win NT 10.0
  • Chrome 56.0.2924.87
    • Weather Display
Re: WS-1001 data direct to WD?
« Reply #126 on: March 18, 2017, 12:26:49 AM »
you dont need to do any  work
its all available
the php script is available
to run on a server
which creates a json file that you set WD to get from the server

Offline Ruben Perez

  • Posts: 28
  • OS/Browser:
  • Win NT 10.0
  • Chrome 51.0.2704.79
Re: WS-1001 data direct to WD?
« Reply #127 on: March 20, 2017, 08:19:49 AM »
Yes, I understant you, but I don't want to mount a server on the WD computer, I don't want to run a remote server that doesn't work when there are a storm and maybe ADSL line goes down. I want a local program that take the data of the weather station and pass it to WD. It is very easy to implement, and much better than depending of other services.

Ruben

Offline Weather Display

  • Posts: 81,003
  • Davis VP2
  • New Zealand
  • OS/Browser:
  • Win NT 10.0
  • Chrome 56.0.2924.87
    • Weather Display
Re: WS-1001 data direct to WD?
« Reply #128 on: March 20, 2017, 09:04:08 AM »
the console version of the station does send data via direct TCP/IP socket connection (via the local WIFI)
and WD works with that
but the hub version does not do that

Offline Ruben Perez

  • Posts: 28
  • OS/Browser:
  • Win NT 10.0
  • Chrome 51.0.2704.79
Re: WS-1001 data direct to WD?
« Reply #129 on: March 20, 2017, 05:05:45 PM »
Well, it is really very very easy to parse incoming HTTP requests sent by IP Observer using a socket, no need to scrap screen, no need to start any server.

The code will be like this one. I am doing in C# UWP application, but is easy to do it in any other language.

Start a listener with a socket on port 80:
Code: [Select]
 
            StreamSocketListener listener = new StreamSocketListener();
            listener.ConnectionReceived += Listener_ConnectionReceived; ;
            listener.Control.KeepAlive = true;
           try
            {
                NotifyUser("Server Online");
                NotifyUser("Waiting messages on port " + port);
                await listener.BindServiceNameAsync(port);
            }
            catch (Exception exception)
            {
                NotifyUser("Fallo inicializando servidor: " + exception.Message);
            }


When receive a packet, htto request is only text. So read, parse parameters and return a correct OK response to de box.

Code: [Select]
       async private void Listener_ConnectionReceived(StreamSocketListener sender, StreamSocketListenerConnectionReceivedEventArgs args)
        {
            NotifyUserFromAsyncThread("Conexión " + DateTime.Now.ToString());

            string peticion = "";
            try
            {
                Stream inStream = args.Socket.InputStream.AsStreamForRead();
                StreamReader reader = new StreamReader(inStream);
                NotifyUserFromAsyncThread("Mensaje recibido " + DateTime.Now.ToString());
                peticion = await reader.ReadLineAsync();
                NotifyUserFromAsyncThread(peticion);

                //Send the line back to the remote client.
                string contestacion = "\nHTTP/1.1 200 OK\nServer: LocalBridge\nContent - Type: text / html\n\nProcesado correcto\n\n";
                Stream outStream = args.Socket.OutputStream.AsStreamForWrite();
                StreamWriter writer = new StreamWriter(outStream);
                await writer.WriteLineAsync(contestacion);
                await writer.FlushAsync();
                await args.Socket.OutputStream.FlushAsync();
            }
            catch(Exception ex)
            {
                NotifyUserFromAsyncThread("Sessión cerrada " + DateTime.Now.ToString());
                args.Socket.Dispose();
                return;
            }
            args.Socket.Dispose();


Now in "petcicion" string I have the full request that appears like I wrote on a explorer plus GET verb:
Code: [Select]
GET /weatherstation/updateweatherstation.php?ID=KCASANFR5&PASSWORD=XXXXXX&dateutc=2000-01-01+10%3A32%3A35&winddir=230&windspeedmph=12&windgustmph=12&tempf=70&rainin=0&baromin=29.1&dewptf=68.2&humidity=90&weather=&clouds=&softwaretype=vws%20versionxx&action=updateraw HTTP/1.1

So now is easy to decode, parse, etc. If you put this on WD, it can receive directly the data from IP Observer. In other case, I will recode this to json in the format of the PHP code and save to disk so WD can read it.

In C# is very easy to convert to JSON, but I realized that some parameters has changed the name and others has been added. So I have to do it yet.

Code: [Select]
           if(peticion.Contains("GET"))
            {
                var listado = DecodeQueryParameters(peticion.Substring(peticion.IndexOf('?')+1));
                string json = JsonConvert.SerializeObject(listado);
                NotifyUserFromAsyncThread(json);
            }


Offline Ruben Perez

  • Posts: 28
  • OS/Browser:
  • Win NT 10.0
  • Chrome 51.0.2704.79
Re: WS-1001 data direct to WD?
« Reply #130 on: March 21, 2017, 09:23:16 PM »
Well, first version of a "micro IP Observer listener".
No scrapping, only waiting at programed port. Receive data sended by IP Observer, parse it and save to a file with the same "ws1001.json" file format. You can choose location.

In the IP Observer you have to select "custom" server, and program the IP of the computer when this program runs, and the port that you put on listening program. The listening port is also programable, defect is 85 to leave untouched HTTP port (80) if you have any web server or other program that uses that port (like Skype).

I can't test it with WD because I am running it on a XP computer, and I have to rebuild it specially for XP. If anyone wants to test it, the program is here: https://www.dropbox.com/s/az9vraga7xsfib9/IPObserverRecMFC.exe?dl=0

The program needs Visual Studio C++ libraries that could be downloaded from here: https://www.microsoft.com/en-US/download/details.aspx?id=53587

UPDATE: Versión recompiled with compatibilty for XP SP3 (need the redistributable libraries from MS of the link above):
https://www.dropbox.com/s/iq1w25agsnxwvku/IPObserverRecMFC_XP.exe?dl=0

Well, I have tested it, but something is wrong, I put the file on WD Stationless Aercus "Select RAW Script Data File", but although WD is telling me that is receiving data, no correct values are displayed. Attach capture.

Offline Weather Display

  • Posts: 81,003
  • Davis VP2
  • New Zealand
  • OS/Browser:
  • Win NT 10.0
  • Chrome 56.0.2924.87
    • Weather Display
Re: WS-1001 data direct to WD?
« Reply #131 on: March 21, 2017, 10:02:48 PM »
I also have added ability into WD to act as a web server, i.e to listen for http direct IP connection
that ability is there as well
i.e the WiFi IP server->client
settings

in any case, if you could email me the file you have your software creating and set in WD to use (raw script data file)

Offline Ruben Perez

  • Posts: 28
  • OS/Browser:
  • Win NT 10.0
  • Chrome 51.0.2704.79
Re: WS-1001 data direct to WD?
« Reply #132 on: March 21, 2017, 10:23:15 PM »
I have updated the "compatible XP" version to generate also a Weather CSV file, but also seem doesn't work.

I will try to use your server->client setting.... If this work, much better not needing extra software.

Ruben

Offline Ruben Perez

  • Posts: 28
  • OS/Browser:
  • Win NT 10.0
  • Chrome 51.0.2704.79
Re: WS-1001 data direct to WD?
« Reply #133 on: March 21, 2017, 10:36:48 PM »
I am not sure that "WiFi IP server->client" are working... I have tried putting on IP the own IP or the client IP, but neither of both seem to work. I also don't understant why specity a IP, as it will be a server, so the IP must be set on client... I have changed also Port to 85 that I am using currently.

I attach the files that I am currently generating, and also the "server program" source Project (Visual Studio 2017 C++ with MFC).

Ruben
PD: .txt extensión to name added to allow attachement


Offline Weather Display

  • Posts: 81,003
  • Davis VP2
  • New Zealand
  • OS/Browser:
  • Win NT 10.0
  • Chrome 56.0.2924.87
    • Weather Display
Re: WS-1001 data direct to WD?
« Reply #134 on: March 22, 2017, 01:36:45 AM »
I see whats wrong
there must have been a slight change to the json format
an extra space
I have accounted for that now
use a new .zip update of WD, ready now