There are a few rather esoteric steps to getting your KML files recognized by the Google Earth search engine. I think I have it set up, but we will see. -ben

[/Projects] permanent link

Well, I found my bug in my lowpass filter. I had two variables, with very similar names, being used for the same thing. But one wasn't recalculated as it should have been, the result was that I had figured weighting numbers based on, say a window of 15 points, but only summing 11. That filter looks pretty good now.

I believe that it was a mistake to put the interpolator inside of the data convertor program. I need to destroy outliers BEFORE I do any interpolation.

Also, I'm not convinced that straight-line interpolation will suit our needs. I'm thinking more in the lines of a circular arc fit.

So now, the path might be:

[/Projects] permanent link

I finally found my last problem with the interpolation. There were two. But last one was the most vexing. It was my calculation of fp seconds from datetime.timedelta. I had put this off in its only routine and had written: dt.seconds + dt.microseconds * exp(-6). Obvious when you see it: exp is exponent of base e not base 10. The correct expression is: dt.seconds + dt.microseconds * 10**(-6).

I really like the Python matplotlib plotting routines and interface for viewing, As you can see, I was able to pinpoint my problems thanks to the ease of this interface and its ability to create images.

[/Projects] permanent link

I went ahead and hacked my tidePlot.py which will do comparisons of time based data (using the timedate class). Using it to visually analyze my interpolation (added as an option to tideconvert), I see that it still needs work. The slope isn't right, and it doesn't catch all gaps in the data (right side of image).

[/Projects] permanent link

from http://seehuhn.de/pages/pdate

Conversion between Unix times and datetime

To convert a Unix time stamp to datetime use the fromtimestamp constructor:

 >>> from datetime import datetime
 >>> datetime.fromtimestamp(1172969203.1)
 datetime.datetime(2007, 3, 4, 0, 46, 43, 100000)

 >>> from datetime import datetime
 >>> from time import mktime
 >>> t=datetime.now()
 >>> mktime(t.timetuple())+1e-6*t.microsecond
 1172970859.472672

My example:

In [26]: d1 = datetime.datetime.now()

In [27]: d1
Out[27]: datetime.datetime(2009, 2, 3, 21, 25, 4, 588232)

In [28]: time.time()
Out[28]: 1233714322.753165

In [29]: time.mktime(d1.timetuple()) + 1e-6 * d1.microsecond
Out[29]: 1233714304.588232

In [30]: 

[/Projects] permanent link

My simplistic interpolation scheme was to count the number of contiguous bad source data lines and interpolate that many points over the gap. I got what I planned, but more often than not, the bad source spans many more sample points than bad lines caught.

Now it looks as though I need to do a little analysis on the incoming data to find an average sample rate, and find any spans that are above +/- 1 second of that rate. Then interpolate over the gap, and sample the interpolation at the computed sample interval.

[/Projects] permanent link

The current process design is:

Convert from pressure to depth

This process shows some gaps in the data, often small due to hiccups in data transfer or data logging. During this process, the time will be converted to epoch time (seconds from January 1, 1970). The gaps will be filled by interpolation. A byproduct of this stage is the statitistics of missing data and interpreted data.

Boxcar Filter with Sampling

The filter will smooth the raw data, taking out jitter that might be introduced by wind, waves, and wakes. This dataset is more dense than is needed to meet the requirements of NOAA, therefore only a fraction of the filtered data is output. The output has to align with the times of NOAA data. The results of this stage will be used in two different production paths: Offsets and Constituents. A byproduct of this stage is the statistical deviation of the raw data from the filtered data.

The two product paths

Offsets

The stages off working offsets are:

1. Find peaks
2. Find Mean Low Low Water
3. Compare peaks to a primary station
4. Produce offset and range value for predictions based on the primary station

Constituents

Using least-squares determination of constituents (the tappy.py program). Use the constituents to produce a tide predictor

Comparison

The two prediction methods will be compared to new data as it goes through the first two stages: Conversion and Filtering.

[/Projects] permanent link

CCOM operates several tide stations. The three that are most likely to be recording data are:

• Jackson Estuarine Lab at Fuber Straights, Great Bay, New Hampsire
• US Coast Guard Station, Fort Point, New Castle, New Hampshire
• Maine Maritime Academy, Castine, Maine

The last of these is generating the most complete data set, but has only been in operation since the summer of 2008.

What we want to achieve is the ability to do predictive tides and compare to actual tides.

I have done this before on the Jackson Lab data using the Perl programming language, but it was for a single evaluation of time offset and amplitude factor with reference to the NOAA Portland primary station (used for the Portsmouth area tides, as well).

Since our prefered programming language is Python, I will probably focus on Python tools, such as tappy.py by Tim Cera of The St. John's River Water Management District. Tappy uses the Least Squares method of finding tidal constituents.

I will probably use Python's matplot for data visualization. There also exist some rather serious math modules for Python, e.g., numpy and scipy. (Both are used by tappy.)

The public result of this work will be web access to data and predictions. I also hope to have some fairly easy-to-use tools for working with tide data, particularly for doing bathymetric surveying.

[/Projects] permanent link

I have been negligent in posting the process of buiding the last bit, the data visualization bit, on this project. But you will be able to read all about it in a forthcoming PowerPoint Presentation that I will post here.

The results are a set of several hundred Google Earth kmz files. You can access the top of the data at Kurt's Developement Web Site.

Now it is time to start the next project: TIDES

[/Projects] permanent link

As pristine an XML implementation as KML is, I feel that giving an object some attributes rather than having to only use separate style objects would make use color so much easier.

I have taken Kurt's first pass on generating KML from the database and added the time field in a timespan attribute to linestrings.

I still have a long way to go to know what elements and attributes fit where in KML.

[/Projects] permanent link