MAPERL:

SVMOP:Projects: Submitting KML content to Google Earth -- not as easy as Flickr

Sun, 15 Feb 2009 15:15:00 EDT

There are a few rather esoteric steps to getting your KML files recognized by the Google Earth search engine. I think I have it set up, but we will see. -ben

SVMOP:Projects: New Design Ideas for Tide Tools

Mon, 09 Feb 2009 19:59:00 EDT

Well, I found my bug in my lowpass filter. I had two variables, with very similar names, being used for the same thing. But one wasn't recalculated as it should have been, the result was that I had figured weighting numbers based on, say a window of 15 points, but only summing 11. That filter looks pretty good now.

I believe that it was a mistake to put the interpolator inside of the data convertor program. I need to destroy outliers BEFORE I do any interpolation.

Also, I'm not convinced that straight-line interpolation will suit our needs. I'm thinking more in the lines of a circular arc fit.

So now, the path might be:

convert from pressure to depth

do data cleaning using a slope corrected standard deviation over a window

interpolate and resample (I think I can combine these)

smooth (low pass filter)

find peaks, MLLW, constituents, etc.

compare prediction methods with cleaned but unsmoothed data

SVMOP:Projects: Don't program when you mind isn't clear.

Sat, 07 Feb 2009 16:05:00 EDT

I finally found my last problem with the interpolation. There were two. But last one was the most vexing. It was my calculation of fp seconds from datetime.timedelta. I had put this off in its only routine and had written: dt.seconds + dt.microseconds * exp(-6). Obvious when you see it: exp is exponent of base e not base 10. The correct expression is: dt.seconds + dt.microseconds * 10**(-6).

I really like the Python matplotlib plotting routines and interface for viewing, As you can see, I was able to pinpoint my problems thanks to the ease of this interface and its ability to create images.

SVMOP:Projects: Still having problems with my interpolation on tideconvert.py

Sat, 07 Feb 2009 07:13:00 EDT

I went ahead and hacked my tidePlot.py which will do comparisons of time based data (using the timedate class). Using it to visually analyze my interpolation (added as an option to tideconvert), I see that it still needs work. The slope isn't right, and it doesn't catch all gaps in the data (right side of image).

SVMOP:Projects: From and To python datetime objects an UNIX time

Tue, 03 Feb 2009 21:29:00 EDT

from http://seehuhn.de/pages/pdate

Conversion between Unix times and datetime

To convert a Unix time stamp to datetime use the fromtimestamp constructor:

 >>> from datetime import datetime
 >>> datetime.fromtimestamp(1172969203.1)
 datetime.datetime(2007, 3, 4, 0, 46, 43, 100000)

To convert a datetime object into a Unix time stamp, one has to first convert it into a struct_time:

 >>> from datetime import datetime
 >>> from time import mktime
 >>> t=datetime.now()
 >>> mktime(t.timetuple())+1e-6*t.microsecond
 1172970859.472672

My example:


In [26]: d1 = datetime.datetime.now()

In [27]: d1
Out[27]: datetime.datetime(2009, 2, 3, 21, 25, 4, 588232)

In [28]: time.time()
Out[28]: 1233714322.753165

In [29]: time.mktime(d1.timetuple()) + 1e-6 * d1.microsecond
Out[29]: 1233714304.588232

In [30]:

SVMOP:Projects: First task = Interpolating while converting

Tue, 03 Feb 2009 13:51:00 EDT

My simplistic interpolation scheme was to count the number of contiguous bad source data lines and interpolate that many points over the gap. I got what I planned, but more often than not, the bad source spans many more sample points than bad lines caught.

Now it looks as though I need to do a little analysis on the incoming data to find an average sample rate, and find any spans that are above +/- 1 second of that rate. Then interpolate over the gap, and sample the interpolation at the computed sample interval.

SVMOP:Projects: The Current Design of the Tides Processing Package

Fri, 30 Jan 2009 09:31:00 EDT

The current process design is:

Convert from pressure to depth
This process shows some gaps in the data, often small due to hiccups in data transfer or data logging. During this process, the time will be converted to epoch time (seconds from January 1, 1970). The gaps will be filled by interpolation. A byproduct of this stage is the statitistics of missing data and interpreted data.: Boxcar Filter with Sampling
The filter will smooth the raw data, taking out jitter that might be introduced by wind, waves, and wakes. This dataset is more dense than is needed to meet the requirements of NOAA, therefore only a fraction of the filtered data is output. The output has to align with the times of NOAA data. The results of this stage will be used in two different production paths: Offsets and Constituents. A byproduct of this stage is the statistical deviation of the raw data from the filtered data.: The two product paths
Offsets

The stages off working offsets are:

Find peaks

Find Mean Low Low Water

Compare peaks to a primary station

Produce offset and range value for predictions based on the primary station

Constituents

Using least-squares determination of constituents (the tappy.py program). Use the constituents to produce a tide predictor: Comparison
The two prediction methods will be compared to new data as it goes through the first two stages: Conversion and Filtering.

SVMOP:Projects: The Tides Project - the begining

Sun, 25 Jan 2009 13:45:00 EDT

CCOM operates several tide stations. The three that are most likely to be recording data are:

Jackson Estuarine Lab at Fuber Straights, Great Bay, New Hampsire
US Coast Guard Station, Fort Point, New Castle, New Hampshire
Maine Maritime Academy, Castine, Maine

The last of these is generating the most complete data set, but has only been in operation since the summer of 2008.

What we want to achieve is the ability to do predictive tides and compare to actual tides.

I have done this before on the Jackson Lab data using the Perl programming language, but it was for a single evaluation of time offset and amplitude factor with reference to the NOAA Portland primary station (used for the Portsmouth area tides, as well).

Since our prefered programming language is Python, I will probably focus on Python tools, such as tappy.py by Tim Cera of The St. John's River Water Management District. Tappy uses the Least Squares method of finding tidal constituents.

I will probably use Python's matplot for data visualization. There also exist some rather serious math modules for Python, e.g., numpy and scipy. (Both are used by tappy.)

The public result of this work will be web access to data and predictions. I also hope to have some fairly easy-to-use tools for working with tide data, particularly for doing bathymetric surveying.

SVMOP:Projects: VOS Tracking Project Coming to an End

Sun, 25 Jan 2009 13:42:00 EDT

I have been negligent in posting the process of buiding the last bit, the data visualization bit, on this project. But you will be able to read all about it in a forthcoming PowerPoint Presentation that I will post here.

The results are a set of several hundred Google Earth kmz files. You can access the top of the data at Kurt's Developement Web Site.

Now it is time to start the next project: TIDES

SVMOP:Projects: Wouldn't it be nice to just have a color attribute

Sun, 21 Dec 2008 21:52:00 EDT

As pristine an XML implementation as KML is, I feel that giving an object some attributes rather than having to only use separate style objects would make use color so much easier.

I have taken Kurt's first pass on generating KML from the database and added the time field in a timespan attribute to linestrings.

I still have a long way to go to know what elements and attributes fit where in KML.