Sun, 15 Feb 2009

Submitting KML content to Google Earth -- not as easy as Flickr 02/15/2009 20:15
There are a few rather esoteric steps to getting your KML files recognized by the Google Earth search engine. I think I have it set up, but we will see. -ben

[/Projects] permanent link

Tue, 10 Feb 2009

New Design Ideas for Tide Tools 02/10/2009 00:59
Well, I found my bug in my lowpass filter. I had two variables, with very similar names, being used for the same thing. But one wasn't recalculated as it should have been, the result was that I had figured weighting numbers based on, say a window of 15 points, but only summing 11. That filter looks pretty good now. I believe that it was a mistake to put the interpolator inside of the data convertor program. I need to destroy outliers BEFORE I do any interpolation. Also, I'm not convinced that straight-line interpolation will suit our needs. I'm thinking more in the lines of a circular arc fit. So now, the path might be:
  • convert from pressure to depth
  • do data cleaning using a slope corrected standard deviation over a window
  • interpolate and resample (I think I can combine these)
  • smooth (low pass filter)
  • find peaks, MLLW, constituents, etc.
  • compare prediction methods with cleaned but unsmoothed data
  • [/Projects] permanent link

    Sat, 07 Feb 2009

    Don't program when you mind isn't clear. 02/07/2009 21:05
    I finally found my last problem with the interpolation. There were two. But last one was the most vexing. It was my calculation of fp seconds from datetime.timedelta. I had put this off in its only routine and had written: dt.seconds + dt.microseconds * exp(-6). Obvious when you see it: exp is exponent of base e not base 10. The correct expression is: dt.seconds + dt.microseconds * 10**(-6). I really like the Python matplotlib plotting routines and interface for viewing, As you can see, I was able to pinpoint my problems thanks to the ease of this interface and its ability to create images.

    [/Projects] permanent link

    Still having problems with my interpolation on tideconvert.py 02/07/2009 12:13
    I went ahead and hacked my tidePlot.py which will do comparisons of time based data (using the timedate class). Using it to visually analyze my interpolation (added as an option to tideconvert), I see that it still needs work. The slope isn't right, and it doesn't catch all gaps in the data (right side of image).

    [/Projects] permanent link

    Thu, 05 Feb 2009

    Youtube video 02/05/2009 00:11
    I have embedded a youtube video of Mother of Perl in the general pages under Videos.

    [/Tech] permanent link

    Wed, 04 Feb 2009

    From and To python datetime objects an UNIX time 02/04/2009 02:29
    from http://seehuhn.de/pages/pdate

    Conversion between Unix times and datetime

    To convert a Unix time stamp to datetime use the fromtimestamp constructor:
     >>> from datetime import datetime
     >>> datetime.fromtimestamp(1172969203.1)
     datetime.datetime(2007, 3, 4, 0, 46, 43, 100000)
    
    To convert a datetime object into a Unix time stamp, one has to first convert it into a struct_time:
     >>> from datetime import datetime
     >>> from time import mktime
     >>> t=datetime.now()
     >>> mktime(t.timetuple())+1e-6*t.microsecond
     1172970859.472672
    
    My example:
    
    In [26]: d1 = datetime.datetime.now()
    
    In [27]: d1
    Out[27]: datetime.datetime(2009, 2, 3, 21, 25, 4, 588232)
    
    In [28]: time.time()
    Out[28]: 1233714322.753165
    
    In [29]: time.mktime(d1.timetuple()) + 1e-6 * d1.microsecond
    Out[29]: 1233714304.588232
    
    In [30]: 
    

    [/Projects] permanent link

    Tue, 03 Feb 2009

    First task = Interpolating while converting 02/03/2009 18:51
    My simplistic interpolation scheme was to count the number of contiguous bad source data lines and interpolate that many points over the gap. I got what I planned, but more often than not, the bad source spans many more sample points than bad lines caught. Now it looks as though I need to do a little analysis on the incoming data to find an average sample rate, and find any spans that are above +/- 1 second of that rate. Then interpolate over the gap, and sample the interpolation at the computed sample interval.

    [/Projects] permanent link

    Fri, 30 Jan 2009

    The Current Design of the Tides Processing Package 01/30/2009 14:31
    The current process design is:
    Convert from pressure to depth
    This process shows some gaps in the data, often small due to hiccups in data transfer or data logging. During this process, the time will be converted to epoch time (seconds from January 1, 1970). The gaps will be filled by interpolation. A byproduct of this stage is the statitistics of missing data and interpreted data.
    Boxcar Filter with Sampling
    The filter will smooth the raw data, taking out jitter that might be introduced by wind, waves, and wakes. This dataset is more dense than is needed to meet the requirements of NOAA, therefore only a fraction of the filtered data is output. The output has to align with the times of NOAA data. The results of this stage will be used in two different production paths: Offsets and Constituents. A byproduct of this stage is the statistical deviation of the raw data from the filtered data.
    The two product paths
    Offsets
    The stages off working offsets are:
    1. Find peaks
    2. Find Mean Low Low Water
    3. Compare peaks to a primary station
    4. Produce offset and range value for predictions based on the primary station
    Constituents
    Using least-squares determination of constituents (the tappy.py program). Use the constituents to produce a tide predictor
    Comparison
    The two prediction methods will be compared to new data as it goes through the first two stages: Conversion and Filtering.

    [/Projects] permanent link

    She Sits on Her Mooring 01/30/2009 08:18
    Well, Mother of Perl is happy sitting on her mooring off the New Castle pier.

    [/Sail] permanent link

    Sun, 25 Jan 2009

    The Tides Project - the begining 01/25/2009 18:45
    CCOM operates several tide stations. The three that are most likely to be recording data are:
    • Jackson Estuarine Lab at Fuber Straights, Great Bay, New Hampsire
    • US Coast Guard Station, Fort Point, New Castle, New Hampshire
    • Maine Maritime Academy, Castine, Maine
    The last of these is generating the most complete data set, but has only been in operation since the summer of 2008. What we want to achieve is the ability to do predictive tides and compare to actual tides. I have done this before on the Jackson Lab data using the Perl programming language, but it was for a single evaluation of time offset and amplitude factor with reference to the NOAA Portland primary station (used for the Portsmouth area tides, as well). Since our prefered programming language is Python, I will probably focus on Python tools, such as tappy.py by Tim Cera of The St. John's River Water Management District. Tappy uses the Least Squares method of finding tidal constituents. I will probably use Python's matplot for data visualization. There also exist some rather serious math modules for Python, e.g., numpy and scipy. (Both are used by tappy.) The public result of this work will be web access to data and predictions. I also hope to have some fairly easy-to-use tools for working with tide data, particularly for doing bathymetric surveying.

    [/Projects] permanent link