MAPERL:

SVMOP:VSTP: Wouldn't it be nice to just have a color attribute

Sun, 21 Dec 2008 21:52:00 EDT

As pristine an XML implementation as KML is, I feel that giving an object some attributes rather than having to only use separate style objects would make use color so much easier.

I have taken Kurt's first pass on generating KML from the database and added the time field in a timespan attribute to linestrings.

I still have a long way to go to know what elements and attributes fit where in KML.

SVMOP:VSTP: Where oh where

Mon, 08 Dec 2008 14:19:00 EDT

did my little database go. It has been several months since I worked on the Volutary Ship Observor Tracking project. I can't recall where I put the most recent version. My subversion repository doesn't seem to have the most recent postings. My computer at work doesn't seem to have it.

I definitely need a more structure procedure to assist this failing brain.

Kurt asked that I do the SQL to KML piece this week so that he could take it to a Google conference.

SVMOP:VSTP: Lessons learned (so far) from this project

Fri, 10 Oct 2008 07:13:00 EDT

Since I still am very green as a Python programmer and still think in Perl, I have been learned from pain the differences in concept. For one, the concept and implementation of global variables is different.

In Perl, the concept is that the language is more likely to be used for utilities rather than applications, therefore everythink is global if not otherwise declared. Global to the namespace (module/package) that is. Python, however, assumes everything is private unless otherwise declared. This brings up the question of when is it appropriate to use globals in Python.

I asked Kurt, whom I regard as a seasoned Python programmer, when he used globals. His answer: "never!" Where do I use globals? When variables (usually application configuration settings and things like file pointers/objects) are used in many of the subroutines/functions. Rather than continually passing these variables as argument to the call, I just make them global.

Here is another wierdness abouth Python globals: the calling function declares them as global rather than the block enclosing the functions (as in Perl when they are explicitly declared global).

I'll get used to it.

Other things that take some getting used to are split, join, and string formatting. They are just enough different that I am having to correct myself often.

I all, I really like Python for application development. I still want to use Perl for utility programming, but I know I won't learn Python unless I use it for utilities as well as applications. It is not likely I will completely forget Perl in the process, at least not until so few of my brain neurons are firing that it does make any difference.

SVMOP:VSTP: rsync - doesn't seem to work on the web server

Wed, 08 Oct 2008 06:28:00 EDT

This is a test to see if rsync works on mv.com. I don't think it is installed, so I'm hoping that the rsh/rcp method works. Let's see.

Nope.. but rcp/scp does work. So I'll make a little script or alias to shorten the call.

SVMOP:VSTP: SQLite experiences

Tue, 07 Oct 2008 23:03:00 EDT

Dropping down to pysqlite from SQLalchemy wasn't too difficult. But the last few details took an experienced pysqlite programmer (Kurt Schwehr) to get things working. The issues were:

Be sure that there is a connection.commit() on a regular basis
Be sure that proper parameter format is used in the INSERT

There are five different ways of specifying parameters. I still need to find the one that works the best for me. Some do automatic value casting.

SVMOP:VSTP: RDBMS and SQL -- the next phase for ShipTracking

Mon, 06 Oct 2008 22:09:00 EDT

The VOS (Voluntary Observation Ship) Tracking project has taken on a new phase. The data will be read into a relational database as well as the original XML format.

For the application user, there is little change. The major difference is the specification for the database fields. Compare the SQL output specification to the XML specification:

 <?xml version='1.0' encoding='iso-8859-1'?>
 <config>

  <inputDescr filetype="ASCII">
    <field name="year"		start="1"   	stop="4" 	type="INT"/>
    <field name="month"		start="5"   	stop="6" 	type="INT"/>
    <field name="day"		start="7"   	stop="8" 	type="INT"/>
    <field name="hour"		start="9"   	stop="12" 	type="FLOAT"/>
    <field name="lat"		start="13"  	stop="17" 	type="FLOAT"/>
    <field name="lon"		start="18"   	stop="23" 	type="FLOAT"/>
    <field name="callSign"	start="35"   	stop="44" 	type="STR"/>
  </inputDescr>

  <outputDescr filetype="XML" >
    <field name="year" format="%d" value="year" />
    <field name="month" format="%d" value="month" />
    <field name="day" format="%d" value="day" />
    <field name="hour" format="%0.4d" value="hour" />
    <field name="hrs" format="%.2f" value="float(int( hour /100) + ( hour /100.00 - int( hour /100))/.6)" />
    <field name="latitude" format="%0.2f" value="lat / 100" />
    <field name="longitude" format="%0.2f" value="lon / 100" />
    <field name="callSign" format="%s" value="trim( callSign )" />
  </outputDescr>

  <outputDescr filetype="SQL">
    <field name="sqltime" datatype="TEXT" format="%s" 
	   value = "SQLtimeStr(year,month,day,0)" />
    <field name="latitude" datatype="REAL" format="%0.2f" 
	   value="lat / 100" />
    <field name="longitude" datatype="REAL" format="%0.2f" 
	   value="lon / 100" />
    <field name="wkt_geographic" datatype="TEXT" format="%s"
	   value="wkt(lat,lon)" />
    <field name="callSign" datatype="TEXT" 
	   format="%s" value="trim( callSign )" />
  </outputDescr>

 </config>

Internally, there are some serious additions for the database specification parsing and record insertion. However, the hard part was deciding which Python modules to use and non use. After looking at SQLalchemy, we have decided to keep it simple and just use pysqlite. (SQLite is the database engine we have picked.)

SVMOP:VSTP: Wrapping it up

Mon, 24 Dec 2007 15:46:00 EDT

For Christmas (and the winter) I'm wrapping up the programming of fields2xml.py. The basic use can be understood by running fields2xml.py --help. It has the option of turning off the metadata in the XML output with -s option. Sample code is on cowfish projects/bendev/ShipTracking.

SVMOP:VSTP: First run with full data set

Sat, 22 Dec 2007 23:27:00 EDT

I've downloaded the entire GTS data set to my server at home, and, after adding some more exception handling to the fields2xml.py program, ran the entire data set: once with tab delimited formating to a single file, and then to separate xml files for each file in the data set. That is 83 files. Each file took about 20 seconds to run. The single tab delimited file resulted in 487565 records!

I need some place on the CCOM servers to put this stuff. If it gets down to the crunch before I leave, I'll just make a CD of it.

SVMOP:VSTP: Benchmark for fields2xml.py

Fri, 21 Dec 2007 13:18:00 EDT

After some code cleanup, I ran the program on a full months data (6400 records) of GTS format. It took approximately 15 seconds to produce the XML. This was done on my Toshiba laptop model A-105 with:

model name      : Genuine Intel(R) CPU           T2050  @ 1.60GHz
stepping        : 8
cpu MHz         : 800.000
cache size      : 2048 KB

Not bad

SVMOP:VSTP: It has some smarts.

Fri, 21 Dec 2007 11:39:00 EDT

I have added data types to input specs so that I can do expression evaluation specs for the output. In other words, instead of just making a reference in the output value to an input field (e.g. "lat"), you can have an expression that evaluates (e.g. "lat / 100" "( hour * 60 * 60 ) + ( min * 60 )

sec" ). Notice that all tokens must be delimited by white space for this to work.

The project sits at https://cowfish/projects/bendev/ShipTracking