Journal: VSTP

Since I still am very green as a Python programmer and still think in Perl, I have been learned from pain the differences in concept. For one, the concept and implementation of global variables is different.

In Perl, the concept is that the language is more likely to be used for utilities rather than applications, therefore everythink is global if not otherwise declared. Global to the namespace (module/package) that is. Python, however, assumes everything is private unless otherwise declared. This brings up the question of when is it appropriate to use globals in Python.

I asked Kurt, whom I regard as a seasoned Python programmer, when he used globals. His answer: "never!" Where do I use globals? When variables (usually application configuration settings and things like file pointers/objects) are used in many of the subroutines/functions. Rather than continually passing these variables as argument to the call, I just make them global.

Here is another wierdness abouth Python globals: the calling function declares them as global rather than the block enclosing the functions (as in Perl when they are explicitly declared global).

I'll get used to it.

Other things that take some getting used to are split, join, and string formatting. They are just enough different that I am having to correct myself often.

I all, I really like Python for application development. I still want to use Perl for utility programming, but I know I won't learn Python unless I use it for utilities as well as applications. It is not likely I will completely forget Perl in the process, at least not until so few of my brain neurons are firing that it does make any difference.

[/VSTP] permanent link

[/VSTP] permanent link

• Be sure that there is a connection.commit() on a regular basis
• Be sure that proper parameter format is used in the INSERT

[/VSTP] permanent link

 <?xml version='1.0' encoding='iso-8859-1'?>
 <config>

  <inputDescr filetype="ASCII">
    <field name="year"		start="1"   	stop="4" 	type="INT"/>
    <field name="month"		start="5"   	stop="6" 	type="INT"/>
    <field name="day"		start="7"   	stop="8" 	type="INT"/>
    <field name="hour"		start="9"   	stop="12" 	type="FLOAT"/>
    <field name="lat"		start="13"  	stop="17" 	type="FLOAT"/>
    <field name="lon"		start="18"   	stop="23" 	type="FLOAT"/>
    <field name="callSign"	start="35"   	stop="44" 	type="STR"/>
  </inputDescr>

  <outputDescr filetype="XML" >
    <field name="year" format="%d" value="year" />
    <field name="month" format="%d" value="month" />
    <field name="day" format="%d" value="day" />
    <field name="hour" format="%0.4d" value="hour" />
    <field name="hrs" format="%.2f" value="float(int( hour /100) + ( hour /100.00 - int( hour /100))/.6)" />
    <field name="latitude" format="%0.2f" value="lat / 100" />
    <field name="longitude" format="%0.2f" value="lon / 100" />
    <field name="callSign" format="%s" value="trim( callSign )" />
  </outputDescr>

  <outputDescr filetype="SQL">
    <field name="sqltime" datatype="TEXT" format="%s" 
	   value = "SQLtimeStr(year,month,day,0)" />
    <field name="latitude" datatype="REAL" format="%0.2f" 
	   value="lat / 100" />
    <field name="longitude" datatype="REAL" format="%0.2f" 
	   value="lon / 100" />
    <field name="wkt_geographic" datatype="TEXT" format="%s"
	   value="wkt(lat,lon)" />
    <field name="callSign" datatype="TEXT" 
	   format="%s" value="trim( callSign )" />
  </outputDescr>

 </config>

[/VSTP] permanent link

For Christmas (and the winter) I'm wrapping up the programming of fields2xml.py. The basic use can be understood by running fields2xml.py --help. It has the option of turning off the metadata in the XML output with -s option. Sample code is on cowfish projects/bendev/ShipTracking.

[/VSTP] permanent link

I've downloaded the entire GTS data set to my server at home, and, after adding some more exception handling to the fields2xml.py program, ran the entire data set: once with tab delimited formating to a single file, and then to separate xml files for each file in the data set. That is 83 files. Each file took about 20 seconds to run. The single tab delimited file resulted in 487565 records!

I need some place on the CCOM servers to put this stuff. If it gets down to the crunch before I leave, I'll just make a CD of it.

[/VSTP] permanent link

After some code cleanup, I ran the program on a full months data (6400 records) of GTS format. It took approximately 15 seconds to produce the XML. This was done on my Toshiba laptop model A-105 with:

model name      : Genuine Intel(R) CPU           T2050  @ 1.60GHz
stepping        : 8
cpu MHz         : 800.000
cache size      : 2048 KB

Not bad

[/VSTP] permanent link

I have added data types to input specs so that I can do expression evaluation specs for the output. In other words, instead of just making a reference in the output value to an input field (e.g. "lat"), you can have an expression that evaluates (e.g. "lat / 100" "( hour * 60 * 60 ) + ( min * 60 ) + sec" ). Notice that all tokens must be delimited by white space for this to work.

The project sits at https://cowfish/projects/bendev/ShipTracking

[/VSTP] permanent link

I have a working program for generating XML (or tab delimited, or ASCII) from ASCII records. I'm trying to find a place that is easy to retrieve this from. I will put it on the Subversion site (cowfish:Projects/ShipTracking) but need to give read access to all and read/write to Kurt. This needs Will F.'s assistance.

Now, I need to get it to align with useful data fields in the VOS ASCII data. This just means editing the configuration file until it works. This is a bit better than awk.

[/VSTP] permanent link

After having had a code review of my python data to xml program for the VOS ship tracking, I've gone back and simplified my "elegant" program. It didn't feel right.. trival things were layers deep in objects and methods. The XML handling was painful. It would have been easier to just parse the XML configuration file by hand to generate the structures I wanted. As it turns out, most of the structures, classes, and methods were not necessary. What do you expect for only my third python program that I want, so much, to be appreciated by my peers (I flatter myself with whom I call "peers").

I've gone back to a more proceedural program.. less layers, less classes, less structures. I give credit to Brian L. and Roland A. for getting things back to a manageable level. But I give even more credit to Kurt S., and his persistance on my using lxml. It really feels good and simple.

[/VSTP] permanent link