Fun with Shapefiles, CRSs and GeoTools

Although I’m now in the “GIS business” for years I had never to deal with shapefiles directly. Now it was time also to investigate tools like QGIS and hack together a simple reader for shp files. At least I thought it was simple but calling me a GIS expert afterwards would be a ridiculous understatement.

GeoTools fun

A quick look and I decided to go with GeoTools as I knew it from name and I needed a tool in Java. Thanks to QGIS I understood quickly that in my case I had to deal with a list of a list of lines containing coordinates but how to read that via GeoTools? The internet provided several solutions, but I didn’t found complete examples for my case. As it turned out: I had to explicitly cast 2 times (!) first from “Feature” to “SimpleFeature” and then from “Geometry” to “MultiLineString”. Not sure if this is really necessary. At least this makes learning a new API very hard.

Now I had the initial code:

Map connect = new HashMap();
// a File is not sufficient as a shapefile consists of multiple files
connect.put("url", file.toURI().toURL());
DataStore dataStore = DataStoreFinder.getDataStore(connect);
String[] typeNames = dataStore.getTypeNames();
String typeName = typeNames[0];
FeatureSource featureSource = dataStore.getFeatureSource(typeName);
CoordinateReferenceSystem sourceCRS = featureSource.getSchema().getCoordinateReferenceSystem();
FeatureCollection collection = featureSource.getFeatures();
// allow for some error due to different datums ('bursa wolf parameters required')
boolean lenient = true;
MathTransform transform = CRS.findMathTransform(sourceCRS, targetCRS, lenient);

List<List<GPXEntry>> lineList = new ArrayList<>();
try (FeatureIterator iterator = collection.features()) {
    while (iterator.hasNext()) {
        SimpleFeature feature = (SimpleFeature) iterator.next();
        MultiLineString mlString = (MultiLineString) feature.getDefaultGeometry();
        ...
    }
}

How short and beautiful. But: It did not compile. And that although I was using the recommended “maven procedure”. It seems that GeoTools seems to follow a bit unusual path that it requires you to define the repositories in your pom.xml – I did only find a solution with the snapshot versions but this was sufficient for the time being.

CRS fun

At least it seemed to work then. But after further longish time I found out that the coordinates had just a tiny offset, so something was wrong with the source or target coordinate reference system (CRS) or with the transformation itself. Again QGIS helped me here and determined the source CRS correctly. But GeoTools was somehow wrong and initially I thought it was GeoTools fault.

But I quickly stumbled over another CRS issue and had to deal with exactly the same CRSs leading to different results. In my case it was CRS.decode(“EPSG:4326”) vs. DefaultGeographicCRS.WGS84 – so they are identical but the results were different!? It turns out that the coordinate axes are mixed! GeoTools fault? No! GeoTools even gave me the solution in its documentation:
“So if you see some data in “EPSG:4326” you have no idea if it is in x/y order or in y/x order”!

Deployment

Puh. Okay. I was ready for deployment and used my usual git and mvn assembly procedure to push stuff on my server but then I got exceptions while runtime about missing classes! Oh no – how can this be when I use maven?
As it turns out GeoTools requires the maven shade plugin in order to bundle the database for correct CRS transformation properly via a plugin architecture I think. And look: the whole jar is now nearly 12MB!

Conclusion

The GIS and Java world are called “enterprise” for a reason. I hope I can help others with my findings. Find the fully working code here.

Problems with nvidia driver 173 and kernel 3.2.0-52. Two monitors and card Quadro NVS 160M

After the recent kernel upgrade my old nvidia driver failed to load and X loads with a very low resolution and without detecting my second monitor. For my graphic card the nouveau driver is not an option. But I managed to fix the problems via removing current nvidia and nvidia-173. But probably you can just skip those commands and use the jockey-gtk as explained below.

sudo apt-get purge nvidia-current
sudo apt-get remove nvidia-173
sudo apt-get install –reinstall nvidia-173-updates
# now REBOOT!

Now it boots as normal with my second monitor but the nice configuration tool called ‘nvidia-settings’ did not work and says ‘The NVIDIA X driver on is not new enough to support the nvidia-settings Display Configuration page.’

Also my firefox 23 had performance problems for canvas rendering. I solved this via opening “about:config” and set layers.acceleration.force-enabled to true. I found this in the comments of this article.

Update: I was able to fix the all the issues just by switching to ‘(additional updates) (version 319-updates)’ when starting the tool called ‘jockey-gtk’! Afterwards also the tool xfce4-display-settings showed two monitors.

Java on iPhone or iPad

What options do I have to make my Java application working on the iPhone? It should not be necessary to jailbreak the phone. Also UI is not necessary for now although e.g. codenameone seems to support it. My application it not a complicated one, but uses Java 1.5 (generics etc) and e.g. memory mapped files. The good thing would be if the JUnit tests could be transformed too with the tool to check functionality.What I found is

Or somehow use the embedded JVM for ARM directly? Do you have experiences with this or more suggestions?

Rage Against the Android – Nearly solved! #eclipse #netbeans

After ranting against Android in my previous post I have mainly solved now the untestable situation Android is producing.

Thanks to Robolectric all my tests are finally fast and do not unpredictable hang or similar!

The tests execute nearly as fast as in a normal Maven project – plus additional 2 seconds to bootstrap (dex?). Robolectric was initially made for maven integration on IntelliJ. But it works without Maven and under Eclipse as described here. I didn’t get robolectric working via Maven under Eclipse – and it was overly complex e.g. you’ll need an android maven bridge but when you use a normal Android project and a separate Java projects for the tests then it works surprisingly well for Eclipse.

Things you should keep in mind for Eclipse:

  • Eclipse and Robolectric description here
  • In both projects put local.properties with sdk.dir=/path/to/android-sdk
  • Android dependencies should come last for the test project as well as in the pom.xml for the maven integration (see below)

But now even better: when you are doing your setup via Maven you can use NetBeans! Without any plugin – so even in the latest NetBeans version. But how is the setup done for Maven? Read this! After this you only need to open the project with NetBeans (or IntelliJ).

Things you should keep in mind for Maven and NetBeans:

  • Use Maven >= 3.0.3
  • Again: put the android dependencies last!
  • Specify the sdk path via bashrc, pom.xml or put it into your settings.xml ! Sometimes this was not sufficient in NetBeans – then I used ANDROID_HOME=xy in my actions
  • Make sure “compile on save” is disabled for tests too, as I got sometimes strange exceptions
  • deployment to device can be done to:
    mvn -DskipTests=true clean install      # produce apk in target or use -Dandroid.file=myApp.apk in the next step
    mvn -Dandroid.device=usb android:deploy # use it
    mvn android:run
  • If you don’t have a the Android plugin for NetBeans just use ‘adb logcat | grep something’ for your UI to the logs

Thats it. Back to hacking and Android TDD!

3D Rotation in Gimp

  1. Erstelle eine zusätzliche transparente Ebene
  2. Selektiere nun die Ebene die 3D rotiert werden soll
  3. Gehe zu Filter->Abbilden->Auf Objekt Abbilden
  4. Wähle auf Quader abbilden
  5. Klicke ‘transparenter Hintergrund’
  6. Gehe zu Tab ‘Quader’. Eine Seite bekommt Ebene aus dem 1. Schritt. Alle anderen bekommen die Ebene aus dem 2. Schritt.
  7. Gehe zu Ausrichtung->Rotation und verändere wie gewünscht

 

Heaven of Mergurial

The previous post about merging with named branches in mercurial was reposted at dzone. Daniel Neugebauer commented there that he wouldn’t use the named branch feature for every small issue instead he suggested to do the merging directly within the same repository and for bigger changes he would use a ‘clone per issue’. Although Fabrizio Giudici sayed that ‘branch per issue’ works really well in practise. Get his introduction!

And here I’ll quote python.org which compares clone vs. branch per issue:

Mercurial has two basic ways of using branches: cloned branches, where each branch is kept in a separate repository, and named branches, where each revision keeps metadata to note on which branch it belongs. The former makes it easier to distinguish branches, at the expense of requiring more disk space on the client. The latter makes it a little easier to switch between branches, but often has somewhat unintuitive results for people (though this has been getting better in recent versions of Mercurial).

The current proposal is to use named branches for release branches and adopt cloned branches for feature branches […]

Differences between named branches and cloned branches:

  • Tags in a different (maintenance) clone aren’t available in the local clone
  • Clones with named branches will be larger, since they contain more data

The Mercurial book discourages the use of named branches, but it is, in this respect, somewhat outdated. Named branches have gotten much easier to use since that comment was written, due to improvements in hg.

So, think what you want 😉

Here are the steps to do a merge from a cloned repository (clone per issue). Which is the recommended stragety for bigger changes. This merging-strategy is also covered in the official hg book.

  1. The working directory is:
    app/

    Now commit the last changes.

  2. cd ..;hg clone -r <oldrevisionnumber> app/ app-clone/
  3. cd app-clone

    DO CHANGES (add, edit, rename, delete files)

  4. hg commit -m "FIXED issueXY"
  5. cd ../app;
    hg pull ../app-clone

    Now your working copy with the folder app/ contains 2 heads. This can be fixed easily via:

  6. hg merge
  7. hg commit -m "merged fix of issueXY into development sources"

Although this strategy needs one or two more commands it is also very clear and logically. Here is a tip from Daniel:

You can also search the history by “hg log -k <keyword>”, so if you use prefixed commit messages like “ISSUE123: …” you could find changesets belonging to that issue almost as easy as using a named branch.

And finally here are the steps to do a merge directly within the repository (direct merge), recommended for smaller changes only:

  1. Working in you current repository at
    app/

    and commit the last changes

  2. hg update -C <oldrevisionnumber>
  3. DO CHANGES (add, edit, rename, delete files)
  4. hg commit -m "FIXED issueXY"

    now hg says something about that a new head was created. Try

    hg heads

    to see that the previous commit created a new head in the repository. A

    hg update

    would fail now. But the following command resolves that ‘problem’ easily:

  5. hg merge

    The working directory will now contain changes of both commits/heads!

  6. Do not forget:
    hg commit -m "merged fix of issueXY into development sources"

    Now the command

    hg heads

    only contains one head

So only 5 easy steps for a quick merge!

Conclusion

All three strategies: “direct merge”, “clone per issue” and “branch per issue” are logically and easy to remember. Nice! Personally I prefer named branches. That page and this question could show its (dis)advantages.

BTW: for people looking for a git vs. hg site. Here are is one link.

Liquibase + Hibernate (annotations): Easy and solid Database Migration

As I pointed out earlier liquibase is a stable and nice migration tool for SQL databases in the Java land. The problem I had is that I couldn’t get it working with hibernate while using annotations.

Now just for my personal memory here are the steps to get it working. Download liquibase 1.9.5 (couldn’t get it working with 2.0.0 :-() and put the following libs in the liquibase/lib folder:

dom4j-1.6.1.jar
h2-1.2.137.jar #or your prefered jdbc database driver
hibernate-annotations-3.5.1-Final.jar
hibernate-commons-annotations-3.2.0.Final.jar
hibernate-core-3.5.1-Final.jar
hibernate-jpa-2.0-api-1.0.0.Final.jar
slf4j-api-1.6.0.jar

You will need to put the 4 hibernate jars into the classpath parameter, too. To do a diff and update the changelog file via the command line do the following:

liquibase --logLevel=FINE \
 --driver=org.h2.Driver \
 --classpath=$CP \
 --url=jdbc:h2:~/.myapp/h2db \
 --username=sa \
 --password=tmp \
 --changeLogFile=src/main/resources/dbchangelog.xml \
 diffChangeLog \
 --baseUrl=hibernate:src/main/resources/hibernate.cfg.xml

This means you compare the ‘old’ database with the new hibernate config. If you have problems while set-up you can look directly into the source file.

BTW: here is the pom snippet for the hibernate deps:

<dependency>
   <groupId>org.hibernate</groupId>
   <artifactId>hibernate-core</artifactId>
   <version>3.5.1-Final</version>
 </dependency>
 <dependency>
   <groupId>org.hibernate</groupId>
   <artifactId>hibernate-annotations</artifactId>
   <version>3.5.1-Final</version>
 </dependency>