Profile J2SE Maven Projects With Yourkit Profiler Under NetBeans 6.7

For my open source timetabler TimeFinder I got a free license of the Java profiler from yourkit.

TimeFinder is a maven project so it starts not out of the box with the latest yk-NetBeans plugin.

But the solution is simple:

  1. Right click the project->Set Configuration->Customize->Add… then enter a name (profile-algo) for the new configuration
  2. Go to the Run entry in this Project Properties dialog and select the Main Class where your mavenized J2SE application should start with
    nb-config-run
  3. Under VM Options I entered (for linux)
    -agentpath:/home/user/Programme/yourkit-profiler/bin/linux-x86-32/libyjpagent.so
    for another OS please look here.
  4. Then select the previously created configuration (profile-algo) in the tool bar under the menu bar and hit F6 to start the mavenized J2SE application
    nb-config
  5. At any time start the yk-profiler (before the J2SE application starts). Now a new entry under Monitor Local Applications should be listed. Click on that and you will be able to “look into” (aka “profile”) a mavenized J2SE application.
    yourkit-overview
    yk-profiler-at-runtime
  6. If you installed the NetBeans 6.7 plugin before you are even able to click on a source file in yk-profiler and this file will be opened automagically in NetBeans

NetBeans 6.7 finally released

Since today the final version of NetBeans 6.7 can be downloaded here. And do not forget to look here for a feature overview.

If you tried the release candidates of 6.7 you will welcome all the new features! Great work! Where can we donate?

The first impressions of this release for me as a NetBeans user are:

  • seems to use less memory
  • imported the settings of 6.5 with no problems
  • you can now simply install plugins of a previous installation. this is especially useful if the plugin is not available from the update center.
  • maven is now integrated (no need to install it as plugin)
  • you can view maven dependencies graphically (like eclipse already had). See below. And: there is a new button “Build with dependencies” in the right-click-on-project-menu.
  • interrupting maven execution is now possible
  • now test cases execute a lot faster for mavenized projects
  • executing one test via CTRL+F6 instead SHIFT+F6, so that you can execute the corresponding test even if only the java class file is open. (CTRL+SHIFT+F6 for debug test file)
  • all tests (java, ruby, …) happen in the same tab and several search tabs are possible now
  • if you open a resource with ALT+SHIFT+O a case insensitive string now works a friend told me that this was already the case for him earlier than 6.7 … hhmhh …
  • Synchronize Editor With Views -> “The project explorer will then snap to the file currently being edited” (taken from here)
  • some small issues remains:
    • cannot open the source file if you click on the failed test output (and only if you execute the tests of one project)
    • if you miss a static import e.g. import static org.junit.Assert.*; the auto fixing does not add this.

Unit tests

nb6.7-unit-tests

Maven dependencies

nb6.7-mvn-dependency-graph

But the cool thing here is that clicking e.g. on spring core shows you all the possible paths to it and grays out all the unimportant

nb6.7-mvn-dependency-graph2

see some comments on JavaLobby

NetBeans = Best Maven IDE?

Today I want show what is possible with Maven 2.0.9 in NetBeans 6.5 and what makes me nervous.

First of all you have to download NetBeans and then the Maven plugin: Click Tools->Plugins->Available Plugins.

I recommend to install the command line of maven and then set the ‘connection’ to the local repository in NetBeans: Tools->Options->Miscellaneous->Maven->Local Repository should be sth. like /home/user/.m2/repository

  • Add a new jar to the project via right click or directly in the pom.xml: there you will have code completion! I.e. you can always choose the latest version of a library or easily switch versions.
  • Add javadoc or sources (download all source)

    add-sources-and-javadoc

    download-sources

  • You can search unknown class in repository

    search-unknown-symbol

    search-in-repositories

    after adding the JodaTime library with this dialog and adding the sources with the first dialog – all is fine:

    auto-added-lib

  • Set different goal to F6: right click the project->properties->actions->run project and then clear all the fields and set the ‘execute goal’ e.g. to
    mvn jetty:run
  • In the same dialog you can set skip test to true if this is necessary or if you want to speed up compilation
  • Profiling and debugging of a maven project works now, although I had some problems under vista with profiling, but under linux were no issues

So I am nearly happen with maven under NetBeans, but what really sucks is maven while compiling. A small native NetBeans project takes under 2 seconds to clean and compile. And with maven? Nearly 10 seconds!
How could one change this? I only found a workaround to use the same project as native NetBeans project, but nothing more. Any ideas how to make ‘maven install’ or ‘mvn compile’ faster??

(BTW: Today wordpress does not want that I make clickable images … if you want to see the images in better quality you have to right click them->view->remove the stuff after ‘?’. Example: https://karussell.files.wordpress.com/2009/02/add-sources-and-javadoc.jpg)

2 Scripts to Make JavaFX + NetBeans Available on Linux

All the code comes from here and here. It worked for me, although I had to ignore 2 exceptions after this.

Update: JavaFX 1.0 only! See the first comment!

So first run:

mkdir javafxnbms
cd javafxnbms
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-api-debugger-javafx.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-debugger-javafx-ant.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-debugger-javafx-projects.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-debugger-javafx-ui.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-debugger-javafx.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-javafx-bestpractices.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-javafx-editor.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-javafx-fxd.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-javafx-lexer.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-javafx-lib.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-javafx-navigation.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-javafx-palette.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-javafx-platform.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-javafx-profiler.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-javafx-project.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-javafx-sdk-mac.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-javafx-sdksamples.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-javafx-source.nbm
wget http://updates.netbeans.org/netbeans/updates/6.5/uc/final/stable/modules/javafx/org-netbeans-modules-javafx-userlib.nbm

Then install the nbms via NetBeans: Tools -> Plugins -> Downloaded -> Add Plugins … and shutdown NetBeans (ignore the exception).

As last step run the following within the javafxnbnms folder:

mkdir unzipsdk
cd unzipsdk
unzip ../org-netbeans-modules-javafx-sdk-mac.nbm
cp -r netbeans/* ~/Programme/netbeans-6.5/javafx2/
chmod 755 ~/Programme/netbeans-6.5/javafx2/javafx-sdk1.0/bin/*

Restart NetBeans (ignore exception, disable the module -> it even works then!) and you should be able to create a JavaFX sample, run it, view it and change it.

Now you are able to help on timefinder 😉

No Warranty for that procedure!!

Skip Executing the Tests of Mavenized Projects

In my project I want to skip that the tests are running on every execution. I know this is not ‘good style’, but I want to execute the tests when I want. I do not recommend this technic on you build server …

One way to achieve this behaviour is to disable the tests in the

<build><plugins>

</plugins><build>

section via

<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.4.3</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>

But now even if we want to run the tests explicitly they will be skipped. Is there a solution for this?

I found a better and very simply solution for NetBeans. Right click the mavenized project in NetBeans, then click on Actions. Now click on an entry; e.g. Run project and select “Skip tests”. This cheat you can apply on all the listed actions (even customized one …).

Do you have a more IDE-independend solution for this?

My stony way to find a ruby on the rails

Ruby is a great language. All can be expressed so short and readable. That’s one side.

The other side is: you have to learn a lot of exceptions to common stuff that you already learned in another languages (In my case Java).

So today I will explain things where it took me a lot of time to understand how to do this and what ruby (and rails) does. Let us start with the very basics: installation and then the pure ruby language.

Setup MySql

If you use debian as OS it is quite simple to install mysql:

sudo apt-get install mysql-server

Then create a database with the following lines

mysql -u root -p
type the following in the mysql shell:

CREATE DATABASE mydb;
GRANT ALL PRIVILEGES ON mydb.* TO 'admin'@'localhost' IDENTIFIED BY 'admin' WITH GRANT OPTION;
FLUSH PRIVILEGES;
quit;

Later on you can log into the mysql database via
mysql -u admin -p
[now type the password ‘admin’]

SHOW TABLES FROM mydb;

I suggest you to use http://www.squirrelsql.org/ to view and even manipulate the database. The nice thing about this java program is: you can simply install the jdbc drivers for your database (e.g sqlite or mysql) and after configuring them you can browse the databases in a nice way!

Install Ruby and Rails

To be independend from Java or if you don’t want to use JRuby you have to install ruby on your machine:

sudo apt-get install ruby libzlib-ruby rdoc irb

Now install gems …

sudo apt-get install rubygems
sudo gem install mysql

… and rails

sudo apt-get install rails

Create a Skeleton Application

Optionally install NetBeans 6.1 (and Java >1.5 before) to get some handy functionality.

Be sure that you choose a package with rails or install the rails plugins via: Tools->Plugins.

Now you should able to create a rails application: New Project->Ruby On Rails Application. Click Next and choose your ruby system. (Or do ‘rails testapp’ on the command line).

You will get a list of created files. To quickly navigate to a file in NetBeans press ALT+SHIFT+O (Oh not zero) and type in the name. Now I will go through some of the generated files:

app/controllers/application.rb

This is the base class for all of our controllers. To study the Model-View-Controller pattern visit wikipedia. In our controllers we put in all of our application logic. For example the password encryption when one of our user logs in.

app/helpers/application_helper.rb

In this helper modul we define some convinient methods for our view

test/test_helper.rb

With this class it is easy to test the implemented (helper) methods. Read “Agile Web Development with Rails” to be surprised that unit and even integration tests are very easy (and fast) to write.
config/database.yml
The generated content looks like:
development:
# your database server:
adapter: mysql
encoding: utf8
database: mydb
username: admin
password: admin
timeout: 5000
# necessary if you are on debian:
socket: /var/run/mysqld/mysqld.sock

config/routes.rb
Here you will define later more fine grained navigation rules for the pages of your web application.

In NetBeans (with rails of course) database-migrating is very easy: right click the project->Migrate Database->To [misc] Version or Clear…

Start the web server with F6 in NetBeans or type

ruby script\server

Then go to http://localhost:3000/ with your browser and you should see ruby’s welcome page.

Now we want to replace the index.html with an erb (like jsp) file. To do this we will remove the public/index.html file and do

script/generate controller home index

add the following to the routes.rb

map.home '', :controller => 'home', :action => 'index'

To add some more functionality to our rails application there are a lot of plugins available. One of them is the restful_authentication, where a tutorial plugin (or here) exists which itself references to a bunch of other useful plugins. Check them out!.

Now some comments on setting up the restful_authentication plugin: Add the following resource to your repositories if you need this plugin (and others from the author):

script/plugin source http://svn.techno-weenie.net/projects/plugins/script/plugin install restful_authentication

or simply do

script/plugin install http://svn.techno-weenie.net/projects/plugins/restful_authentication/

On my machine it took a lot of time and finished with an timeout error so I did:

sudo apt-get install git-corecd vendor/plugin && git clone git://github.com/activefx/restful_authentication.gitmv restful-authentication/ restful_authentication/ && cd ../..

Now generate the necessary classes:

script/generate authenticated user sessions --include-activation --stateful

My (unsorted) Comments

Try getting started with rails and come back here to read the following comments (or post your own!) if you figured out sth. worth enough to be said for beginners.

  1. For restful_authentication:
    Be sure that you ignore the config/initializers/site_keys.rb in your subversion!! But save it, because you will need it for your application!
  2. If need a login snippet like the one from the authentication tutorial:
    app/views/users/_user_bar.html.erb in your start up weü page app/views/home/index.html.erb
    Use the following line:
    <%= render(:partial => “users/user_bar”) %>
  3. If you need ssh2 try:
    # require ‘digest/sha1’
    require ‘digest/sha2’
    def User.mydigest(str)
    Digest::SHA256.digest(str)
    #Digest::SHA1.hexdigest(str)
    end
  4. What the hell is this string with a colon? I mean the ‘:password’ here:
    update_attributes :password => ‘secure’
    It is called symbol and is the name of a variable e.g. of a class. Where ‘secure’ is the value of the variable password. So with this method call e.g. within the User class you do password = ‘secure’.
    Example:
    var = { ‘a’ => ‘b’, 1 => ‘c’}
    assert_equals(‘b’, var[‘a’])
  5. And what is the :password => ‘secure’ statement?
    This is a key => value expression without the {} braces of the hash (Map in Java) , which are optional if the hash is the last parameter. This is very handy if you have several properties to set.
  6. What is ‘self’? It depends! Read more about self here!
    • global variables begins with @@
    • object variables begins with an @ (All the instance methods of a class can access this variable. So every object has its own set of object variables.)
    • local variable begins with a lower case character (except ‘self’)
    • constant variables begins with an upper case character (except ‘:symbol’)

    Warning: if you don’t add self in front of you object variables, ruby assumes that they are local!?

  7. What is attr_accessor :name? This will create a getter and a setter for the variable ‘name’.
    Or define a setter manually (via operator overloading)
    def volume=(vol)
    leftChannel = self.rightChannel = vol
    end
  8. Where is the constructor of User? Do
    user = User.new
  9. escape html in the view via h(user_input).
    For more security related stuff look here, here and here.
  10. The following line is not always secure (‘mass assignment’)
    User.new(params[:user])
    To make it secure do:
    attr_accessible :name
    Now only the variable name can be change via url parameters.
  11. The following statement is unsecure
    User.find(:all, :conditions => [“name like ‘%#{params[:query]}%'”])
    Use
    :conditions => “name like ?”, ‘%’ + params[:query] + ‘%’
  12. add filter_parameter_logging :password to application.rb
    => all fields with names like “password” will get [FILTERED] in the logs
  13. Rails add a very handy method to objects:
    o.blank?
    An object is blank if it‘s false, empty, or a whitespace string. For example, “”, ” “, nil, [], and {} are blank. Taken from here.
  14. There is no counter++
    Use counter += 1
  15. Swapping variables:
    a, b = b, a
    This work in methods too! E.g. return a pair of variables via
    def meth
    return a, b
    end
    Now call the method
    b, a = meth
  16. if sugar – ‘the other wayround’
    puts ‘hello world’ if str == ‘name’
  17. defined? var
    will return true if there is a field named var
  18. You can use ‘hello’ or “hello”
    The only difference for those strings is the expression evaluation like “#{var}”
    This will work only in “”
  19. comments are like in bash: #
    Or use this to comment several lines:
    =begin
    here is comment
    here too. WARING: =begin and =end has to be the first characters on the lines!
    =end
  20. Default values for method parameters via
    def meth(param=’test’)
    end
  21. def see_how_to_throw
    begin
    # Do something nifty
    raise SomeError, “This is the error message!”
    rescue SomeError
    # This is executed when a SomeError exception is raised
    # to rethrow do
    # raise $!
    rescue AnotherError => error
    # Here, the exception object is referenced from the
    # `error’ variable
    else
    # This is executed only if no exceptions were raised
    ensure
    # This is always executed, exception or not
    end
    end
  22. A very useful tool for testing is autotest:
    gem install ZenTest
  23. It is eays to test with rails! Very easy to write unit and functional testing of your controllers and the pages.
    In NetBeans press ALT+F6 to start them.
  24. use %{ } for xml strings directly unescaped in the code! Very handy.
  25. Time sugar:
    3.minutes + 3.hours
    returns the result in seconds! Much more …
  26. counter ||= 0
    This statement will only be executed if counter is nil. Useful for singletons you think? Use self for this!
  27. Unicode support is not that good. Use Chars instead of the built in String class. Or convert to Chars before working with strings.
  28. Why are there named routes in routes.rb?
    These routes are accessible from the view. E.g. the following line is a named route (map.hello instead of map.resource)
    map.hello ‘/fhello’, :controller => ‘users’, :action => ‘hello’
    So the method hello will be invoke if the user click the link in the view which is generated via <%=hello_url%>
    One special name ‘map.resources’ indicates that rails should create 7 standard routes (4 of them are the CRUD methods). For example map.resources :articles will create the following routes:
    /articles (GET=>shows all, POST=>create a new)
    /articles/new (shows html for new article)
    /articles/1 (GET=>shows the first article, PUT=>update, DELETE=>deletes the article)
    /articles/1;edit
  29. parsing invalid xml with require ‘rubygems’ and require ‘rubyful_soup’. Then do:
    xml_data = Net::HTTP.get_response(URI.parse(url)).body
    doc = BeautifulSoup.new(xml_data)

Mavenized projects in NetBeans 6.1

Some weeks ago I mavenized timefinder to allow developers coding in other IDEs than NetBeans. I am new to all the maven stuff, but NetBeans 6.1 support is good (I have not compared it e.g. to eclipse :-/)

Now I explored a really great feature within NetBeans: you can resolve dependencies very easy: hit ALT+Enter. For example:



and hit "Search Dependency at Maven Repository".

Then choose a repository and hit add. After installing the jar in your local maven repository you will be able to resolve this particular dependency via CTRL+SHIFT+I. That’s it!

One big problems remains: you cannot profile maven projects. So, I thought it would be good to create a separate ‘pure’ NetBeans-Project where I link the source which are already used from the mavenized project. But this isn’t allowed! I don’t know why. So I hacked it under linux. This is straightforward:

create symbolic links from the pure-netbeans project to the mavenized one!

go into the ‘pure’ NetBeans-Project ‘mkdir source && cd source’  then type:

ln -s ../../../timefinder-core/src/main/resources/ core-res
ln -s ../../../timefinder-core/src/main/java/ core-src
ln -s ../../../timefinder-core/src/test/resources/ core-test-res
ln -s ../../../timefinder-core/src/test/java/ core-test-src

After this step right click the pure-project and add the source and test folders to the project



One more reason to do this is that compilation (clean & build; without tests) and
starting (until the first log statement) are slower with mavenized projects.
In my simple project with approx. 100 files you get with jdk 1.6:
     pure project compilation: max. 3 sec
mavenized project compilation: > 15 sec
             pure project run: max. 1 sec
        mavenized project run: > 4 sec

That's why I like the eclipse approach of mvn eclipse:eclipse 😉 but this has other disadvantages ...

Encoding issues. Solutions for linux and within Java apps.

Puh, encoding! Did you ever have trouble with it? No? You must be a lucky guy!

Even none-developers should have problems with it e.g. if they use different operating systems.

What is encoding? And what’s so difficult with the encoding?

First encoding (better: character encoding) defines how characters have to be saved to be displayed correctly in your editor or look at the wikipedia definition to be correct. Update: here is a nice introduction.

For example if your editor only reads ASCII files all is very simple: it will use every 8 bits of the bitstream to get a number. Then it will interpret this number according to the ASCII-table. So, if it finds a 97 (this is 0x61 in hexadecimal) it prints ‘a’.

(BTW: look at this nice ASCII-art.)

But what if the encoding is another one? Or if even the bitstream should be splitted into 16-bits-packages instead of 8-bits-packages?

Then the user won’t see the correct information!

Second: On linux everything is in UTF-8. Windows uses CP 1252. and so on. Not good!

(With everything I means: clipboard, default file encoding, …)

How can you (as an end user) handle this under linux?

There are at least 4 programs that helps you with encoding issues under linux:

  • There are command line utilities in linux where you can determine automatically the encoding of a file: enconv and enca or open the file in firefox and go to View -> Encoding and view the detected encoding!
  • To change the encoding of file-content the editor kate is really great:
    Go to extras -> encoding and try it out.
  • Change the encoding of the content of several files which come from windows and you want to have them in linux then use recode:
    recode CP1252..UTF-8 *
    recode ISO-8859-1..UTF-8 *

    do the following to backup the original files:

    mkdir test && cp * test/ && cd test
  • Another command line utility is iconv (or here)
  • Change the encoding of the filenames with convmv (files e.g. from windows).
    To preview the change do:

    convmv -f cp1252 -t utf8 *

    To do the change:

    convmv --notest -f cp1252 -t utf8 *

How does Java handle encoding?

Java is platform independent one should think, but it isn’t regarding to the encoding.

For example: if you read a file correctly under linux, this could fail if you don’t specify the encoding explicitly, because it assumes it is utf8 and under windows it will use another default!

To override the default use: ‘java -Dfile,encoding=UTF-8’ or be explicit with the encoding! E.g read characters from a stream with the following lines:

BufferedInputStream iStream = new BufferedInputStream(urlConn.getInputStream());
InputStreamReader reader = new InputStreamReader(iStream, "UTF-8");

Another issue could be Java source files. They can have different encoding. You should use UTF8, because this is the encoding Java uses for its Strings.

In NetBeans 6.1 change it in the project properties (right-click on the project->properties)->Source->Encoding

In Eclipse 3.4 go to the preferences (menu Window) -> General ->Workspace->text file encoding

But this is only useful for desktop applications like my open source timetabler. But what if you do web development? All fine there? No not really. Then you might get additional problems with url encoding or xml parsing. For the latter one the fix is simple:

  • XML: <?xml version=”1.0″ encoding=”UTF-8″?>

But for url encoding the following does not really work:

  • JSP: <%@page contentType=”text/html; charset=UTF-8″ language=”java”%>

Apropos JSP – I had an encoding issue with the request. Try the following:

<% out.print(“RESPONSE character encoding=” + response.getCharacterEncoding() + ” “);
out.print(“REQUEST character encoding=” + request.getCharacterEncoding() + ” “);
out.print(“JVM encoding ” + System.getProperty(“file.encoding”) + ” “);

//EVEN here we get request parameter in wrong encoding
bean.setRequest(request);
%>

You will see that the request is null if I am not wrong. And then Java will use utf8? NO!

It will use ISO-8859-1! Why? It is written in the standard!

A simple request.setCharacterEncoding(“UTF-8”); would help if all browsers would send its request according to the header of the jsp. But this isn’t actually working for my use case. So I grabbed the strings from the request via this helper method:

private String toUTF8(String str) {
        try {
            return new String(str.getBytes("8859_1"), "UTF8");
        } catch (UnsupportedEncodingException ex) {
            return str;
        }
}

Update 1: Read this or this to get a better workaround with a javax.servlet.Filter, webserver parameters and jsp configs.

Update 2: The following snippets could be useful if you are using maven and want to make the application UTF-8 aware:

<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.6</source>
<target>1.6</target>
<encoding>UTF-8</encoding>
</configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<configuration>
<encoding>UTF-8</encoding>
</configuration>
</plugin>

Update 3:

A good side with a lookup table for Unicode characters

http://unicode.coeurlumiere.com/

Summary

I invite you to post all your experiences with encoding problems in java.
E.g. how to force jboss or jetty to use utf8?