Jetwick Twitter Search is now free software! Wicket and Solr pearls for Developers.

Today we released Jetwick under the Apache 2 license.

Why we made Jetwick free software

I would like to start with an image I made some years ago for TimeFinder:

This is one reason and we are very interested in your contributions as patches or bug reports.

But there are some more interesting opportunities when releasing jetwick as open source:

  • Open architecture: several jetwick-hosters could provide parts of the twitter index and maybe at some day we can have a way to freely explore all tweets at twitter in the jetwicked way
  • Personalized jetwick: every user has different interests. If you only feed tweets from searches of terms that your are interested in and also only your timeline, then you will be able to search this personalized twitter, sort against retweets, see personal URL-trends, etc.
    This way you’ll be informed faster, wider and more personalized than with an ordinary rss feed or the ordinary twitter timeline. Without reading a lot of unrelated content. If jetwick would stay closed then this task would be too resource intensive or even impossible to convince every user.

In our further development we will concentrate on the second point, because then jetwick will have at least one user

Explore Jetwick

Why you should install Jetwick and try it out?

First you can look at the features and see if something interesting – for you as a user – is shown.

For developers there could be the following things (and more) worth to be investigated:

Jetwick can be used as a simple showcase how to use wicket and solr:

  1. show the use of facets and facet queries
  2. show date facets as explained in here
  3. make autocompletion working
  4. instant results when you select a suggestion from autocompletion as explained in this video

If you are programming some twitter stuff you should keep an eye on the following points:

  • spam detection as explained in this post
  • how you can use oAuthentication with twitter4j and wicket
  • transform tweets with clickable users, links and hashtags
  • translate tweets with google translate

If you are new to wicket

  • Jetwick is configured that if you run ‘mvn jetty:run’ you can update html and code and you can hit refresh on the browser 1-2 seconds later to see the updated results. For css it will be updated immediately
  • query wikipedia and show the results in a lazyload panel

Some solr gems:

  • simple near realtime set up with solr. And that although we make heavy usage of facets where a lot autowarming is required.
  • if you re-enable the user search you can use twitters person suggestions on your own data. I’m relative sure that twitter uses the ‘more like this’ feature of lucene that jetwick had implemented with solr.

fluid database and the PermGenSpace checker

  • fluid update of your database from hibernate changes (via liquibase). Hint: at the moment we only use the db for the ytags
  • a simple helper method ‘getPermGenSpace’ to check if reload is possible

The following interesting issues are still open:

  • using a storage for tweets (mysql or redis or …). This will increase the index time dramatically, because we had to switch to a pure solr solution (we had problems with h2 and hibernate for over 4mio tweets)
  • a mature queue like 0mq and protobuf or something else
  • a ‘real’ realtime solution for solr, if we use solr from trunk

One thought on “Jetwick Twitter Search is now free software! Wicket and Solr pearls for Developers.

  1. Pingback: Jetwick is now Open Source : Pannous GmbH

Comments are closed.