Sunday, July 12, 2009

Wicket, Spring, JDO on Google App Engine - Sample Application

In my previous post on Wicket on Google App Engine I described the basics of how to set up a simple Wicket application that runs on GAE. This post takes the next step and describes how to set up a simple CRUD application. The source code of the sample application is available here.

Overview
The sample application is a simple contacts application, where phone numbers can be associated to persons. It uses Spring for dependency injection and transaction management, JDO for persistence, and Wicket for the presentation layer.

Basic GAE/Wicket Setup
As mentioned, the basic setup of a GAE/Wicket application is the topic this post. So, as described, put the required jars in WEB-INF/lib, set up the appengine-web.xml, set up the Wicket servlet filter in web.xml, set up a Wicket WebApplication class and a home page, set up development mode and turn off the resource modification watcher. After that, you should have a basic application up and running.

Modification Watching
Let's first tackle the one issue left with the basic setup. The resource modification watching is not working, because the Wicket implementation of modification watching uses threads, which is not allowed on app engine. As of version 1.4-RC6, Wicket allows for changing the implementation, so we can write our own that does not use threads and set it in our application's init() method. The sample application uses a custom WebRequestCycle to call the modification watcher on each request.

Note: If you are using an earlier version of Wicket (pre 1.4-RC6), there's a workaround. Simply put the custom modification watcher in the same location on the classpath as the Wicket one. You can do this, by setting up your own org.apache.wicket.util.watch.ModificationWatcher in your project, which replaces the Wicket implementation.

The sample application's code to setup the custom modification watcher looks like this.

public class WicketApplication extends WebApplication {

@Override
protected void init() {
getResourceSettings().setResourceWatcher(
new GaeModificationWatcher());
}

@Override
public RequestCycle newRequestCycle(final Request request,
final Response response) {
return new MyWebRequestCycle(this, (WebRequest) request,
(WebResponse) response);
}


with a custom WebRequestCycle, that calls the modification watcher on each request (in development mode):

class MyWebRequestCycle extends WebRequestCycle {

MyWebRequestCycle(final WebApplication application,
final WebRequest request, final Response response) {
super(application, request, response);
}

@Override
protected void onBeginRequest() {
if (getApplication().getConfigurationType() == Application.DEVELOPMENT) {
final GaeModificationWatcher resourceWatcher = (GaeModificationWatcher) getApplication()
.getResourceSettings().getResourceWatcher(true);
resourceWatcher.checkResources();
}

}
}


Spring / JDO transaction management
First things first. Let's configure Spring for dependency injection and transaction handling. Our goal is to set up a JDO persistence manager factory bean, so we can persist our entities to the datastore. First step is to put the spring jars (and others: cglib, commons-logging, ..) along with the wicket-ioc and wicket-spring jars into WEB-INF/lib and include them on the classpath in our Eclipse project. Then we set up web.xml to configure a ContextLoaderListener and use the Wicket Application.init() method to setup Wicket/Spring annotation-driven dependency injection:

@Override
protected void init() {
addComponentInstantiationListener(new SpringComponentInjector(this));
}


Then we need an application context xml configuration file. This should look like this:

<tx:annotation-driven />

<bean id="persistenceManagerFactory"
class="org.springframework.orm.jdo.LocalPersistenceManagerFactoryBean">
<property name="persistenceManagerFactoryName"
value="transactions-optional" />
</bean>

<bean id="transactionManager"
class="org.springframework.orm.jdo.JdoTransactionManager">
<property name="persistenceManagerFactory" ref="persistenceManagerFactory" />
</bean>


On GAE we cannot use component-scanning or annotation-driven configuration with <configuration:annotation-driven/>, since this introduces a dependency on javax.Naming, which is not on the GAE whitelist. So we have to configure our beans through XML. But we can use annotation-driven transaction configuration.

For persistence I chose JDO instead of JPA. GAE is backed by BigTable as a datastore, and since BigTable isn't a relational database, I thought JDO might be a better fit. But JPA might be a good choice, as well. For JDO we have to configure a LocalPersistenceManagerFactoryBean and a JDO transaction manager. The factory's name must match the name in the jdoconfig.xml file, which is normally created by the Eclipse plugin. The sample application also contains a simple persistence manager factory bean for JPA, that works on GAE.

After that we can create transactional services and DAOs as Spring beans. But first, let's set up a simple, persistable domain model.

Domain Model / Persistence
Our first persistable domain object class is a person pojo entity with some JDO annotations, the persistence meta data (similar to JPA):

@PersistenceCapable(identityType = IdentityType.APPLICATION)
public class Person {

@PrimaryKey
@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
private Long id;

@Persistent
private String firstName;

@Persistent
private String lastName;

@Persistent
private Date birthday;

public Person() {
super();
}

public Person(final Date birthday,
final String firstName,
final String lastName) {
super();
this.birthday = birthday;
this.firstName = firstName;
this.lastName = lastName;
}

public Long getId() {
return id;
}

// ... Getters and Setters


All persistent entities have to be annotated with @PersistenceCapable. The primary key is annotated with @PrimaryKey. On GAE/BigTable primary keys have special semantics in addition to just uniquely identifying an entity, see the GAE documentation for details. For this example, we'll keep things simple and just choose the primary key to be of type Long (we'll change that later for some reason). All persistent fields have to be annotated with @Persistent.

That's it, for this simple entity for now. Let's go for actually persisting it to the database.

Persisting entities
The sample application uses a PersonDAO to persist person entities. This DAO could inherit from Spring's JdoDaoSupport base class, but we can also do it without it. So, our DAO might look like this:

public class PersonDao /* extends JdoDaoSupport */
implements IPersonDao {

private PersistenceManagerFactory pmf;

@Override
public Person makePersistent(final Person person) {
return getPersistenceManager().makePersistent(
person);
}

private PersistenceManager getPersistenceManager() {
return PersistenceManagerFactoryUtils
.getPersistenceManager(pmf, true);
}

public void setPmf(final PersistenceManagerFactory pmf) {
this.pmf = pmf;
}

}

The PersistenceManagerFactory will be injected by Spring. If you look at the getPersistenceManagerFactory() method you'll notice, that it's not just pmf.getPersistenceManager(), instead it calls PersistenceManagerFactoryUtils to get a persistence manager. This way we get a persistence manager that participates in Spring's transaction handling, which cares for opening and closing the persistence manager and transaction handling on our behalf.

We configure this DAO by defining it as a Spring bean in the application context xml file and inject the PersistenceManagerFactory. We can then use the makePersistent(person) method to persist person instances.

At this point we could now easily implement a simple Wicket form to create new person instances. But I'll leave this task up to you (or have a look at the sample application).

Retrieving Entities
But let's have a closer look at JDO, for those not familiar with it (like me). What we'll probably want to do, for example, is to query a single person by ID or to find all persons, applying some paging parameters. Here are some sample methods from the Person DAO to get an impression of JDO:

public Person get(final Long id) {
final Person person = getPersistenceManager()
.getObjectById(Person.class, id);
return getPersistenceManager().detachCopy(person);
}

@Override
public int countPersons() {
final Query query = getPersistenceManager()
.newQuery(Person.class);
query.setResult("count(id)");
final Integer res = (Integer) query.execute();
return res;
}

@SuppressWarnings("unchecked")
public List<Person> findAllPersons() {
final Query query = getPersistenceManager()
.newQuery(Person.class);
query.setOrdering("lastName asc");
final List<Person> list = (List<Person>) query
.execute();
return Lists.newArrayList(getPersistenceManager()
.detachCopyAll(list));
}

@Override
@SuppressWarnings("unchecked")
public List<Person> findAllPersons(final int first,
final int count) {
final Query query = getPersistenceManager()
.newQuery(Person.class);

query.setOrdering("lastName asc");
query.setRange(first, first + count);

final List<Person> list = (List<Person>) query
.execute();
return Lists.newArrayList(getPersistenceManager()
.detachCopyAll(list));
}

For further details about querying with JDO, see the resources section below. Especially for JDO on GAE, there are quite some restrictions. I think, for example, it's not possible to do something like wildcard matching in queries.

One word on detaching: all the DAO methods above detach the retrieved entities before returning them to the caller by calling detachCopy or detachCopyAll. This way the entities are detached from the persistence manager that fetched them. These instances can then be modified by the web layer and passed back to another persistence manager instance in order to update their database representation. The sample application uses the OpenPersistenceManagerInView pattern instead, configured in web.xml. This way, a persistence manager instance is opened at the beginning of the request and closed afterwards, so the web layer can freely navigate the object graph.

Mapping relationships
Let's add a simple relationship to our domain model. Assume a person has a one-to-many relationship to phone numbers. In GAE such a relationship could be owned or unowned. In an owned relationship the entities on the one side are the parents of the associated entities, which are then called child entities in this relationship. Child entities in an owned relationship cannot exist without their parents. For more information on the different types of relationships and their transaction semantics, see the GAE documentation. The following relationship between persons and phone numbers is an owned, bi-directional one-to-many relationship.

@PersistenceCapable(identityType = IdentityType.APPLICATION)
public class PhoneNumber {

@PrimaryKey
@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
private Key key;

@Persistent
private String type;

@Persistent
private String number;

@Persistent
private Person person;

//...


The phone number has a field person of type Person, annotated with @Persistent. This makes phone number a child entity of person. You'll notice the field key of type Key. This is a possible type for primary keys in the GAE. We cannot use Long in this case, because phone number is a child entity of person and as such its primary key has to contain the key of its parent (see the docs).

Note: For root entities, it's normally okay to use a key of type Long, but this did not work in my example, so I changed the key of Person to Key, too.


public class Person {

@PrimaryKey
@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
private Key key;

@Persistent
private String firstName;

@Persistent
private String lastName;

@Persistent
private Date birthday;

@Persistent(mappedBy = "person")
private List<PhoneNumber> phoneNumbers = Lists.newArrayList();

// ...


The relationship is made bi-directional by adding a list of phone numbers to the person class. The mappedBy parameter denotes the property of PhoneNumber that points to the person. With this, we can now add phone numbers to persons and these will be persisted to the database along with the owning person.

Note: Somehow there's an issue with removing phone numbers from the list by calling person.getPhoneNumbers().remove(phoneNumber), which throws an exception. But removing it by its index works, though. Let me say, that this was not the only time I had problems with the datastore. I have quite a feeling that there are still some open issues in this area.

UserService
To close this post, I'll shortly describe GAE's simple support for authentication. In a GAE application users may login with their Google account.

Checking, if a user is logged in, is easy:

final UserService userService = UserServiceFactory.getUserService();
boolean loggedIn = userService.isUserLoggedIn();


Accessing the current user's details is also easy:

User currentUser = userService.getCurrentUser();
String email = currentUser.getEmail();
String nickname = currentUser.getNickname();


And redirecting a user to a login/logout URL in Wicket can be done e.g. with a ExternalLink:

class LoginLink extends ExternalLink {

private static final long serialVersionUID = 1L;

LoginLink(final String id) {
super(id, getURL());
add(new Label("label", new LoginLabelModel()));
}

public static String getURL() {

final RequestCycle requestCycle = RequestCycle.get();

// Get the URL of this app to redirect to it
final WebRequest request = (WebRequest) requestCycle.getRequest();
final HttpServletRequest req = request.getHttpServletRequest();
final String appUrl = req.getRequestURL().toString();

// Create the login/logout page URL with with redirect tho this app
final String targetUrl = getUserService().isUserLoggedIn() ? loginUrl(appUrl)
: logoutUrl(appUrl);

return targetUrl;

}


Resources
[1] The sample application source code
[2] Apache JDO
[3] JDO Spec
[4] GAE datastore docs



18 comments:

Oleg said...

Nick, you work is very interested to me. I've downloaded sources from hg-trunk. It seams current codebase is little bit incomplete. I.e. MainPage class moved to another package, etc.
Have you already wicketgae get working at GAE?

Chris Popov said...

there is a hack to make it work:

"
On GAE we cannot use component-scanning or annotation-driven configuration with <configuration:annotation-driven/>, since this introduces a dependency on javax.Naming, which is not on the GAE whitelist. So we have to configure our beans through XML. But we can use annotation-driven transaction configuration.
"

Oleg said...

Chris, do you mean this is a reason, why wicketgae from clean sources can not be uploaded to the GAE right now?
I've tied and it fails. I'm recieving

Unable to upload app: Error posting to URL: http://appengine.google.com/api/appversion/create?app_id=wicketgaeclone&version=1&
403 Forbidden
You do not have permission to modify this app (app_id=u'wicketgaeclone').


See the deployment console for more details

Note that I've renamed to avoid possible conflict with original project name

Oleg said...

Hey, I'm get working the project. See http://wicketgaeclone.appspot.com to review. It was a trivial mistake that cause the deploy exception.
BTW, wicketgae.appspot.com is deployed too, by Nick hope, but it set as private up.

Nick Wiedenbrück said...

@Oleg: I'm new to Mercurial. Sorry, if there are any issues with the codebase. Just checked it shortly, though, and it seems okay at the first look. You can download the project as a zip file, anyway.

And yes, I've deployed the project to app engine successfully (under wicketgae.appengine.com, but not publicly available).

Thanks for the comments.

Oleg said...

Nick, first issue was not compiled sources, second was a not fresh version of the google api sdk, that are in the WEB-INF/lib. I think, they are required because of Spring. While for 'simple' project this jars do not located here.
Third surprise is why application uploaded so quick? I have relative slow connection for upload and sure all my fat jars can not be uploaded in 30 seconds! Does Google receive only jar names (wicket, spring, etc.)?
and reference its on the own side using some assumption and/or mdsums?

Anonymous said...

Wow great work, it helped me ALOT
Thanks!!

Unknown said...

I have just deployed wicketgae to the GAE but it seems extremely slow. When I create a new Person it takes ages for it to confirm and then appear in the list. Any ideas?

Nick Wiedenbrück said...

@Anthony Didn't have that experience. Feels reasonably fast, far less than a second to save a person. Did you configure development or deployment mode?

Unknown said...

@Nick. Thanks for the speedy response! It's in deployment mode. I basically downloaded your sources and uploaded them. It's at ajw-people.appspot.com.

Nick Wiedenbrück said...

@Anthony: Sorry, I don't have any problems. But there are some blogs on the web that tell the same problem with Wicket on GAE. Maybe you'll find an answer there.

Anonymous said...

transaction is not working..

@Transactional
public void save(Staff staff) {
PersistenceManager pm = PersistenceManagerFactoryUtils.getPersistenceManager(persistenceManagerFactory, true);
pm.makePersistent(staff);
throw new RuntimeException();
}

saved staff. why??

Nick Wiedenbrück said...

Sometimes some weird things can happen, especially with Wicket/OpenSessionInView/Spring Transactions. See for example:

http://stronglytypedblog.blogspot.com/2009/03/wicket-patterns-and-pitfalls-3.html

Maybe the entity was updated even before the method call? Have you called the save() method from inside the same SpringBean (then no TX will be started)?

Good luck ...

tubingen said...

[code]

FINE: ending request for page [Page class = org.radiosai.saibaba.web.page.HomePage, id = 0, version = 0], request [method = GET, protocol = HTTP/1.1, requestURL = http://localhost:8888/app/, contentType = null, contentLength = -1, contextPath = , pathInfo = null, requestURI = /app/, servletPath = /app/, pathTranslated = null]
Jan 8, 2011 1:04:01 AM com.google.appengine.tools.development.LocalResourceFileServlet doGet
WARNING: No file found for: /app/resources/org.apache.wicket.ajax.WicketAjaxReference/wicket-ajax.js
Jan 8, 2011 1:04:01 AM com.google.appengine.tools.development.LocalResourceFileServlet doGet
WARNING: No file found for: /app/resources/org.apache.wicket.markup.html.WicketEventReference/wicket-event.js
Jan 8, 2011 1:04:01 AM com.google.appengine.tools.development.LocalResourceFileServlet doGet
WARNING: No file found for: /app/resources/org.apache.wicket.extensions.ajax.markup.html.modal.ModalWindow/res/modal.css
Jan 8, 2011 1:04:01 AM com.google.appengine.tools.development.LocalResourceFileServlet doGet
WARNING: No file found for: /app/resources/org.apache.wicket.extensions.ajax.markup.html.modal.ModalWindow/res/modal.js

[/code]


please advice me how to resolve this ?

Anonymous said...

is there any issues with resource loading I keep getting this error




WARNING: No file found for: /app/resources/org.apache.wicket.markup.html.WicketEventReference/wicket-event.js
Jan 8, 2011 7:45:57 PM com.google.appengine.tools.development.LocalResourceFileServlet doGet
WARNING: No file found for: /app/resources/org.apache.wicket.ajax.WicketAjaxReference/wicket-ajax.js
Jan 8, 2011 7:45:57 PM com.google.appengine.tools.development.LocalResourceFileServlet doGet
WARNING: No file found for: /app/resources/org.apache.wicket.extensions.ajax.markup.html.modal.ModalWindow/res/modal.js
Jan 8, 2011 7:45:57 PM com.google.appengine.tools.development.LocalResourceFileServlet doGet
WARNING: No file found for: /app/resources/org.apache.wicket.extensions.ajax.markup.html.modal.ModalWindow/res/modal.css


could you pelase help me with this ?

Chris Merrill said...

Nick, thanks for this post! I found a few answers that helped me get our Wicket/Spring/JPA app ported over to GAE. Thanks!

Chris

Anonymous said...

its really a great work as a beginner in google app engine

Anonymous said...

its really a great work. as am a beginner in google app engine it help me to get structure for my IDE