While true

Me and my other half write our weekly shopping list on a notepad. It's not a spectacularly inefficient system but it has its downsides: we can't both work on it at the same time, we often forget it when we go shopping, and we often think of things to add to it while we're away from home (and the wretched pad). In any case, it gave my inner geek a great excuse to digitise the process and also to write something else in Go, which I really like but have yet to make anything non-trivial with. As I've been wanting to give Google App Engine (GAE) a go, again this was the perfect excuse.

As it turns out, I'm glad that my first time deploying to GAE was with a small, simple app, because if this had been part of a larger project I would have given up and used a different service. Most of these 'gotchas' were my own fault, partly because I failed to appreciate the great lengths that GAE goes to make an app scale, even if it's just intended for two users (not that it has any way of knowing what my intentions are!). This involves making simple behavioral choices for the developer, assuming that he will understand them.

For my part, I was just looking for a way to quickly push a very simple app, so it was not the best fit, especially since I'm used to the likes of Rackspace and EC2 where the developer has (almost) complete control. That said, now that I understand these quirks and why they're here, I would certainly consider using GAE in the future--just not for tiny personal projects. In hope that others don't waste their time too, here's a list of the issues I faced when deploying to GAE and how I resolved them.
It's likely that these issues will affect any GAE deployment (not just Golang apps), but I only tested this language.

Note: This post does not aim to be a tutorial in GAE app creation or deployment. If you want the basics, head over to Google's tutorial.

1. Don't use mutable global variables for anything

(Updated following some comments on Reddit)

In general, you wouldn't use mutable global variables in a server application anyway. If the server gets rebooted, you'll lose whatever is stored there. But when testing a new cloud platform (GAE), you want to know what you can and can't do. If I deploy an application on EC2, I can use global state to store things if I feel like it, and everything will behave perfectly well until the VM gets rebooted. In my experience they don't get rebooted that often, so if the variable contains something that did not have strong persistence requirements to begin with, you may not notice it being reset or it may not annoy you very much.

For example, in this shopping list app I wanted to create a map of weekdays to booleans to indicate whether or not to cross that weekday off the list (if the list was complete for that day), and thought instead of using the Datastore I'd use a global array of booleans - big mistake.

In GAE, it seems that VMs get rebooted much more often, and your global state is obviously not shared between VMs, so there's no point in using it to store anything. Memcached and the Datastore are the only ways of persisting information. As has been pointed out to me, the development server goes to great lengths to replicate and even exaggerate this behavior so that you aren't surprised by it in production.
The bottom line is: don't use mutable globals, especially in GAE!

2. Mime-types must be specified for all static handlers

This puzzled me for a while because it results in cryptic error messages from appcfg.py when you try and deploy:
UnicodeDecodeError: 'ascii' codec can't decode byte 0x89 in position 0: ordinal not in range(128)

The fix is simple:

All static file handlers in your app.yaml file must include a mime-type attribute, despite what it says here.

GAE is supposed (by its own admission) to know the Mime-Type of some common extensions, like CSS, but if you don't include them, you sometimes get the above exception. If you do, then you're probably missing some Mime-Types in app.yaml that the app engine can't figure out on its own. It would be nice if it could say this in a human-readable way, but there are a number of posts on SO (see eg. here) relating to this so it shouldn't be too difficult to figure out.
This is a particular pain point for web-font folders, since you'll have to specify a different handler for each font file type just so they can specify separate Mime-types for them all (or at least this was true on my system).

3. Static files don't update when you appcfg.py update

The command python appcfg.py update [app path] is supposed to update the deployed version of your app with the changes that you have made to the local version. Unfortunately, if you expect it to update all your static files seamlessly, you're in for a world of pain. The process went something like this:
1. Deploy version 1 of your app
2. Go to [your-app-id].appspot.com and confirm that everything works as planned.
3. Actually, you notice something is not quite right in the styling and you end up making a small adjustment to the CSS
4. Re-deploy your app with a version bump.
So, you expect that if you go back to [your-app-id].appspot.com, you'll see the new styling, right? Nope. Not even if you clear your browser cache. GAE is very good at caching static files - so good that when you push a new version, it will happily ignore it.

The fix

According to what I could find online, some "cache-busting" semantics may work for you (see also this post). This involves changing all references to static files in browser-facing code (HTML/JS/CSS) to include an identifier that is unique to your version number, preventing GAE from serving up old versions.
Example: Instead of linking to js/app.js in your index.html, link to js/app.js#[version number].
Sounds good, but for some reason it didn't work for me and I still don't know why. Fortunately, GAE also uses a subdomain identifier to version your app, so if you want to access, say, version [xx] of your app, you can go to [xx].latest.[your-app-id].appspot.com. This worked for me, although it is not a pleasant solution by any means since I'll have to update the bookmark on my iPhone every time I make any small change to the app.
If I can find out why the cache busting technique failed for me I'll update this post.