Use google App Engine to setup a file/image server from scratch

Introduction:
This post will talk about how to host image or files from google app engine. It use python, and the free datastore API (note: not blobstore, which is not free). This app also include functions like cache control (304 content not modified), and simple authentication. Before reading the post, I encourage you to read the two AppEngine docs http://code.google.com/appengine/docs/python/tools/webapp/ and http://code.google.com/appengine/docs/python/datastore/.

Anyway, the final app looks like this:

Get CGI mapping setup:
First is to define a WSGI mapping, which is described in http://code.google.com/appengine/docs/python/tools/webapp/ .

def main():
    application = webapp.WSGIApplication(
          [('/upload', UploadHandler),
           ('/admin', MainHandler),          
           ('/image/([^/]+)?', ServeHandler),
          ], debug=True)
    run_wsgi_app(application)

UploadHandler and MainHandler is for admin, serveHandler is used to server the actual page/image to users.

How to store data into AppEngine?
The answer is datastore API.
First we need  to define a database Model, basically a database model in AppEngine act like a table in usual databases.

class Picture(db.Model):
  title = db.StringProperty()
  picture = db.BlobProperty(default=None)
  contentType = db.StringProperty()
  etag = db.StringProperty()

This class has four elements, it is same to say table Picture has four fields/columns. title, contentType, etag are string types, element "picture" is BlobProperty type. BlobProperty is used to store actual image/file data. Now we have declared the database "table" "Picture", then how to insert actual record, which in this case is our image/file data?
The answer is easy

            temp = Picture()
            pic = self.request.get("file")
            temp.picture = db.Blob(pic)
            temp.put()

"temp = Picture()" is used to declare a table record.
"pic = self.request.get("file")" is used to get raw data from client, which is usally extracted from form-data posted from the client. Here is a example snippet from user post request.

Request headers:
Content-Type:multipart/form-data; boundary=----WebKitFormBoundaryewAu4RNTG8b8u2j7

Request payload:
------WebKitFormBoundaryewAu4RNTG8b8u2j7
Content-Disposition: form-data; name="file"; filename="æ— æ ‡é¢˜.jpg"
Content-Type: image/jpeg


------WebKitFormBoundaryewAu4RNTG8b8u2j7
Content-Disposition: form-data; name="Upload"

Upload
------WebKitFormBoundaryewAu4RNTG8b8u2j7--

"temp.picture = db.Blob(pic)" is used to store the raw data as db.Blob, for reference about db.Blob, please check http://code.google.com/appengine/docs/python/datastore/typesandpropertyclasses.html#Blob
temp.put() is used to insert the "record" “temp” into our "table" Picture.

How to post the raw data to server?

It is very easy, basically you only need to implement a client form like this:

<form action="/upload" enctype="multipart/form-data" method="POST">
Upload File: <input name="file" type="file" />
<input name="Upload" type="submit" value="Upload" /> 
</form>

How to do "304 content not modified"? First you need to calculate a etag for the data, I use:

temp.etag =  hashlib.sha1(temp.picture).hexdigest()

Then you need to push header "Cache-Control" "Etag" and "Expires" (note: please check http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html to understand the relation between cache-control and "expires") to force the browser to send head "If-None-Match" along with etag, then you only need to check the request etag is equal to the server etag or not.

if (self.request.headers.get("If-None-Match") == pic.etag):
    self.response.set_status(304)

Full Code is listed for your reference:

app.yaml

application: your_app_name_here
version: 1
runtime: python
api_version: 1

handlers:
- url: /admin
  script: admin.py
  
- url: /upload
  script: admin.py
  
- url: /image/([^/]+)?
  script: admin.py  

admin.py

#save it as admin.py
#!/usr/bin/env python
#

import os
import urllib


from google.appengine.ext import webapp
from google.appengine.ext.webapp import blobstore_handlers
from google.appengine.ext.webapp import template
from google.appengine.ext.webapp.util import run_wsgi_app
from google.appengine.api import images
from google.appengine.ext import db
import datetime
import hashlib
from google.appengine.api import users

password = "ENTER_YOUR_PASSWORD_HERE"

class Picture(db.Model):
  title = db.StringProperty()
  picture = db.BlobProperty(default=None)
  contentType = db.StringProperty()
  etag = db.StringProperty()

def getPicture(title):
  result = db.GqlQuery("SELECT * FROM Picture WHERE title = :1 LIMIT 1",
    title).fetch(1)
  
  if (len(result) > 0):
    return result[0]
  else:
    return None

class MainHandler(webapp.RequestHandler):
    def get(self):
        try:
            if (self.request.cookies.get("session") != password):
                self.response.out.write('Not Authenticated! Please authenticate!:')
                self.response.out.write('<br /><form action="admin" method="POST">')
                self.response.out.write("""Password: <input name="password" type="text" /><br /><input name="submit" type="submit" value="submit" /> </form>
""")   
                return                    
            if (self.request.get("delete") == "1"):
                id =  self.request.get("id");
                deleteObject = getPicture(id);
                if (deleteObject != None):
                    deleteObject.delete();
                    self.response.out.write("Delete Operation Succeed!!!");  
                    self.redirect('/admin')
                return    
            self.response.out.write('')
            self.response.out.write('<br /><form action="/upload" enctype="multipart/form-data" method="POST">')
            self.response.out.write("""Upload File: <input name="file" type="file" /><br /><input name="Upload" type="submit" value="Upload" /> </form>
""")
            query = db.GqlQuery("SELECT * FROM Picture")
            images = query.fetch(1000);
            self.response.out.write("FileList:=========================" + str(len(images)));        
            for image in images:
                if (image.title != None):
                    self.response.out.write("<BR><a href='/image/" + image.title + "'>" + image.title + "</a>" + "   <a href='/admin?delete=1&id=" + image.title + "'>delete </a>");
        except:
            self.response.out.write("Unexpceted error happen in  MainHandler! get");        

    def post(self):
        try:
            self.response.headers.add_header('Set-Cookie','session=' + self.request.get("password"));
            self.response.out.write('<script>window.location= window.location.toString();
</script>') 
        except:
            self.response.out.write("Unexpceted error happen in  MainHandler! post");        
            
class UploadHandler(webapp.RequestHandler):
    def post(self):
        try:
            if (self.request.cookies.get("session") != password):
                self.redirect('/admin')
                return
            temp = Picture()
            pic = self.request.get("file")
            temp.picture = db.Blob(pic)
            temp.title = self.request.POST["file"].filename 
            if (temp.title and temp.title.endswith(".jpg")):
                temp.contentType = "image/jpeg"
            elif  (temp.title and temp.title.endswith(".png")):   
                temp.contentType = "image/png"        
            elif  (temp.title and temp.title.endswith(".gif")):   
                temp.contentType = "image/gif" 
            else:
                temp.contentType = "text/html"            
            temp.etag =  hashlib.sha1(temp.picture).hexdigest()   
            temp.put()
        except:
            self.response.out.write("Unexpceted error happen in  UploadHandler! post");        
#        self.response.headers['Content-Type'] = temp.contentType
#        self.response.out.write(temp.picture)
        
        self.redirect('/admin')

class ServeHandler(webapp.RequestHandler):
    def get(self, resource):
#        try:
            resource = str(urllib.unquote(resource))
            
            pic = Picture.get_by_key_name(resource)
            if (pic is None):
                pic = getPicture(resource);
            
            now = datetime.datetime.utcnow();
            delta = datetime.timedelta(days=14)
            seconds = 14 * 24 * 60 * 60
            date = now + delta
            dateString = date.strftime("%a, %d-%b-%Y %H:%M:%S GMT");
            dateString2 = date.strftime("%a, %d %b %Y %H:%M:%S GMT");
            if (pic and pic.picture):
                if (self.request.headers.get("If-None-Match") == pic.etag):
                    self.response.headers['Etag'] = pic.etag
                    self.response.headers.add_header('Set-Cookie','etag=' + pic.etag + '; expires=' + dateString)
                    self.response.headers['Expires'] = dateString2
                    self.response.headers['Cache-Control'] = "public, max-age=" + str(seconds)
                    self.response.set_status(304)
                else:
                    self.response.headers['Content-Type'] = pic.contentType
                    self.response.headers['Cache-Control'] = "public, max-age=" + str(seconds)
                    self.response.headers['Etag'] = pic.etag
                    self.response.headers.add_header('Set-Cookie','etag=' + pic.etag + '; good=1; expires=' + dateString)
                    self.response.headers['Expires'] = dateString2
                    
                    self.response.out.write(pic.picture)

#        except:
#                self.response.out.write('File not Found in ServeHandler!');

def main():
    application = webapp.WSGIApplication(
          [('/upload', UploadHandler),
           ('/admin', MainHandler),           
           ('/image/([^/]+)?', ServeHandler),
          ], debug=True)
    run_wsgi_app(application)

if __name__ == '__main__':
  main()

ALL set! Hope it helps!