Google App Engine

Google App Engine
Feedback on this document

Getting Started

Before running through the steps below, make sure that:

  • You have activated your project for GCS and App Engine, including creating at least one GCS bucket.
  • You have downloaded the client lib distribution and unzipped it.
  • You have installed and configured the latest App Engine Java SDK.

Running the LocalExample.java Sample (Eclipse)

LocalExample.java is a non-deployable sample useful for quick testing and investigation of the GCS functionality. It has no UI components other than Eclipse console output. (GCS client library deployable samples with UI are also available.)

To run LocalExample.java in Eclipse:

  1. Start Eclipse.
  2. In Eclipse, click Windows > Preferences > Google > Appengine and then click Add.
  3. Follow the prompts to supply the install path of the App Engine SDK, then click OK.
  4. In the Files menu, click Files > New > Java Project and create a project named LocalExample with a package name of com.google.appengine.demos.
  5. Select the project in the Package Explorer pane and click Files > New > Class and give the class the name LocalExample with a package name of com.google.appengine.demos.
  6. Copy the contents of the LocalExampleJava source into that class file.
  7. Select the project again in the Package Explorer pane and right-click, then click Properties > Java Build Path.
  8. In the Libraries tab, click Add External Jars. You must add the following JARs:
    • appengine-gcs-client.jar from wherever you installed the GCS client library
    • guava-15.0.jar from wherever you installed the GCS client library
    • joda-time-2.3.jar from wherever you installed the GCS client library
    • appengine-testing.jar from the App Engine install subdirectory /lib/testing.
    • appengine-api-stubs.jar from the App Engine install subdirectory /lib/impl.
  9. Build, then Run As > Java Application.
  10. You should see the following output in the Eclipse Console pane:

Investigating the LocalExample.java Sample

LocalExample.java is described in detail below, to illustrate the use of the GCS client library.

Imports

there are a number of imports used that you might not need, or that are used to enable local testing. The followng snippet lists this sample's imports:

import com.google.appengine.tools.cloudstorage.GcsFileOptions;
import com.google.appengine.tools.cloudstorage.GcsFilename;
import com.google.appengine.tools.cloudstorage.GcsInputChannel;
import com.google.appengine.tools.cloudstorage.GcsOutputChannel;
import com.google.appengine.tools.cloudstorage.GcsService;
import com.google.appengine.tools.cloudstorage.GcsServiceFactory;
import com.google.appengine.tools.cloudstorage.RetryParams;
import com.google.appengine.tools.development.testing.LocalBlobstoreServiceTestConfig;
import com.google.appengine.tools.development.testing.LocalDatastoreServiceTestConfig;
import com.google.appengine.tools.development.testing.LocalServiceTestHelper;

import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.nio.ByteBuffer;
import java.nio.channels.Channels;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;

These imports are briefly described below:

  • com.google.appengine.tools.cloudstorage.* this lets us use the GCS client library.
  • com.google.appengine.tools.development.testing.* Only needed for local unit testing of certain App Engine features while running your app.
  • java.io.ObjectInputStream and java.io.ObjectOutputStream are used for reading and writing objects.
  • java.nio.ByteBuffer is used for unbuffered reads and writes.
  • Although not shown in the snippets,java.io.IOException is needed for error handling.
  • java.nio.channels.Channels is used to convert the input or output channel to a stream.
  • java.nio.channels.ReadableByteChannel is used for the data being read in from GCS.

Creating GcsService To Send Requests to GCS

To send and receive requests to GCS via this library, you need a GcsService instance:

private final GcsService gcsService =
    GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());

In the snippet, notice the use of RetryParams in the call to createGcsService. Supplied as shown, createGcsService(RetryParams.getDefaultInstance()), this sets the default retry settings to be used for retries in the event of request failures due to timeouts or other unexpected failures in accessing GCS. To specify different values, such as for maximum number of retries, you use RetryParams.Builder to change settings in a new RetryParams object and supply it when you create GcsService. Note that once a GcsService object is created, its retry parameters cannot be changed.

You can create as many GcsService instances as you need. Each instance is independent, immutable (and therefore threadsafe), and reusable. For example, one can be used to write a file with one set of parameters, and another can read a different file with different retry parameters at the same time.

A recommended practice is to use a separate (possibly static) instance in each of your classes that does I/O; this will help keep your classes independent and has very little overhead.

Writing Data to GCS

The following samples show how to write data to a file stored in GCS. Separate snippets are provided for writing serialized object data and byte arrays.

GCS files are not fully created until close is invoked.

In the snippets below, close does not appear in a finally block, as is often done in Java. This is so that in the event of an exception, any partly written data will automatically be cleaned up.

Writing an Object to a GCS File

Here is how to serialize an object to a GCS file. First, get a writable byte channel for GCS:

GcsOutputChannel outputChannel =
    gcsService.createOrReplace(fileName, GcsFileOptions.getDefaultInstance());

In the snippet, GcsService.createOrReplace is invoked with the GcsFilename as the first parameter. This object contains the name of the GCS bucket to be used and name of the object. If there already is an object with that same name in that bucket, and your app has write permissions to it, this call will overwrite the existing file. (GCS does not support appending.) If there is no file of that name, this call results in the creation of a new file.

The second parameter shown is GcsFileOptions. To supply default options, use GcsFileOptions.getDefaultInstance. To supply your own settings, use GcsFileOptions.Builder to set options and create a GcsFileOptions to supply as the second parameter.

The file options are passed through to GCS to tell it the content type of the file, any user metadata you want passed in the headers, the ACL to govern access to the file, and so forth. If you don't supply a content type (mimeType), the file will be served from GCS in the default MIME type, which is currently binary/octet-stream. If you don't specify an ACL, the access assigned to the object will be the current default object ACL.

Note that all of the file options information can be obtained from an object after a close without having to download the object itself, by calling GcsService.getMetadata(fileName).

For more information on the above settings, visit the Google Cloud Storage documentation on ACLs and file options.

Now write the data using an output stream: 

ObjectOutputStream oout = new ObjectOutputStream(Channels.newOutputStream(outputChannel));
oout.writeObject(content);
oout.close();

Writing a Byte Array to a GCS File

Here is how to write a byte array to a GCS file:

GcsOutputChannel outputChannel =
    gcsService.createOrReplace(fileName, GcsFileOptions.getDefaultInstance());
outputChannel.write(ByteBuffer.wrap(content));
outputChannel.close();

For a description of the parameters used in the createOrReplace call, see the description under Writing an Object to a GCS File

Reading Data from GCS

The following samples show how to read data from a file stored in GCS. Separate snippets are provided for reading the GCS file into an object (serialize) and into a byte array.

Reading GCS Data into an Object

This approach is useful for streaming larger files into a buffer: 

GcsInputChannel readChannel = gcsService.openPrefetchingReadChannel(fileName, 0, 1024 * 1024);

The call to GcsService.openPrefetchingReadChannel takes aGcsFilename, which contains the name of the GCS bucket to be read from and name of the object to be read. The second parameter is the number of bytes from the start of the file, with 0 as shown causing the read to start at the beginning of the file. If you supply some other starting point within the file, say, at byte 300, the read operation begins at byte 300 and continues to the end of the file, or until you stop reading. (The prefetch buffer will read ahead, though, so it will contain bytes past the point where you stop.)

Prefetching provides is a major performance advantage for most applications, because it allows processing part of the file while more data is being downloaded in the background in parallel.

The third parameter is the size of the prefetch buffer, set in the example to 1 megabyte.

Now, read the file from the channel:

ObjectInputStream oin = new ObjectInputStream(Channels.newInputStream(readChannel));
try {
  return oin.readObject();
} finally {
  oin.close();
}

Reading GCS Data into a Byte Array

For smaller files, you could read the file all at once into a byte array: 

int fileSize = (int) gcsService.getMetadata(fileName).getLength();
ByteBuffer result = ByteBuffer.allocate(fileSize);
GcsInputChannel readChannel = gcsService.openReadChannel(fileName, 0);
try {
  readChannel.read(result);
} finally {
  readChannel.close();
}

In the snippet above, note the use of java.nio.ByteBuffer, especially the setting of the buffer size to be equal to the size of the file being read in from the channel.

Note, however, that for most applications, the prefered method is to stream the file (see Reading GCS Data into an Object) because that doesn't require holding the all of the data in memory at once.

Deployable GCS Client Library Samples with UI

For deployable samples with UI, visit the source code for:

You'll need to build the samples before you can run them against the development server or deploy them. To build the samples:

  1. Checkout the source code by invoking the following command from a terminal window: 
    svn checkout http://appengine-gcs-client.googlecode.com/svn/trunk/ appengine-gcs-client-read-only
  2. Change directory to /gcs-samples/appengine-gcs-client-read-only/java.
  3. Invoke ant compile_example, which will build the samples using the build.xml in that directory.
  4. Run the dev server by invoking 
    /path/to/AppEngSDK/dev_appserver.sh  /path/to/example/war
  5. In your browser, visit localhost:8080. For GCSExampleServlet you should see something like this:

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 3.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies.

Authentication required

You need to be signed in with Google+ to do that.

Signing you in...

Google Developers needs your permission to do that.