Using Silverlight and WCF to create a RESTful File Upload Utility

12 02 2009

Overview

During the development of Zleek, I ran into some problems uploading files from Silverlight. I wanted a file upload utility that provided progress notification to the user. However, there is no built-in progress notification for the WebClient or HttpWebRequest classes! More specifically, the event exists but it currently does not provide enough resolution to be meaningful. It fires at only 100%, so it can only be used as a completion event. This is not good news if you want to provide file upload progress to users.

The second problem is that if you wanted to upload large files, you had to set your WCF Service‘s MaxMessageSize to whatever the maximum file size you are expecting — and if you’re expecting video files, it would have to be absurdly large.

The solution to both of these problems is to have the client break the files up into smaller chunks before sending them off to the server, which would then reassemble them in the correct order. This scenario is not overly complicated, but since it requires a good deal of busy work, I decided to put together a Silverlight library for the client and a WCF library for the service that allows you to quickly and easily build file upload functionality using Silverlight and WCF. This post explores how the libraries were built, and how to use them.

REST Refresher

First off, you should familiarize yourself with REST, WCF, and consuming WCF services from Silverlight by taking a look at Rob Bagby‘s REST in WCF series and my earlier post on Silverlight/WCF communication.

Here is a quick refresher:

REST is part of a service architecture (ROA) that attempts to simplify services by limiting communication to URIs, HTTP Verbs, and HTTP Status Codes. There are no complicated SOAP envelopes that a client needs to create. It just needs to know the URI it wishes to connect to.

URIs should be designed to be hackable, e.g. If sending a GET to http://myhost/myservice.svc/Media/1 retrieves the media item with ID 1, the consumer of the service would be able to guess that a DELETE would delete the media with that ID, while a PUT would insert or update the media with that ID.

Status Codes should be used effectively to report error conditions. Some common ones are:

  • 404 (Not Found), example: Returned for a GET/DELETE attempt if the specified ID does not exist
  • 400 (Bad Request), example: One of the parameters is not valid
  • 403 (Forbidden), example: Client is not authorized to perform the operation

eTag and Last Modified headers should be used effectively to permit caching of GET requests.

Use the WCF REST Starter Kit to create the service.

Client Library

Let’s start with the client. The client is responsible for breaking the files to be uploaded into smaller chunks and sending them off to the server. The main client class is UploadUtility (located in Agnition.Silverlight.Utils/Net), which uses RestfulServiceWrapper (located in Agnition.Silverlight.Utils/ServiceModel) to make calls into a RESTful WCF service.

UploadUtility.PerformUpload() takes a URI, URI format string, and a list of FileInfo objects. It creates a FileWrapper for each of these files, and splits them up into smaller “chunks” (FileChunk). These files are then queued up for upload.

After queueing up the files, the client starts a background process which continually checks to see how many “chunks” are currently being uploaded, and if it is under the MaxUploadCount, pulls the next file out of the queue. When the file is pulled out of the queue, the first “chunk” for the file is uploaded. When the upload completes, progress is reporting and if there is a next “chunk”, it starts the upload for that chunk.

Once the queue is empty, the entire upload process reports as complete.

The URI format string is responsible for telling the client how to perform variable substitution. This format string should at a minimum contain the following placeholders:

  • BaseUri – the service URI.
  • FileName – the original file name of the file being uploaded.
  • FileKey – a unique identifier for the file being uploaded.
  • ChunkNumber – the sequence number for the current chunk being uploaded.
  • TotalChunks – the total number of chunks that make up the file.

The default format string is “{BaseUri}/{FileKey}/{TotalChunks}/{ChunkNumber}?Name={FileName}“, which is used to create the target URI when upload requests are issued. You can also specify custom parameters by inheriting from FileWrapper and overriding UploadUtility.CreateWrapper(), more on this later. FileName and all custom parameters will automatically be URL Encoded.

Important:If a parameter (such as FileName) contains values that might include characters that must be URL encoded, they should be formatted as a query string parameter rather than as part of the main URI. Otherwise, the RESTful WCF service will not correctly map the request to the appropriate method.

The upload utility provides the following settings which can be used to tune the upload process:

  • MaxUploadCount controls the number of concurrent uploads that the client issues. Defaults to 1.
  • MaxUploadSize controls the “chunk” size of the files. If a file size is greater than this setting, it will be split into two or more “chunks”. Defaults to 200KB.
  • MaxFileSize is the maximum file size for any file being uploaded. If a file size is greater than this, it will not be uploaded. Defaults to 30MB.
  • IsApproximatingProgress will increase the resolution of progress notification by using the detected transfer rate to report a calculated progress using the specified reporting interval. When a “chunk” completes upload, the calculated progress will be rest to the actual progress. A “speed test” will be performed prior to uploading the first “chunk” in order to seed the transfer rate.
  • ProgressReportingFrequency sets the progress reporting interval when IsApproximatingProgress is true. Defaults to 20 msec.
  • ProgressReportingMultiplier is a “fudge factor” multiplier to apply to any calculated progress reporting when IsApproximatingProgress is true. This should be between 0 and 1. Defaults to 1.
  • SpeedTestMinimumSize sets the minimum size of the “speed test” to perform when IsApproximatingProgress is true. Defaults to 32KB
  • SpeedTestMaximumSize sets the maximum size of the “speed test” to perform when IsApproximatingProgress is true. Defaults to 200KB.
  • SpeedTestSizePercentage sets the size as a percentage of the total upload size of the “speed test” to perform when IsApproximatingProgress is true. Defaults to 5%.

Server Library

The server logic is very straightforward. Its only job is to reassemble temporary “chunks” uploaded by a client into a complete file, and delete those chunks when the completed file is assembled or the upload is cancelled. It is the job of UploadHandler.cs (located in Agnition.Utils/ServiceModel) to perform these tasks. UploadServiceHostFactory is a subclass of RestServiceHostFactory that sets the MaxMessageSize. Important: This should match the MaxUploadSize of the UploadUtility client!

Consuming services should create a singleton instance of UploadHandler and call UploadHandler.ProcessUpload() and UploadHandler.CancelUpload() to perform these processes.

Putting It All Together

We can use the client and server libraries to put together a file uploader for media files. Let’s start off by building the client.
ui
The UI is pretty straightforward. It contains an upload/cancel button, an overall progress display, and a file progress display. The individual file progress displays are seperated out into another user control that takes an instance of FileWrapper in order to display a preview.

To extend the libraries for media support, I created a few additional classes: MediaFileWrapper, which inherits from FileWrapper by providing a FileType custom parameter, and MediaUploadUtility, which inherits from UploadUtility. The source code for these are as follows:

MediaFileWrapper.cs

public class MediaFileWrapper : FileWrapper

{

    public const string IMAGE_TYPES = "bmp;jpg;jpeg;gif;png;tif;tiff";

    public const string VIDEO_TYPES = "avi;mpg;mpeg;wmv;asf;dvr-ms;m2v;ts;m2t;vob;mod;avs;mov;m4v;mp4;3gp;3g2;dv;mp2";

    public const string SILVERLIGHT_IMAGE_TYPES = "jpg;jpeg;png";

    public const string SILVERLIGHT_VIDEO_TYPES = "wmv";

 

    public FileType Type {

        get { return _type; }

    }

 

    public bool CanPreview {

        get { return _canPreview; }

    }

 

    public MediaFileWrapper(FileInfo file) : base(file) {

        _type = DetermineType(file);

        this.Parameters.Add(new KeyValuePair<string, string>("FileType", Type.ToString()));

    }

 

    private FileType DetermineType(FileInfo file) {

        string extension = file.Name.Substring(File.Name.LastIndexOf('.') + 1).ToLower();

        List<string> imageExtensions = new List<string>(IMAGE_TYPES.Split(';'));

        List<string> silverlightImageExtensions = new List<string>(SILVERLIGHT_IMAGE_TYPES.Split(';'));

        if (imageExtensions.Contains(extension)) {

            _canPreview = silverlightImageExtensions.Contains(extension);

            return FileType.Image;

        }

 

        List<string> videoExtensions = new List<string>(VIDEO_TYPES.Split(';'));

        List<string> silverlightVideoExtensions = new List<string>(SILVERLIGHT_VIDEO_TYPES.Split(';'));

        if (videoExtensions.Contains(extension)) {

            _canPreview = silverlightVideoExtensions.Contains(extension);

            return FileType.Video;

        }

 

        return FileType.Other;

    }

 

    private FileType _type;

    private bool _canPreview;

}

MediaUploadUtility.cs

public class MediaUploadUtility : UploadUtility

 {

     public List<MediaFileWrapper> MediaFiles {

         get {

             return this.Files.Cast<MediaFileWrapper>().ToList();

         }

     }

 

     public MediaUploadUtility() : base() { }

 

     protected override FileWrapper CreateWrapper(FileInfo file) {

         return new MediaFileWrapper(file);

     }

 

     protected override void ValidateFile(FileWrapper file) {

         ValidateFileType((MediaFileWrapper) file);

         base.ValidateFile(file);

     }

 

     protected void ValidateFileType(MediaFileWrapper file) {

         if (file.Type == FileType.Other) {

             throw new Exception(string.Format("File '{0}' is an unsupported file format.", file.File.Name));

         }

     }

 }

In the actual UI, the pertinent code is the UploadFiles() method, which calls the MediaUploadUtility to initiate the upload process, and the progress notification event handler Uploader_UploadProgressChanged(), which updates the progress notification for the overall upload progress and the progress for each individual file.

FileUpload.xaml.cs

public partial class FileUpload : UserControl

{

    protected MediaUploadUtility Uploader {

        get { return _uploader; }

    }

 

    public FileUpload() {

        InitializeComponent();

 

        _uploader = new MediaUploadUtility();

        _uploader.UploadProgressChanged += new UploadProgressChangedHandler(Uploader_UploadProgressChanged);

        _uploader.UploadCompleted += new UploadCompletedHandler(Uploader_UploadCompleted);

        _uploader.UploadCancelled += new UploadCancelledHandler(Uploader_UploadCancelled);

 

        _uploader.IsApproximatingProgress = true;

        _uploader.ProgressReportingMultiplier = 0.8;

        _uploader.MaxUploadCount = BrowserUtil.Name == BrowserName.Firefox ? 6 : 2;

    }

 

    private void UploadFiles(IEnumerable<FileInfo> files) {

        // Start upload

        Progress.Value = 0;

        ProgressText.Text = "0.00%";

        Uploader.PerformUpload(files, new Uri(Configuration.UploadServiceAddress + "/Media/Upload", UriKind.Absolute),

            "{BaseUri}/{FileType}/{FileKey}/{ChunkNumber}/{TotalChunks}?Name={FileName}");

 

        // Add files to preview window

        FileProgressContainer.Children.Clear();

        foreach (MediaFileWrapper file in Uploader.MediaFiles) {

            FileProgressContainer.Children.Add(new FileProgressDisplay(file));

        }

    }

 

    private void CancelUpload() {

        if (Uploader.IsWorking) {

            UploadButton.Content = "Cancelling...";

            UploadButton.IsEnabled = false;

            Uploader.Cancel(new Uri(Configuration.UploadServiceAddress + "/Media/CancelUpload", UriKind.Absolute),

                "{BaseUri}/{FileKey}");

        }

    }

 

    private static string CreateFileFilter() {

        // Get image extensions

        string imageExtensions = string.Empty;

        foreach (string ext in MediaFileWrapper.IMAGE_TYPES.Split(';')) {

            if (imageExtensions.Length > 0) {

                imageExtensions += ";";

            }

            imageExtensions += "*." + ext;

        }

 

        // Get video extensions

        string videoExtensions = string.Empty;

        foreach (string ext in MediaFileWrapper.VIDEO_TYPES.Split(';')) {

            if (videoExtensions.Length > 0) {

                videoExtensions += ";";

            }

            videoExtensions += "*." + ext;

        }

 

        // Create filter string

        return string.Format("ImageFiles|{0}|Movie Files|{1}|All Supported Media|{0};{1}",

          imageExtensions, videoExtensions);

    }

 

    private void UploadButton_Click(object sender, RoutedEventArgs e) {

        switch (UploadButton.Content.ToString()) {

            case "Upload!":

                OpenFileDialog uploadDialog = new OpenFileDialog();

                uploadDialog.Filter = CreateFileFilter();

                uploadDialog.FilterIndex = 3;

                uploadDialog.Multiselect = true;

                if (uploadDialog.ShowDialog() == true) {

                    UploadButton.Content = "Cancel";

                    UploadAnimation.Begin();

                    UploadFiles(uploadDialog.Files);

                }

                break;

            case "Cancel":

                CancelUpload();

                break;

        }

    }

 

    private void Uploader_UploadCancelled(object sender, EventArgs e) {

        Dispatcher.BeginInvoke(delegate() {

            ProgressText.Text = "Cancelled";

            UploadButton.Content = "Upload!";

            UploadButton.IsEnabled = true;

            UploadCompleteAnimation.Begin();

        });

    }

 

    private void Uploader_UploadCompleted(object sender, EventArgs e) {

        Dispatcher.BeginInvoke(delegate() {

            ProgressText.Text = "Completed";

            UploadButton.Content = "Upload!";

            UploadButton.IsEnabled = true;

            UploadCompleteAnimation.Begin();

        });

    }

 

    private void Uploader_UploadProgressChanged(object sender, UploadProgressEventArgs e) {

        Dispatcher.BeginInvoke(delegate() {

            Progress.Value = e.TotalPercentComplete;

            ProgressText.Text = string.Format("{0:n2}%", e.TotalPercentComplete);

            foreach (FileProgress fileProgress in e.FileProgress) {

                foreach (FileProgressDisplay fileDisplay in FileProgressContainer.Children) {

                    if (fileProgress.File.File == fileDisplay.UploadedFile.File) {

                        fileDisplay.Progress = fileProgress.Progress;

                        fileDisplay.ProgressText = string.Format("{0:n0}%", fileProgress.Progress);

                        break;

                    }

                }

            }

        });

    }

 

    private readonly MediaUploadUtility _uploader;

}

Easy, right? UploadUtility abstracted away all of the mess!

Now let’s build the service. I created the MediaUploadHandler class, which inherits from UploadHandler, to provide the correct upload paths and provide additional processing dependent on the file type.

MediaUploadHandler.cs

public class MediaUploadHandler : UploadHandler

{

    public override string UploadPath {

        get { return AppDomain.CurrentDomain.BaseDirectory + "/Uploads/"; }

    }

 

    public override string TemporaryUploadPath {

        get { return UploadPath; }

    }

 

    protected override void ProcessCompletedFile(string fileKey, string fileName, string outputFilePath, object[] additionalParameters) {

        MediaType mediaType = (MediaType) additionalParameters[0];

 

        // TODO: Process images/videos differently based on media type

    }

}

Next, add a new service to your WCF project. Edit the XAML to specify the Agnition.Utils.ServiceModel.UploadServiceHostFactory, and then create three service methods: UploadChunk, CancelChunk, and SpeedTest as follows. Be sure to pay attention to the URI format strings!

Upload.svc

<%@ ServiceHost

  CodeBehind="Upload.svc.cs"

  Debug="true"

  Language="C#"

  Service="Agnition.Test.Web.Service.Upload"

  Factory="Agnition.Test.Web.Service.UploadServiceHostFactory"

%>

Upload.svc.cs

[ServiceContract(Namespace = "Agnition.Test.Web.Service.Upload")]

[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]

public class Upload

{

    public static MediaUploadHandler MediaUploadHandler {

        get {

            return _uploadHandler;

        }

    }

 

    [OperationContract]

    [WebInvoke(Method = HttpVerbs.PUT,

    UriTemplate = "Media/Upload",

    ResponseFormat = WebMessageFormat.Json,

    RequestFormat = WebMessageFormat.Json)]

    public string SpeedTestResponse(Stream data) {

        byte[] binaryData = UploadHandler.ReadInputStream(data);

        return "Speed Test Complete. " + binaryData.Length + " bytes received.";

    }

 

    [OperationContract]

    [WebInvoke(Method = HttpVerbs.DELETE,

    UriTemplate = "Media/CancelUpload/{fileKey}",

    ResponseFormat = WebMessageFormat.Json,

    RequestFormat = WebMessageFormat.Json)]

    public void CancelUpload(string fileKey) {

        fileKey = HttpUtility.UrlDecode(fileKey);

 

        // Check input variables

        if (string.IsNullOrEmpty(fileKey)) {

            RestfulServiceHelper.ReturnBadRequest("Invalid FileKey");

            return;

        }

 

        MediaUploadHandler.CancelUpload(fileKey);

    }

 

    [OperationContract]

    [WebInvoke(Method = HttpVerbs.PUT,

    UriTemplate = "Media/Upload/{fileType}/{fileKey}/{chunkNumber}/{totalChunks}?Name={fileName}",

    ResponseFormat = WebMessageFormat.Json,

    RequestFormat = WebMessageFormat.Json)]

    public void UploadChunk(string fileKey, string fileName, string fileType, string chunkNumber, string totalChunks, Stream data) {

        // Check input variables

        if (string.IsNullOrEmpty(fileKey)) {

            RestfulServiceHelper.ReturnBadRequest("Invalid FileKey");

            return;

        }

        if (string.IsNullOrEmpty(fileName)) {

            RestfulServiceHelper.ReturnBadRequest("Invalid FileName");

            return;

        }

        if (string.IsNullOrEmpty(fileType)) {

            RestfulServiceHelper.ReturnBadRequest("Invalid FileType");

            return;

        }

        if (string.IsNullOrEmpty(chunkNumber)) {

            RestfulServiceHelper.ReturnBadRequest("Invalid ChunkNumber");

            return;

        }

        if (string.IsNullOrEmpty(totalChunks)) {

            RestfulServiceHelper.ReturnBadRequest("Invalid TotalChunks");

            return;

        }

        if ((data == null) || (!data.CanRead)) {

            RestfulServiceHelper.ReturnBadRequest("Invalid Data");

            return;

        }

 

        // Parse input

        int iChunkNumber = 0;

        int iTotalChunks = 0;

        FileType eFileType = FileType.Other;

        if (!int.TryParse(chunkNumber, out iChunkNumber)) {

            RestfulServiceHelper.ReturnBadRequest("Invalid ChunkNumber");

            return;

        }

        if (!int.TryParse(totalChunks, out iTotalChunks)) {

            RestfulServiceHelper.ReturnBadRequest("Invalid TotalChunks");

            return;

        }

        fileName = HttpUtility.UrlDecode(fileName).Trim();

        if (fileName.Length == 0) {

            RestfulServiceHelper.ReturnBadRequest("Invalid FileName");

            return;

        }

        try {

            fileType = HttpUtility.UrlDecode(fileType);

            eFileType = (FileType) Enum.Parse(typeof(FileType), fileType, true);

        }

        catch {

            RestfulServiceHelper.ReturnBadRequest("Invalid FileType");

            return;

        }

 

        try {

            // Process the uplaoad

            MediaUploadHandler.ProcessUpload(fileKey, fileName, iChunkNumber, iTotalChunks, data, eFileType);

        }

        catch (Exception ex) {

            RestfulServiceHelper.ReturnServerError(ex.Message);

        }

    }

 

    private enum FileType

    {

        Other = 0,

        Image = 1,

        Video = 2,

    }

 

    private readonly static MediaUploadHandler _uploadHandler = new MediaUploadHandler();

}

That’s it! All you had to do was take care of parsing the input, and UploadHandler takes care of the rest.

Source Code

Get the Source Code for this article here. It also includes the Microsoft.ServiceModel.Web library from the WCF Rest Starter Kit.

About these ads

Actions

Information

8 responses

20 03 2009
Restful Uploader and Extenders Update « Zleek Speak

[…] and Extenders Update 20 03 2009 There have been some slight modifications made to both the RESTful Uploader and Silverlight Extenders libraries. In addition, I have combined them into a single […]

31 07 2009
Martin Ortiz

I tried opening solution in VS 2008, and it complained that the sub projects in solution were not supported(?)

I tried opening them up individually and got same error

“The project type is not supported by this installation”

31 07 2009
zleek

Probably because it’s a Silverlight 2 project — I’ll upload the Silverlight 3 version shortly. Sorry about that!

23 09 2009
zleek

I apologize — that was a long “shortly”. I’ve updated the library with the final production code.

28 12 2009
Muthu Vijayan

Hi Good Try ! Some what helpful.

7 01 2010
Denis Vuyka

The article seems to be nice but sample code is not well prepared to my greatest regret. I would expect the raw and working functionality to be demonstrated instead of the 10 projects that require additional configuration on Vista/7 machines like IIS6 Configuration, Windows Authentication, virtual directories and elevated rights for Visual Studio.
My suggestion is that you clean up the mess in the solution and remove the projects that are not related to this very article.

7 01 2010
zleek

Thanks for the feedback. When I updated the code last month I forgot to remove the other projects that demo’d other functions in the Agnition.Utils libraries.

Here’s what I did:
– Removed projects that had nothing to do with uploader
– Removed classes from the utils libraries that were unnecessary for the uploader
– Switched the web projects to use Cassini instead of IIS

Hope this helps.

21 03 2011
zleek

Yes – on the server side just remove the pieces that manipulate the image and instead just save the received file to disk.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




Follow

Get every new post delivered to your Inbox.

%d bloggers like this: