current community

your communities

more stack exchange communities

company blog
Stack Exchange Inbox Reputation and Badges
tour help
Join the Stack Overflow Community
Stack Overflow is a community of 6.7 million programmers, just like you, helping each other.
Join them; it only takes a minute:
Sign up
up vote 8 down vote favorite
6

I have read a number of closely related questions but not one that hits this exactly. If it is a duplicate, please send me a link.

I am using an angular version of the flowjs library for doing HTML5 file uploads (https://github.com/flowjs/ng-flow). This works very well and I am able to upload multiple files simultaneously in 1MB chunks. There is an ASP.Net Web API Files controller that accepts these and saves them on disk. Although I can make this work, I am not doing it efficiently and would like to know a better approach.

First, I used the MultipartFormDataStreamProvider in an async method that worked great as long as the file uploaded in a single chunk. Then I switched to just using the FileStream to write the file to disk. This also worked as long as the chunks arrived in order, but of course, I cannot rely on that.

Next, just to see it work, I wrote the chunks to individual file streams and combined them after the fact, hence the inefficiency. A 1GB file would generate a thousand chunks that needed to be read and rewritten after the upload was complete. I could hold all file chunks in memory and flush them after they are all uploaded, but I'm afraid the server would blow up.

It seems that there should be a nice asynchronous solution to this dilemma but I don't know what it is. One possibility might be to use async/await to combine previous chunks while writing the current chunk. Another might be to use Begin/EndInvoke to create a separate thread so that the file manipulation on disk was handled independent of the thread reading from the HttpContext but this would rely on the ThreadPool and I'm afraid that the created threads will be unduly terminated when my MVC controller returns. I could create a FileWatcher that ran completely independent of ASP.Net but that would be very kludgey.

So my questions are, 1) is there a simple solution already that I am missing? (seems like there should be) and 2) if not, what is the best approach to solving this inside the Web API framework?

Thanks, bob

share|improve this question
up vote 8 down vote accepted

I'm not familiar with that kind of chunked upload, but I believe this should work:

  • Use flowTotalSize to pre-allocate the file when the first chunk comes in.
  • Have one SemaphoreSlim per file to serialize the asynchronous writes for that file.
  • Each chunk will write to its own offset (flowChunkSize * (flowChunkNumber - 1)) within the file.

This doesn't handle situations where the uploads are unexpectedly terminated. That kind of solution usually involves allocating/writing a temporary file (with a special extension) and then moving/renaming that file once the last chunk arrives.

Don't forget to ensure that your file writing is actually asynchronous.

share|improve this answer
    
That sounds perfect! I didn't know that could be done. Hopefully, I can get back to this code this afternoon and try this out. Thanks! – bob May 7 '14 at 15:32
    
I tried this but it appears that you cannot write chunks out of order without throwing an exception. Jon Skeet says as much in this question tinyurl.com/kelqk9k. – bob May 13 '14 at 16:42
    
It's not possible to insert. What I'm suggesting is overwrite. It should work. – Stephen Cleary May 13 '14 at 17:36
    
Sorry, I forgot to come back and mark this as a correct answer. It worked great when I implemented correctly. – bob Jun 16 '14 at 13:42
2  
@Bob - Trying to do the same thing here. Is your successful solution available publicly? Could use some sample code to help me get started. – Herb Caudill Aug 24 '14 at 20:25
up vote 6 down vote

Using @Stephen Cleary's answer, and this thread: https://github.com/flowjs/ng-flow/issues/41 I was able to make an ASP.NET Web Api Implementation and uploaded it for those still wondering about this question such as @Herb Caudill

https://github.com/samhowes/NgFlowSample/tree/master.

The original answer is the real answer to this question, but I don't have enough reputation yet to comment. I did not use a SemaphoreSlim, but instead enabled file Write sharing. But did in fact pre-allocate and make sure that each chunk is getting written to the right location by calculating an offset.

I will be contributing this to the Flow samples at: https://github.com/flowjs/flow.js/tree/master/samples

share|improve this answer
    
FWIW, "I don't have enough reputation yet" isn't a valid reason. Link only answers are also discouraged. But thanks for helping; our rules take a while to get used to. – Veedrac Oct 1 '14 at 13:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.