WebRequest and Binary Data

Tuesday, October 5, 2004
Someone sent me an email today describing a problem when downloading a binary file with the WebRequest class. There are plenty of articles (including some of mine) with screen scraping code like the following:

WebRequest request = WebRequest.Create(someUrl);        
using(WebResponse response = request.GetResponse())
{
   using(StreamReader reader = new StreamReader(response.GetResponseStream()))
   {
      string result = reader.ReadToEnd();
   }         
}

Unfortunately, StreamReader is only good for reading text. When it comes to binary data the result has a good chance of being incomplete. The approach for binary data is to stick to the basic Stream type and read raw bytes.

byte[] result;
byte[] buffer = new byte[4096];
 
WebRequest wr = WebRequest.Create(someUrl);
 
using(WebResponse response = wr.GetResponse())
{
   using(Stream responseStream = response.GetResponseStream())
   {
      using(MemoryStream memoryStream = new MemoryStream())
      {
         int count = 0;
         do
         {
            count = responseStream.Read(buffer, 0, buffer.Length);
            memoryStream.Write(buffer, 0, count);
 
         } while(count != 0);
 
         result = memoryStream.ToArray();
 
      }
   }
}

P.S. IDisposable lurks everywhere!. It’s a shame some classes use an explicit interface implementation and hide the Dispose method from Intellisense.

P.P.S. Commercials are the best things going on Monday Night Football these days. Except the commercials for other ABC shows. I don't know why I turn on television.


Comments
Andy Wednesday, October 6, 2004
You know I've been working with Java for the last few days on a wireless http connection that downloads images and it looks almost identical to what you have above. The class names are a bit different but d@mn, I had no idea they were so syntactically close to each other.
<br>
<br>In Java I have to close the connections in a try finally block. Do you need to also close the C# ones or do they close automatically when they pass from scope?
Scott Allen Thursday, October 7, 2004
Yes - Java and C# are really close! The 'using' clause sets up a try catch finally block in C#. It's just a shortcut to use with a resource that has to be closed or disposed of even when an exception happens. If I had written it that way I bet it would have looked exactly like Java, eh?
Ren Wednesday, February 9, 2005
Downloading - that's easy. Now, try to *upload* files with just webrequest :P I've been 100% unable to find ANYthing remotely relating to that.
Scott Thursday, February 10, 2005
Interesting - I'll have to give it a try.
Joe Ehrenfeld Wednesday, April 6, 2005
I am using POST to get some binary data and when I try to write the data to a file I get this error:
<br>
<br>Additional information: Non-negative number required.
<br>
<br>What am I doing worng
Scott Wednesday, April 6, 2005
Hi Joe:
<br>
<br>It's hard to tell without seeing some code. Feel free to send me an email and I can take a look.
Joe Ehrenfeld Wednesday, April 6, 2005
I sent you an email with some code, but I don't know if I used the right address? Did you get an email from joseph.ehrenfeld?
Scott Wednesday, April 6, 2005
No - you can use scott @ OdeToCode .com
Joe Ehrenfeld Wednesday, April 6, 2005
Scott, I did send it to that address but our firewall may have blocked it. I'll paste the code here:
<br> //Joe Code
<br> ASCIIEncoding encodedData=new ASCIIEncoding();
<br> byte[] byteArray = encodedData.GetBytes(postData);
<br> Stream stream;
<br>
<br> byte[] result;
<br> byte[] buffer = new byte[4096];
<br>
<br> HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(url);
<br> wr.Method = &quot;POST&quot;;
<br> wr.ContentType = &quot;application/x-www-form-urlencoded&quot;;
<br> wr.ContentLength = byteArray.Length;
<br>
<br> stream = wr.GetRequestStream();
<br> stream.Write(byteArray, 0, byteArray.Length);
<br>
<br> stream.Flush();
<br> stream.Close();
<br>
<br> using(WebResponse response = wr.GetResponse())
<br> {
<br> Console.WriteLine(&quot;Headers: &quot; + response.Headers.ToString());
<br>
<br> using(Stream responseStream = response.GetResponseStream())
<br> {
<br> using(MemoryStream memoryStream = new MemoryStream())
<br> {
<br> int count = 0;
<br> do
<br> {
<br> count = responseStream.Read(buffer, 0, buffer.Length);
<br> memoryStream.Write(buffer, 0, count);
<br>
<br> } while(count != 0);
<br>
<br> result = memoryStream.ToArray();
<br> }
<br> }
<br> }
<br>
<br> //Console.ReadLine();
<br> }
Scott Wednesday, April 6, 2005
Seems odd, the code looks ok. Do you know what line the exception is thrown?
Joe Ehrenfeld Wednesday, April 6, 2005
yes, it throws an exception at the line:
<br>
<br>while(count != 0);
<br>
<br>What's odd is I know I have a response, but for some reason I just can't open that attachment and read it to save it onto my server.
Swee Friday, May 20, 2005
I believe the problem lies in these two lines:
<br>
<br>count = responseStream.Read(buffer, 0, buffer.Length);
<br>memoryStream.Write(buffer, 0, count);
<br>
<br>Read() returns -1 when it has no more data. Trying to subsequently do a Write() with count of -1 most likely caused the problem.
<br>
<br>/Swee
geoff Wednesday, March 8, 2006
Thanks for the code. Is there anyway to tell through C# if the file the user wants to download is text or binary? I've got a shopping cart that allows users to purchase downloadable items but I'm stuck on the page that actually delivers them. I don't want to give a direct url to the file for security, so I figured I'd stream the data, but I'm stuck on how to figure out the encoding of the original file. Any ideas? Thanks.
Marc Wednesday, May 17, 2006
I have an ASPX page with the following code. It's a variation of the above. The goal is to get a PDF from a URL and display in in Acrobat. This works in Firefox, but in IE the actual contects (binary) of the PDF display in the browser, they don't launch Acrobat.

Any Ideas would be appreciated!
Marc
Lampcov@aim.com

<%@ Import Namespace="System.Net" %>
<%@ Import Namespace="System.IO" %>

<script language="c#" runat="server" debug="true">

private void Page_Load(object sender, System.EventArgs e)
{
byte[] result;
byte[] buffer = new byte[4096];

WebRequest wr = WebRequest.Create("localhost:16200/?MRN=123&doc_type=pdf");
WebResponse response = wr.GetResponse();
Stream responseStream = response.GetResponseStream();
MemoryStream memoryStream = new MemoryStream();

Response.AddHeader("content-disposition", "filename=report.PDF");
Response.ContentType = "application/pdf";

int count = 0;
do
{
count = responseStream.Read(buffer, 0, buffer.Length);
memoryStream.Write(buffer, 0, count);
Response.BinaryWrite(buffer);
//Response.OutputStream.Write(buffer,0,count);
}
while (count != 0);
result = memoryStream.ToArray();
}

</script>
Comments are now closed.
by K. Scott Allen K.Scott Allen
My Pluralsight Courses
The Podcast!