Monday, May 3, 2010

How to Fix Net::HTTPBadResponse: wrong status line: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">" in Ruby

Using Ruby 1.8.7:
$ ruby -version
ruby 1.8.7 (2008-08-11 patchlevel 72) [universal-darwin10.0
If I do the following, you can see where it might be confusing that the error has nothing to do with me not specifying that I wanted to use HTTPS vs HTTP:
$ irb
>> require 'net/http'
=> true
>> Net::HTTP.get_response(URI.parse('http://www.google.com/'))
=> #<Net::HTTPOK 200 OK readbody=true>
>> Net::HTTP.get_response(URI.parse('https://www.verisign.com/'))
Net::HTTPBadResponse: wrong status line: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">"
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2022:in `read_status_line'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2009:in `read_new'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:1050:in `request'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:948:in `request_get'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:380:in `get_response'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:543:in `start'
 from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:379:in `get_response'
 from (irb):3
But if I change the code to specify I want to use HTTPS:
$ irb
>> require 'net/http'
=> true
>> require 'net/https'
=> true
>> url = URI.parse('https://www.verisign.com/')
=> #<URI::HTTPS:0x101200588 URL:https://www.verisign.com/>
>> http = Net::HTTP.new(url.host, url.port)
=> #<Net::HTTP www.verisign.com:443 open=false>
>> http.use_ssl = true
=> true
>> request = Net::HTTP::Get.new(url.path)
=> #<Net::HTTP::Get GET>
>> response = http.start {|http| http.request(request) }
warning: peer certificate won't be verified in this SSL session
=> #<Net::HTTPOK 200 OK readbody=true>
all is good!

This came about because an example for an HTTP GET in the Ruby standard documentation doesn't mention HTTPS, and it is only natural that at some point you'll try to use that same code to get something via SSL/HTTPS, and it will fail with a similar error to the above.

Here is an imperfect generic implementation of GET utility method based on the original:

  # copied from http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/classes/Net/HTTP.html then modified to support SSL
  def self.fetch(uri_str, limit = 10)
    # You should choose better exception.
    raise ArgumentError, 'HTTP redirect too deep' if limit == 0
    
    url = URI.parse(uri_str)      
    http = Net::HTTP.new(url.host, url.port)
    http.use_ssl = (url.scheme == 'https')
    request = Net::HTTP::Get.new(url.path)
    response = http.start {|http| http.request(request) }
      
    case response
    when Net::HTTPSuccess     then response
    when Net::HTTPRedirection then fetch(response['location'], limit - 1)
    else
      response.error!
    end
  end

3 comments:

Peter Degen-Portnoy said...

Thanks for posting this; it was the perfect solution to my oversight of not supporting HTTPS!

Alexandra said...

Thanks so much. After about an hour of googling, I had not come upon an answer until this post. It seems like this issue is relatively common and should be more well-documented.

Furia Bhavesh said...

I am also getting a similar error but only when I am posting very long data (1 lakh characters) via rails form. Also, is there any config to be made on server side to make https work ?