Class Rightscale::HttpConnection
In: lib/right_http_connection.rb
Parent: Object
HttpConnection lib/right_http_connection.rb Rightscale dot/m_1_1.png

HttpConnection maintains a persistent HTTP connection to a remote server. Each instance maintains its own unique connection to the HTTP server. HttpConnection makes a best effort to receive a proper HTTP response from the server, although it does not guarantee that this response contains a HTTP Success code.

On low-level errors (TCP/IP errors) HttpConnection invokes a reconnect and retry algorithm. Note that although each HttpConnection object has its own connection to the HTTP server, error handling is shared across all connections to a server. For example, if there are three connections to www.somehttpserver.com, a timeout error on one of those connections will cause all three connections to break and reconnect. A connection will not break and reconnect, however, unless a request becomes active on it within a certain amount of time after the error (as specified by HTTP_CONNECTION_RETRY_DELAY). An idle connection will not break even if other connections to the same server experience errors.

A HttpConnection will retry a request a certain number of times (as defined by HTTP_CONNNECTION_RETRY_COUNT). If all the retries fail, an exception is thrown and all HttpConnections associated with a server enter a probationary period defined by HTTP_CONNECTION_RETRY_DELAY. If the user makes a new request subsequent to entering probation, the request will fail immediately with the same exception thrown on probation entry. This is so that if the HTTP server has gone down, not every subsequent request must wait for a connect timeout before failing. After the probation period expires, the internal state of the HttpConnection is reset and subsequent requests have the full number of potential reconnects and retries available to them.

Methods

Constants

HTTP_CONNECTION_RETRY_COUNT = 3   Number of times to retry the request after encountering the first error
HTTP_CONNECTION_OPEN_TIMEOUT = 5   Throw a Timeout::Error if a connection isn‘t established within this number of seconds
HTTP_CONNECTION_READ_TIMEOUT = 120   Throw a Timeout::Error if no data have been read on this connnection within this number of seconds
HTTP_CONNECTION_RETRY_DELAY = 15   Length of the post-error probationary period during which all requests will fail

Attributes

http  [RW] 
logger  [RW] 
params  [RW] 
server  [RW] 

Public Class methods

Params hash:

 :user_agent => 'www.HostName.com'    # String to report as HTTP User agent
 :ca_file    => 'path_to_file'        # A path of a CA certification file in PEM format. The file can contain several CA certificates.
 :logger     => Logger object         # If omitted, HttpConnection logs to STDOUT
 :exception  => Exception to raise    # The type of exception to raise if a request repeatedly fails. RuntimeError is raised if this parameter is omitted.
 :http_connection_retry_count         # by default == Rightscale::HttpConnection.params[:http_connection_retry_count]
 :http_connection_open_timeout        # by default == Rightscale::HttpConnection.params[:http_connection_open_timeout]
 :http_connection_read_timeout        # by default == Rightscale::HttpConnection.params[:http_connection_read_timeout]
 :http_connection_retry_delay         # by default == Rightscale::HttpConnection.params[:http_connection_retry_delay]

[Source]

     # File lib/right_http_connection.rb, line 137
137:     def initialize(params={})
138:       @params = params
139:       @params[:http_connection_retry_count]  ||= @@params[:http_connection_retry_count]
140:       @params[:http_connection_open_timeout] ||= @@params[:http_connection_open_timeout]
141:       @params[:http_connection_read_timeout] ||= @@params[:http_connection_read_timeout]
142:       @params[:http_connection_retry_delay]  ||= @@params[:http_connection_retry_delay]
143:       @http   = nil
144:       @server = nil
145:       @logger = get_param(:logger) ||
146:                 (RAILS_DEFAULT_LOGGER if defined?(RAILS_DEFAULT_LOGGER)) ||
147:                 Logger.new(STDOUT)
148:     end

Query the global (class-level) parameters:

 :user_agent => 'www.HostName.com'    # String to report as HTTP User agent
 :ca_file    => 'path_to_file'        # Path to a CA certification file in PEM format. The file can contain several CA certificates.  If this parameter isn't set, HTTPS certs won't be verified.
 :logger     => Logger object         # If omitted, HttpConnection logs to STDOUT
 :exception  => Exception to raise    # The type of exception to raise
                                      # if a request repeatedly fails. RuntimeError is raised if this parameter is omitted.
 :http_connection_retry_count         # by default == Rightscale::HttpConnection::HTTP_CONNECTION_RETRY_COUNT
 :http_connection_open_timeout        # by default == Rightscale::HttpConnection::HTTP_CONNECTION_OPEN_TIMEOUT
 :http_connection_read_timeout        # by default == Rightscale::HttpConnection::HTTP_CONNECTION_READ_TIMEOUT
 :http_connection_retry_delay         # by default == Rightscale::HttpConnection::HTTP_CONNECTION_RETRY_DELAY

[Source]

     # File lib/right_http_connection.rb, line 110
110:     def self.params
111:       @@params
112:     end

Set the global (class-level) parameters

[Source]

     # File lib/right_http_connection.rb, line 115
115:     def self.params=(params)
116:       @@params = params
117:     end

Public Instance methods

[Source]

     # File lib/right_http_connection.rb, line 415
415:     def finish(reason = '')
416:       if @http && @http.started?
417:         reason = ", reason: '#{reason}'" unless reason.blank?
418:         @logger.info("Closing #{@http.use_ssl? ? 'HTTPS' : 'HTTP'} connection to #{@http.address}:#{@http.port}#{reason}")
419:         @http.finish
420:       end
421:     end

[Source]

     # File lib/right_http_connection.rb, line 150
150:     def get_param(name)
151:       @params[name] || @@params[name]
152:     end

Set the maximum size (in bytes) of a single read from local data sources like files. This can be used to tune the performance of, for example, a streaming PUT of a large buffer.

[Source]

     # File lib/right_http_connection.rb, line 178
178:     def local_read_size=(newsize)
179:       Net::HTTPGenericRequest.local_read_size=(newsize)
180:     end

Query for the maximum size (in bytes) of a single read from local data sources like files. This is important, for example, in a streaming PUT of a large buffer.

[Source]

     # File lib/right_http_connection.rb, line 171
171:     def local_read_size?
172:       Net::HTTPGenericRequest.local_read_size?
173:     end

Send HTTP request to server

 request_params hash:
 :server   => 'www.HostName.com'   # Hostname or IP address of HTTP server
 :port     => '80'                 # Port of HTTP server
 :protocol => 'https'              # http and https are supported on any port
 :request  => 'requeststring'      # Fully-formed HTTP request to make

Raises RuntimeError, Interrupt, and params[:exception] (if specified in new).

[Source]

     # File lib/right_http_connection.rb, line 327
327:     def request(request_params, &block)
328:       # We save the offset here so that if we need to retry, we can return the file pointer to its initial position
329:       mypos = get_fileptr_offset(request_params)
330:       loop do
331:         # if we are inside a delay between retries: no requests this time!
332:         if error_count > @params[:http_connection_retry_count] &&
333:            error_time + @params[:http_connection_retry_delay] > Time.now
334:           # store the message (otherwise it will be lost after error_reset and
335:           # we will raise an exception with an empty text)
336:           banana_message_text = banana_message
337:           @logger.warn("#{err_header} re-raising same error: #{banana_message_text} " +
338:                       "-- error count: #{error_count}, error age: #{Time.now.to_i - error_time.to_i}")
339:           exception = get_param(:exception) || RuntimeError
340:           raise exception.new(banana_message_text)
341:         end
342: 
343:         # try to connect server(if connection does not exist) and get response data
344:         begin
345:           request_params[:protocol] ||= (request_params[:port] == 443 ? 'https' : 'http')
346: 
347:           request = request_params[:request]
348:           request['User-Agent'] = get_param(:user_agent) || ''
349: 
350:           # (re)open connection to server if none exists or params has changed
351:           unless @http          &&
352:                  @http.started? &&
353:                  @server   == request_params[:server] &&
354:                  @port     == request_params[:port]   &&
355:                  @protocol == request_params[:protocol]
356:             start(request_params)
357:           end
358: 
359:           # Detect if the body is a streamable object like a file or socket.  If so, stream that
360:           # bad boy.
361:           setup_streaming(request)
362:           response = @http.request(request, &block)
363: 
364:           error_reset
365:           eof_reset
366:           return response
367: 
368:         # We treat EOF errors and the timeout/network errors differently.  Both
369:         # are tracked in different statistics blocks.  Note below that EOF
370:         # errors will sleep for a certain (exponentially increasing) period.
371:         # Other errors don't sleep because there is already an inherent delay
372:         # in them; connect and read timeouts (for example) have already
373:         # 'slept'.  It is still not clear which way we should treat errors
374:         # like RST and resolution failures.  For now, there is no additional
375:         # delay for these errors although this may change in the future.
376: 
377:         # EOFError means the server closed the connection on us.
378:         rescue EOFError => e
379:           @logger.debug("#{err_header} server #{@server} closed connection")
380:           @http = nil
381: 
382:             # if we have waited long enough - raise an exception...
383:           if raise_on_eof_exception?
384:             exception = get_param(:exception) || RuntimeError
385:             @logger.warn("#{err_header} raising #{exception} due to permanent EOF being received from #{@server}, error age: #{Time.now.to_i - eof_time.to_i}")
386:             raise exception.new("Permanent EOF is being received from #{@server}.")
387:           else
388:               # ... else just sleep a bit before new retry
389:             sleep(add_eof)
390:             # We will be retrying the request, so reset the file pointer
391:             reset_fileptr_offset(request, mypos)
392:           end
393:         rescue Exception => e  # See comment at bottom for the list of errors seen...
394:           @http = nil
395:           # if ctrl+c is pressed - we have to reraise exception to terminate proggy
396:           if e.is_a?(Interrupt) && !( e.is_a?(Errno::ETIMEDOUT) || e.is_a?(Timeout::Error))
397:             @logger.debug( "#{err_header} request to server #{@server} interrupted by ctrl-c")
398:             raise
399:           elsif e.is_a?(ArgumentError) && e.message.include?('wrong number of arguments (5 for 4)')
400:             # seems our net_fix patch was overriden...
401:             exception = get_param(:exception) || RuntimeError
402:             raise exception.new('incompatible Net::HTTP monkey-patch')
403:           end
404:           # oops - we got a banana: log it
405:           error_add(e.message)
406:           @logger.warn("#{err_header} request failure count: #{error_count}, exception: #{e.inspect}")
407: 
408:           # We will be retrying the request, so reset the file pointer
409:           reset_fileptr_offset(request, mypos)
410: 
411:         end
412:       end
413:     end

Set the maximum size (in bytes) of a single read from the underlying socket. For bulk transfer, especially over fast links, this is value is critical to performance.

[Source]

     # File lib/right_http_connection.rb, line 164
164:     def socket_read_size=(newsize)
165:       Net::BufferedIO.socket_read_size=(newsize)
166:     end

Query for the maximum size (in bytes) of a single read from the underlying socket. For bulk transfer, especially over fast links, this is value is critical to performance.

[Source]

     # File lib/right_http_connection.rb, line 157
157:     def socket_read_size?
158:       Net::BufferedIO.socket_read_size?
159:     end

Private Instance methods

Adds new EOF timestamp. Returns the number of seconds to wait before new conection retry:

 0.5, 1, 2, 4, 8

[Source]

     # File lib/right_http_connection.rb, line 226
226:     def add_eof
227:       (@@eof[@server] ||= []).unshift Time.now
228:       0.25 * 2 ** @@eof[@server].size
229:     end

Error message stuff…

[Source]

     # File lib/right_http_connection.rb, line 215
215:     def banana_message
216:       return "#{@server} temporarily unavailable: (#{error_message})"
217:     end

Reset a list of EOFs for this server. This is being called when we have got an successful response from server.

[Source]

     # File lib/right_http_connection.rb, line 244
244:     def eof_reset
245:       @@eof.delete(@server)
246:     end

Returns first EOF timestamp or nul if have no EOFs being tracked.

[Source]

     # File lib/right_http_connection.rb, line 232
232:     def eof_time
233:       @@eof[@server] && @@eof[@server].last
234:     end

[Source]

     # File lib/right_http_connection.rb, line 219
219:     def err_header
220:       return "#{self.class.name} :"
221:     end

add an error for a server

[Source]

     # File lib/right_http_connection.rb, line 205
205:     def error_add(message)
206:       @@state[@server] = { :count => error_count+1, :time => Time.now, :message => message }
207:     end

number of consecutive errors seen for server, 0 all is ok

[Source]

     # File lib/right_http_connection.rb, line 190
190:     def error_count
191:       @@state[@server] ? @@state[@server][:count] : 0
192:     end

message for last error for server, "" if all is ok

[Source]

     # File lib/right_http_connection.rb, line 200
200:     def error_message
201:       @@state[@server] ? @@state[@server][:message] : ""
202:     end

reset the error state for a server (i.e. a request succeeded)

[Source]

     # File lib/right_http_connection.rb, line 210
210:     def error_reset
211:       @@state.delete(@server)
212:     end

time of last error for server, nil if all is ok

[Source]

     # File lib/right_http_connection.rb, line 195
195:     def error_time
196:       @@state[@server] && @@state[@server][:time]
197:     end

[Source]

     # File lib/right_http_connection.rb, line 258
258:     def get_fileptr_offset(request_params)
259:       request_params[:request].body.pos
260:     rescue Exception => e
261:       # Probably caught this because the body doesn't support the pos() method, like if it is a socket.
262:       # Just return 0 and get on with life.
263:       0
264:     end

Returns true if we are receiving EOFs during last @params[:http_connection_retry_delay] seconds and there were no successful response from server

[Source]

     # File lib/right_http_connection.rb, line 238
238:     def raise_on_eof_exception?
239:       @@eof[@server].blank? ? false : ( (Time.now.to_i-@params[:http_connection_retry_delay]) > @@eof[@server].last.to_i )
240:     end

[Source]

     # File lib/right_http_connection.rb, line 266
266:     def reset_fileptr_offset(request, offset = 0)
267:       if(request.body_stream && request.body_stream.respond_to?(:pos))
268:         begin
269:           request.body_stream.pos = offset
270:         rescue Exception => e
271:           @logger.warn("Failed file pointer reset; aborting HTTP retries." +
272:                              " -- #{err_header} #{e.inspect}")
273:           raise e
274:         end
275:       end
276:     end

Detects if an object is ‘streamable’ - can we read from it, and can we know the size?

[Source]

     # File lib/right_http_connection.rb, line 249
249:     def setup_streaming(request)
250:       if(request.body && request.body.respond_to?(:read))
251:         body = request.body
252:         request.content_length = body.respond_to?(:lstat) ? body.lstat.size : body.size
253:         request.body_stream = request.body
254:         true
255:       end
256:     end

Start a fresh connection. The object closes any existing connection and opens a new one.

[Source]

     # File lib/right_http_connection.rb, line 280
280:     def start(request_params)
281:       # close the previous if exists
282:       finish
283:       # create new connection
284:       @server   = request_params[:server]
285:       @port     = request_params[:port]
286:       @protocol = request_params[:protocol]
287: 
288:       @logger.info("Opening new #{@protocol.upcase} connection to #@server:#@port")
289:       @http = Net::HTTP.new(@server, @port)
290:       @http.open_timeout = @params[:http_connection_open_timeout]
291:       @http.read_timeout = @params[:http_connection_read_timeout]
292: 
293:       if @protocol == 'https'
294:         verifyCallbackProc = Proc.new{ |ok, x509_store_ctx|
295:           code = x509_store_ctx.error
296:           msg = x509_store_ctx.error_string
297:             #debugger
298:           @logger.warn("##### #{@server} certificate verify failed: #{msg}") unless code == 0
299:           true
300:         }
301:         @http.use_ssl = true
302:         ca_file = get_param(:ca_file)
303:         if ca_file
304:           @http.verify_mode     = OpenSSL::SSL::VERIFY_PEER
305:           @http.verify_callback = verifyCallbackProc
306:           @http.ca_file         = ca_file
307:         end
308:       end
309:       # open connection
310:       @http.start
311:     end

[Validate]