How to disable proxy

[meridius]

Hi there

I found out that Faraday is being used in the Inspec framework https://github.com/inspec/inspec for making HTTP requests.

My question is: How can I make Faraday ignore the global proxy settings (the http_proxy env var).

Since I just found out about Faraday, I'm fairly new to Ruby and my use case is to make working Inspec test so I don't even know how to test it otherwise; I made a custom Inspec resource (copy&paste of https://github.com/inspec/inspec/blob/master/lib/inspec/resources/http.rb#L142).

NOTE: The initial module Inspec::Resources and the last end needs to be removed from http.rb for Inspec to register it as new resource. Also new class name and name should be given to that resource.

I tried setting Faraday.ignore_env_proxy = true according to #752 but that didn't help.

I also tried Faraday.new(url, :proxy => '') and conn.proxy='' but that didn't work either.

The only thing that works is to remove the ENV variable for whole Inspec by http_proxy= inspec exec /tests.

Could you please advise what am I missing? I'd really want to be able to disable the proxy for specific calls instead.

[iMacTia]

Hi @meridius, I did some digging and found that Faraday is indeed ignoring the env proxy, but Net::HTTP is still picking it automatically 🤔
https://ruby-doc.org/stdlib-2.7.1/libdoc/net/http/rdoc/Net/HTTP.html#class-Net::HTTP-label-Proxies

That is clearly unwanted and I agree that Faraday should be forcing Net::HTTP not to do so if possible.

In the meantime, if you need to disable it just for some calls, you could set ENV['http_proxy'] to nil before the request and then set it back after it, but I agree that's not an ideal solution!

[meridius]

Hi @iMacTia, I thought I give Discussions a try. ;) Thank you for the speedy reply.

Your proposed workaround would be really fine - provided it worked.
I tried following, but it didn't help, unfortunately.

original_proxy = ENV['http_proxy']
ENV['http_proxy'] = nil
host_result = host(host)
ENV['http_proxy'] = original_proxy

Could you please tell me, whether it should be combined with Faraday.ignore_env_proxy = true or any other approaches?

[iMacTia]

It shouldn't need to be combined actually! I'm surprised it still doesn't work to be honest.
Anyway, I've just got a PR merged that fixes this issue (#1221) 🎉
If you want to give it a try before the next release, it's already in master 👍

[iMacTia]

Oh BTW, is your host called in http or https?

[meridius]

OK, I just verified that it doesn't help to set Faraday.ignore_env_proxy = true and/or conn.proxy='' ... before reading your message, oh well.

I'll gladly try the master, once I figure out how to integrate it to Inspec, but that's my problem.

As for your question - the related hosts are HTTP.

[meridius]

I can confirm that it finally works! Thank you @iMacTia!

To integrate it to Inspec I just had to replace contents of Faraday libs in Inspec (/usr/local/bundle/gems in https://hub.docker.com/r/chef/inspec)

• faraday-0.17.3 for faraday-1.2.0
• faraday_middleware-0.12.2 for faraday_middleware-1.0.0

and had to create custom library myhttp by copy&pasting the existing https://github.com/inspec/inspec/blob/master/lib/inspec/resources/http.rb and making some tweaks:

libraries/myhttp.rb

# copyright: 2017, Criteo
# copyright: 2017, Chef Software Inc
# license: Apache v2

require "inspec/resources/command"
require "faraday" unless defined?(Faraday)
require "faraday_middleware"
require "hashie"

class MyHttp < Inspec.resource(1)
    name "myhttp"
    supports platform: "unix"
    desc "Use the http InSpec audit resource to test http call."
    example <<~EXAMPLE
        # To disable proxy, set it to empty string.
        describe myhttp('http://localhost:8080/ping', auth: {user: 'user', pass: 'test'}, params: {format: 'html'}, proxy: 'http://foo:8080') do
            its('status') { should cmp 200 }
            its('body') { should cmp 'pong' }
            its('headers.Content-Type') { should cmp 'text/html' }
        end

        describe myhttp('http://example.com/ping').headers do
            its('Content-Length') { should cmp 258 }
            its('Content-Type') { should cmp 'text/html; charset=UTF-8' }
        end
    EXAMPLE

    def initialize(url, opts = {})
        @url = url
        @opts = opts

        # Prior to InSpec 2.0 the HTTP test had to be instructed to run on the
        # remote target machine. This warning will be removed after a few months
        # to give users an opportunity to remove the unused option from their
        # profiles.
        if opts.key?(:enable_remote_worker) && !inspec.local_transport?
            warn "Ignoring `enable_remote_worker` option, the `http` resource ",
                     "remote worker is enabled by default for remote targets and ",
                     "cannot be disabled"
        end

        # Run locally if InSpec is ran locally and remotely if ran remotely
        if inspec.local_transport?
            @worker = Worker::Local.new(http_method, url, opts)
        else
            @worker = Worker::Remote.new(inspec, http_method, url, opts)
        end
    end

    def status
        @worker.status
    end

    def headers
        @headers ||= Inspec::Resources::Http::Headers.create(@worker.response_headers)
    end

    def body
        @worker.body
    end

    def http_method
        @opts.fetch(:method, "GET")
    end

    def to_s
        if @opts && @url
            "HTTP #{http_method} on #{@url}"
        else
            "HTTP Resource"
        end
    end

    class Worker
        class Base
            attr_reader :http_method, :opts, :url

            def initialize(http_method, url, opts)
                @http_method = http_method
                @url = url
                @opts = opts
                @response = nil
            end

            private

            def params
                opts.fetch(:params, nil)
            end

            def proxy
                opts.fetch(:proxy, nil)
            end

            def username
                opts.fetch(:auth, {})[:user]
            end

            def password
                opts.fetch(:auth, {})[:pass]
            end

            def request_headers
                opts.fetch(:headers, {})
            end

            def request_body
                opts[:data]
            end

            def open_timeout
                opts.fetch(:open_timeout, 60)
            end

            def read_timeout
                opts.fetch(:read_timeout, 60)
            end

            def ssl_verify?
                opts.fetch(:ssl_verify, true)
            end

            def max_redirects
                opts.fetch(:max_redirects, 0)
            end
        end

        class Local < Base
            def status
                response.status
            end

            def body
                response.body
            end

            def response_headers
                response.headers.to_h
            end

            private

            def response
                return @response if @response

                conn = Faraday.new(url: url, headers: request_headers, params: params, ssl: { verify: ssl_verify? }) do |builder|
                    builder.request :url_encoded
                    builder.use FaradayMiddleware::FollowRedirects, limit: max_redirects if max_redirects > 0
                    builder.adapter Faraday.default_adapter
                end

                # set proxy
                # This has to be set either to nil or "".
                # "ENV['http_proxy'] = nil" doesn't have any effect.
                conn.proxy = proxy unless proxy.nil?

                # set basic authentication
                conn.basic_auth username, password unless username.nil? || password.nil?

                # set default timeout
                conn.options.timeout      = read_timeout  # open/read timeout in seconds
                conn.options.open_timeout = open_timeout  # connection open timeout in seconds

                @response = conn.run_request(http_method.downcase.to_sym, nil, nil, nil) do |req|
                    req.body = request_body
                end
            end
        end

        class Remote < Base
            attr_reader :inspec

            def initialize(inspec, http_method, url, opts)
                unless inspec.command("curl").exist?
                    raise Inspec::Exceptions::ResourceSkipped,
                                "curl is not available on the target machine"
                end

                @ran_curl = false
                @inspec = inspec
                super(http_method, url, opts)
            end

            def status
                run_curl
                @status
            end

            def body
                run_curl
                @body&.strip
            end

            def response_headers
                run_curl
                @response_headers
            end

            private

            def run_curl
                return if @ran_curl

                cmd_result = inspec.command(curl_command)
                response = cmd_result.stdout
                @ran_curl = true
                return if response.nil? || cmd_result.exit_status != 0

                # strip any carriage returns to normalize output
                response.delete!("\r")

                # split the prelude (status line and headers) and the body
                prelude, remainder = response.split("\n\n", 2)
                loop do
                    break unless remainder =~ %r{^HTTP/}

                    prelude, remainder = remainder.split("\n\n", 2)
                end
                @body = remainder
                prelude = prelude.lines

                # grab the status off of the first line of the prelude
                status_line = prelude.shift
                @status = status_line.split(" ", 3)[1].to_i

                # parse the rest of the prelude which will be all the HTTP headers
                @response_headers = {}
                prelude.each do |line|
                    line.strip!
                    key, value = line.split(":", 2)
                    @response_headers[key] = value.strip
                end
            end

            def curl_command # rubocop:disable Metrics/AbcSize
                cmd = []
                cmd << http_proxy=proxy unless proxy.nil?
                cmd << "curl -i"

                # Use curl's --head option when the method requested is HEAD. Otherwise,
                # the user may experience a timeout when curl does not properly close
                # the connection after the response is received.
                if http_method.casecmp("HEAD") == 0
                    cmd << "--head"
                else
                    cmd << "-X #{http_method}"
                end

                cmd << "--connect-timeout #{open_timeout}"
                cmd << "--max-time #{open_timeout + read_timeout}"
                cmd << "--user \'#{username}:#{password}\'" unless username.nil? || password.nil?
                cmd << "--insecure" unless ssl_verify?
                cmd << "--data #{Shellwords.shellescape(request_body)}" unless request_body.nil?
                cmd << "--location" if max_redirects > 0
                cmd << "--max-redirs #{max_redirects}" if max_redirects > 0

                request_headers.each do |k, v|
                    cmd << "-H '#{k}: #{v}'"
                end

                if params.nil?
                    cmd << "'#{url}'"
                else
                    cmd << "'#{url}?#{params.map { |e| e.join("=") }.join("&")}'"
                end

                cmd.join(" ")
            end
        end
    end

    class Headers < Hash
        def self.create(header_data)
            header_data.each_with_object(new) { |(k, v), memo| memo[k.to_s.downcase] = v }
        end

        def [](requested_key)
            fetch(requested_key.downcase, nil)
        end

        def method_missing(requested_key)
            fetch(requested_key.to_s.downcase, nil)
        end
    end
end

The most important one is to add conn.proxy = proxy unless proxy.nil? which allows to disable the proxy by defining its value as empty string myhttp(url, proxy: "").

I also learned that setting ENV['http_proxy'] to nil or "" doesn't have any effect.