GZip content not automatically being deflated

[KNejad]

When making a HTTParty GZipped request with Accept-Encoding set to gzip, deflate I would expect HTTParty to automatically un-GZip the response. What I am seeing is content which is still GZipped.

To reproduce

Create HTTParty get request with Accept-Encoding set to gzip, deflate as below

r = HTTParty.get("https://example.com", {headers: { "Accept-Encoding" => "gzip, deflate"}})
r.body => # Still GZipped content

Expected result

r = HTTParty.get("https://example.com", {headers: { "Accept-Encoding" => "gzip, deflate"}})
r.body => # Plain text content

This is related to post I created on campaignmonitor/createsend-ruby#58 where they are doing the above.

Edit: The response body has to be over 1KB for the compressions to kick in. Which is where the bug appears

[jnunemaker]

What version of ruby and httparty are you using? GZip should be handled by net/http. HTTParty removed its decompression logic because of this. Any of that trigger some thoughts on your end?

[jackross]

Seeing exactly the same issue.

ruby 2.4.2p198
createsend (4.1.1)
httparty (0.15.6)

FYI, if you remove the header "Accept-Encoding"=>"gzip, deflate" from createsend's generated request headers and then perform the request, you'll get back an uncompressed response body.

i.e.

require 'createsend'

AUTH = { api_key: ENV['API_KEY'] }
client_id = ENV['CLIENT_ID']

client = CreateSend::Client.new AUTH, client_id

response = client.send(:get, 'campaigns')
response.body => # Still GZipped content

request = response.request
request.options.merge!(headers: request.options[:headers].tap { |h| h.delete "Accept-Encoding" })
request.perform.response.body => # Unzipped content

Could it be a Campaign Monitor problem, perhaps? Not well-versed enough in this stuff to know.

[KNejad]

@jnunemaker I think @jackross is right. This seems to be an issue with createsend and not httparty.

Create Send probably shouldn't be handling the compression since net/http does it by itself.

For the record here is my version info:

ruby: 2.4.1
httparty: 0.15.6
createsend: 4.1.1

I will leave this issue open for you to determine if it should be closed since I am not entirely sure.

[liu33173847]

Hello @jackross,

My Ruby and Createsend Gem version as below:

ruby: 2.4.2p198 
httparty: 0.15.6
createsend: 4.1.1

I could not reproduce the issue due to identical response body received with the code you provided, here are my code and the output:

require 'createsend'

AUTH = { api_key: ENV['API_KEY'] }
client_id = ENV['CLIENT_ID']

client = CreateSend::Client.new AUTH, client_id

response = client.send(:get, 'lists')

p"response.body.encoding: #{response.body.encoding}" # UTF-8
p"response.body: #{response.body}" # [{\"ListID\":\"12345679\",\"Name\":\"test\"}]

request = response.request
request.options.merge!(headers: request.options[:headers].tap { |h| h.delete "Accept-Encoding" })
request.perform.response.body 

p"request.perform.response.body.encoding: #{request.perform.response.body.encoding}" # UTF-8
p"request.perform.response.body: #{request.perform.response.body}" # [{\"ListID\":\"12345679\",\"Name\":\"test\"}]

Can you please double check?
Thanks

[KNejad]

@liu33173847 The response from Createsend has to be over 1KB. I forgot to mention that in my issue. I have updated it now.

[cmtiml]

Hello @KNejad,

We have merged the change for removing "Accept-Encoding"=>"gzip, deflate" and released it now for version 4.1.2 (https://rubygems.org/gems/createsend).

Please kindly check and verify if it solved the issue.
Thank you very much for your solution to the issue.

[mattprivman]

Removing Accept-Encoding only disables gzip, which is why that fix for createsend worked.

I think this is still a problem with HTTParty. I can reproduce it on ruby 2.4 and httparty 0.15.6.

[jnunemaker]

I am definitely seeing this on 2.4 with the examples provided. When I add handle_deflation back it fixes the problem so I am inclined to do that. I'll try to get a failing spec and PR soon.

fyi @jaypatrickhoward
xref #541

[dvodvo]

I am experienceing this with ruby 2.3 and httparty v 0.16.2. But I am lost a bit here... what is the recommended solution for deflation?

[BookOfGreg]

Still running into this issue. Not just with CreateSend, but also Azure APIM will not be un-zipped correctly when you send the request with 'Accept-Encoding' => 'gzip,deflate',.

@dvodvo If you DONT send the 'Accept-Encoding', header then it DOES work correctly and is gzipped.

[kellynabours]

Also just had this issue.
Using with emaildirect gem, which uses httparty with 'Accept-Encoding' => 'gzip,deflate'.
Ruby 2.3.4
Httparty 0.16.2 (also tried with 0.15.6, and 0.15.5).
emaildirect 2.0.0

[philipfong]

I'm running into this issue as well. When rolling back all the way to 0.14.0 I do not run into this problem. Httparty 0.16.3 and ruby 2.5.1

[TheSmartnik]

Okay. Here is what is happening.
If you don't specify Accept-Encoding net/http by defaults adds default header

gzip;q=1.0,deflate;q=0.6,identity;q=0.3

But if you do ruby assumes that you want to decode response yourself and does nothing.
HTTParty since version this fix follows the same idea.

It's counter intuitive, confusing and not user friendly.
However, I'm not sure what a correct behavior would be. I'm thinking that maybe when user adds gzip,deflate, we should decode_content anyway, but for other cases leave it as is.
@jnunemaker would love to hear your opinion on this