Bug 60362 - Missing reason phrase in response
Summary: Missing reason phrase in response
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 8
Classification: Unclassified
Component: Catalina (show other bugs)
Version: 8.5.x-trunk
Hardware: PC All
: P2 enhancement with 32 votes (vote)
Target Milestone: ----
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
: 60183 60618 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-11-11 03:43 UTC by Todd Pierce
Modified: 2021-01-28 16:54 UTC (History)
6 users (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Todd Pierce 2016-11-11 03:43:53 UTC
Tomcat 8.5 removed the reason phrase from the response. This breaks legacy clients. Can it not be brought back as a config option?
Comment 1 Remy Maucherat 2016-11-11 08:16:35 UTC
No.
Comment 2 Christopher Schultz 2016-11-11 18:06:01 UTC
I would argue that Tomcat 8.0.x should not have removed the reason phrase as that represents a fairly big change. Removal from 8.5.x is okay with me.
Comment 3 Todd Pierce 2016-11-14 00:34:03 UTC
Reinstating the reason phrase to 8.0.x would be a good outcome.
Comment 4 Todd Pierce 2016-11-18 00:01:58 UTC
I suggest that the spec for HTTTP 1.1 is ambiguous regarding whether the reason phrase is optional at all. It says that the reson phrase is defined as zero or more text characters. This could be interpreted to mean, by example, So there -could- be a response code that has with a zero length reason phrase, but when a response code has a corresponding reason phrase, it will be returned.
Comment 5 Mark Thomas 2017-01-20 17:14:18 UTC
*** Bug 60618 has been marked as a duplicate of this bug. ***
Comment 6 Mateusz Nowakowski 2017-01-23 13:14:37 UTC
Facts: 
8.0.39 still returns the reason phase.
8.5.x and onward do not return the reason phase.

Purpose of this story is to add a config option to bring it back in 8.5.x and onward.

Apache Tomcat Team & Product Owner,

Do you agree to add configuration option for this?

If yes, what the potential ETA is?

This information is needed to decide what kind of actions I need to perform (whether change all legacy / old clients & postpone or not  migration to the latest Tomcat 8.5.x)
Comment 7 Michael Osipov 2017-01-23 13:47:00 UTC
(In reply to Mateusz Nowakowski from comment #6)
> Facts: 
> 8.0.39 still returns the reason phase.
> 8.5.x and onward do not return the reason phase.
> 
> Purpose of this story is to add a config option to bring it back in 8.5.x
> and onward.
> 
> Apache Tomcat Team & Product Owner,
> 
> Do you agree to add configuration option for this?
> 
> If yes, what the potential ETA is?
> 
> This information is needed to decide what kind of actions I need to perform
> (whether change all legacy / old clients & postpone or not  migration to the
> latest Tomcat 8.5.x)

Do your clients require just some text after the space or do they really analyze the text for predefined values?
Comment 8 Mateusz Nowakowski 2017-01-23 14:56:32 UTC
>Do your clients require just some text after the space or do they really analyze >the text for predefined values?

They analyze the text, however they are compatible with previous behavior.
They analyze/require:  "HTTP/1.1 200 OK" to be returned.

I thought about just a boolean configuration option to bring back previous behavior.
Comment 9 Michael Osipov 2017-01-23 15:29:59 UTC
(In reply to Mateusz Nowakowski from comment #8)
> >Do your clients require just some text after the space or do they really analyze >the text for predefined values?
> 
> They analyze the text, however they are compatible with previous behavior.
> They analyze/require:  "HTTP/1.1 200 OK" to be returned.
> 
> I thought about just a boolean configuration option to bring back previous
> behavior.

What about "HTTP/1.1 200 Fine, mate!"? This would work or hard requirement for "OK"?
Comment 10 Josh Soref 2017-01-24 06:11:31 UTC
https://github.com/jech/polipo/blob/master/tunnel.c#L302
const char *message = "HTTP/1.1 200 Tunnel established\r\n\r\n";
Comment 11 Mateusz Nowakowski 2017-01-24 16:12:10 UTC
>What about "HTTP/1.1 200 Fine, mate!"? This would work or hard requirement for "OK"?

For me it is not a requirement to tune reason phase.

However I dig into it and saw that there is some System property org.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER:
and org.apache.coyote.Response with String message
in Tomcat 8.0.x

and response.sendError(int,String) and response.sendStatus(int,String) 
sets coyote Response message.


In principle I agree with Apache Tomcat decision to not return reason phase I just want to simplify upgrades.
Comment 12 Ken DeLong 2017-01-27 19:39:06 UTC
Our software interfaces with IoT devices which contain firmware (beyond our control) that expects the "OK" (they parse for "200 OK"; just "200" is parsed as an error).

I'm sympathetic with the argument that these devices are not spec-compliant.  However, getting a new build from the manufacturer, and replacing thousands of units in the field around the world with spec-compliant IC boards would be, to say the least, prohibitively expensive.

Leave the reason phrase off by default; but give those of us stuck between a rock and a hard place an option that we can configure so that we can continue to use Tomcat.
Comment 13 Mark Thomas 2017-01-27 19:48:53 UTC
Is running on 7.0.x or 8.0.x not an option?
Comment 14 Ken DeLong 2017-01-27 21:21:48 UTC
For a short time, I'm happy to run on 8.0.x (that's what I'm doing now).  But that's unsustainable; eventually the rest of my tech stack (Spring Boot) will outpace me and I'll be in a can't-upgrade-ever-again situation.  Been there, done that before.

Even if there was a way to add a Valve or something that could decorate the response, that would work for me.  I poked around the source code a bit but could not figure out how one might do that.
Comment 15 Remy Maucherat 2017-01-27 21:25:50 UTC
The only good place to put all these non upgradeable IoT devices is the trash.
Comment 16 Michael Osipov 2017-01-27 22:25:30 UTC
(In reply to Remy Maucherat from comment #15)
> The only good place to put all these non upgradeable IoT devices is the
> trash.

Therefore, IoT = Internet of Trash
Comment 17 Josh Soref 2017-01-29 17:48:16 UTC
These devices should be behind a firewall/proxy (to protect them from the scary Internet).

Your firewall/proxy can manage the 200 OK fix for you. That way if you upgrade to Tomcat 9, you'll still be ok.
Comment 18 Todd Pierce 2017-01-29 22:39:53 UTC
(In reply to Remy Maucherat from comment #15)
> The only good place to put all these non upgradeable IoT devices is the
> trash.

In my case the devices are head units inside cars. A bit expensive to throw in the trash.
Comment 19 Ken DeLong 2017-01-30 02:37:54 UTC
And mine are in consumer electronic devices...
Comment 20 Christopher Schultz 2017-01-30 19:52:06 UTC
(In reply to Michael Osipov from comment #16)
> (In reply to Remy Maucherat from comment #15)
> > The only good place to put all these non upgradeable IoT devices is the
> > trash.
> 
> Therefore, IoT = Internet of Trash

IoT: the 'S' stands for "security"

Please have a look at http://markmail.org/message/pqzvca26hihiysiu
Comment 21 Ralf Hauser 2017-02-01 15:47:19 UTC
I am also in favour of allowing the reason code.

Although the discussion so far appears to be odd.

> "HTTP/1.1 200 Fine, mate!" 
is probably of little value.

However in particular in the 4XX and 5XX case, there is value for the client to learn more specifically what is wrong.

There is org.apache.catalina.valves.ErrorReportValve that might partially fix the problem.

But in particular non-HTML Clients like DAV clients using the
org.apache.catalina.servlets.WebdavServlet
for file transfers will not necessarily expect/render html-responses.

Therefore, I second the plea of Todd !
Comment 22 Ken DeLong 2017-02-01 16:51:16 UTC
I talked to our firmware guys - the offending code is actually in Texas Instruments firmware on the chip we use.  So there are likely many devices affected, although I'm sure not all use Tomcat.
Comment 23 Michael Osipov 2017-02-01 17:38:54 UTC
(In reply to Ralf Hauser from comment #21)
> I am also in favour of allowing the reason code.
> 
> Although the discussion so far appears to be odd.
> 
> > "HTTP/1.1 200 Fine, mate!" 
> is probably of little value.
> 
> However in particular in the 4XX and 5XX case, there is value for the client
> to learn more specifically what is wrong.
> 
> There is org.apache.catalina.valves.ErrorReportValve that might partially
> fix the problem.

No, it won't. The ErrorReportValve satisfies the Servlet Spefication for Respose#sendError(). This valve does not have access to the status line.
Comment 24 Michael Osipov 2017-02-01 17:39:47 UTC
(In reply to Ken DeLong from comment #22)
> I talked to our firmware guys - the offending code is actually in Texas
> Instruments firmware on the chip we use.  So there are likely many devices
> affected, although I'm sure not all use Tomcat.

You don't have a support contract with TI to raise this obvious bug in ther FW?
Comment 25 Ken DeLong 2017-02-01 17:51:36 UTC
We have switched to another chip maker, and in any event, the chips that are out in the field are out there, no way to upgrade except by replacing them.
Comment 26 Michael Osipov 2017-02-01 17:54:35 UTC
(In reply to Ken DeLong from comment #25)
> We have switched to another chip maker, and in any event, the chips that are
> out in the field are out there, no way to upgrade except by replacing them.

So there is no way to perform an OTA upgrade of the firmware just like Tesla does or at the auto repair when the custumer arrives for an inspection? How do you handle serious issues in such devices when the car is sold?
Comment 27 Todd Pierce 2017-02-01 23:04:20 UTC
(In reply to Michael Osipov from comment #26)
> (In reply to Ken DeLong from comment #25)
> > We have switched to another chip maker, and in any event, the chips that are
> > out in the field are out there, no way to upgrade except by replacing them.
> 
> So there is no way to perform an OTA upgrade of the firmware just like Tesla
> does or at the auto repair when the custumer arrives for an inspection? How
> do you handle serious issues in such devices when the car is sold?

You're conflating two people here. You are replying to Ken DeLong, who has consumer electronic devices. I'm the one with in-car. 

We do upgrades all the time. The problem is within manufacturer code we can't change. In older cars. It is not practicable to do what you suggest.
Comment 28 Ken DeLong 2017-02-10 01:31:08 UTC
Our situation is the same as Todd's: we can and do upgrade our own firmware, but cannot update the manufacturer's.

I tried making a Valve that wrapped the Response object, and forwarded the setStatus(int code), if code == 200, to setStatus(200, "OK").  That did not work either as a Context valve or an Engine valve.
Comment 29 Blazej Bucko 2017-02-23 09:25:54 UTC
Are you sure that you are still spec-compliant? rfc2616 defines Reason Phrase as part of Status Line. Additionally, spec suggests that it is for automata only clients are not required to read it but it doesn't say that it is OK for the server to drop it. This is not an "obvious bug" as someone suggested because "not required to read" is completely different from "are not supposed to read it". Do you have an API that allows me to set a Reason Phrase even for 200/300/400 responses? Especially when spec clearly states that I can "replace recommended Reason Phrase with local equivalent without affecting the protocol".
Comment 30 Michael Osipov 2017-02-23 09:42:46 UTC
(In reply to Blazej Bucko from comment #29)
> Are you sure that you are still spec-compliant? rfc2616 defines Reason
> Phrase as part of Status Line. Additionally, spec suggests that it is for
> automata only clients are not required to read it but it doesn't say that it
> is OK for the server to drop it. This is not an "obvious bug" as someone
> suggested because "not required to read" is completely different from "are
> not supposed to read it". Do you have an API that allows me to set a Reason
> Phrase even for 200/300/400 responses? Especially when spec clearly states
> that I can "replace recommended Reason Phrase with local equivalent without
> affecting the protocol".

As far as I know the Tomcat, there is no such option.
Comment 31 Blazej Bucko 2017-02-23 12:00:56 UTC
OK. I'd also like to point out that RFC 7230 mentioned in some of the comments is quite new (2014). Even legacy systems (in this case systems that are older than 2-3 years), which are fully compliant with rfc2616, will be affected by this change.
Comment 32 Mark Thomas 2017-02-23 12:25:33 UTC
(In reply to Blazej Bucko from comment #31)
> OK. I'd also like to point out that RFC 7230 mentioned in some of the
> comments is quite new (2014). Even legacy systems (in this case systems that
> are older than 2-3 years), which are fully compliant with rfc2616, will be
> affected by this change.

Zero length reason phrases are valid in RFC 2616 as well. Any client that can't handle a zero length reason phrase is not spec compliant.
Comment 33 Blazej Bucko 2017-02-23 13:25:17 UTC
That's true. But client that relies on information provided in Reason Phrase is also spec-compliant. And you are implying that it's not.
Comment 34 Mark Thomas 2017-02-23 13:41:14 UTC
(In reply to Blazej Bucko from comment #33)
> That's true. But client that relies on information provided in Reason Phrase
> is also spec-compliant. And you are implying that it's not.

That certainly goes against the intention of RFC2626:

<quote>
The Status-Code is intended for use by automata and the Reason-Phrase is intended for the human user.
</quote>
Comment 35 Blazej Bucko 2017-02-23 14:50:33 UTC
But it is not explicitly forbidden (especially when one reads the next sentence). It would also mean that, according to rfc2616, humans are forbidden to look at Status Code.

I understand that there's an ambiguity in rfc2616 and subsequent RFCs are more strict about this topic but it is quite possible that there are embedded systems that are fully compliant with rfc2616 and rely on Reason Phrase just because someone decided to use it in regexp check.
Comment 36 Todd Pierce 2017-02-23 23:19:31 UTC
Appeal to Reason (phrase).

As I stated in comment 4, my interpretation is the spec does not state that the reason phrase is optional, it simply says it is allowed to be zero characters. This doesn't say that the reason phrase should be abandoned in all instances. 

However, if we consider that it is optional, surely the best way to comply with the spec is for tomcat to allow the reason phrase to be optional.

And if these arguments don't resonate, how about just agreeing that the removal of the reason phrase from Tomcat 8+ is creating some major headaches and restoring it?
Comment 37 Neil Brown 2017-03-15 13:42:14 UTC
Hi,

I've read through the discussion of this issue to date, in both the original bug report and this enhancement request.

The Reason Phrase may be optional in the HTTP spec, but it's nonetheless useful to human beings. Given it's part of the HTTP/1.1 spec, I feel Tomcat users should be able to decide whether its usefulness is or isn't more important than the efficiency of not returning it.

More importantly, removing the Reason Phrase, whether it is optional or not, represents a breaking change. Obviously it's up to the Tomcat team to decide whether the benefits of a breaking change outweighs its costs - the additional hit their users will take on upgrading. As an example, we have many APIs, running on older versions of Tomcat, with a large external user base. Whilst we have a suite of functional tests for these APIs (which include checking the Reason Phrase in the HTTP response status line) that can be easily fixed, we have no way of knowing whether existing customer's integrations currently rely on matching the Reason Phrase. Like many other users, we therefore couldn't risk upgrading to Tomcat 8.5+ without first going through an additional testing phase with these customers. Depending on the outcome, we would then need to take a view on whether it's acceptable to ask our customers to make changes to their API clients. 

I'd therefore like to add my vote for reinstating support for Reason Phrase, making it a configurable option if necessary. 

Thanks.
Comment 38 Ralph Moser 2017-03-23 17:30:57 UTC
Please provide some option to reenable the reason phrase. We have several embedded systems which are not able to work without it. Yes they are not standard compliant but it's not feasible to fix them all.
I also tried to add the phrase with our Apache HTTP Server which we use as reverse proxy but as it looks it's not possible.
Comment 39 Violeta Georgieva 2017-03-27 07:36:35 UTC
Hi,

I created a patch that adds an option for sending a reason phrase with the response.
https://github.com/apache/tomcat85/pull/7

By default a reason phrase will not be sent and also this option is marked as deprecated and will not be available in Tomcat9.

The change is based on r1702765 and r1785641.

What do you think?

Regards,
Violeta
Comment 40 Remy Maucherat 2017-03-27 07:49:46 UTC
-1. It should be removed at some point, so let it be now. It turns out (given what I see) that we're going to maintain 7.0 and 8.0 for at least as long as 8.5, so it's good enough.
Comment 41 Mark Thomas 2017-03-27 08:11:01 UTC
One of the main reasons 8.0.x is maintained is because this option isn't available in 8.5.x. (The other is the sendfile issues which I think we are getting to the bottom of).

The patch looks reasonable. A couple of minor whitespace issues but nothing significant.

I think we should apply this patch with a view to reducing / halting 8.0.x releases. I don't have a specific time-frame in mind, but certainly much earlier than 8.5.x.
Comment 42 Remy Maucherat 2017-03-27 08:16:09 UTC
Ok, I thought the 8.0 plan was long gone. But the patch is only for 8.5 then ? If so, then I can withdraw the -1.
Comment 43 Violeta Georgieva 2017-03-27 08:28:58 UTC
(In reply to Remy Maucherat from comment #42)
> Ok, I thought the 8.0 plan was long gone. But the patch is only for 8.5 then
> ? If so, then I can withdraw the -1.

The patch is only for Tomcat 8.5 and I explicitly specified in the documentation and in the code that this option is deprecated and removed in Tomcat 9. Also the default remains false i.e. we will not send reason phrase by default.
Comment 44 mgrigorov 2017-03-27 09:47:10 UTC
+1 for the change!
Comment 45 Violeta Georgieva 2017-03-27 11:16:56 UTC
Hi,

A new Connector configuration 'sendReasonPhrase' is added. When this attribute is set to 'true', a reason phrase will be sent with the response.
By default a reason phrase will not be sent. This option is deprecated and is not available in Tomcat 9.

The functionality will be available in 8.5.13 onwards.

Regards,
Violeta
Comment 46 Remy Maucherat 2017-03-27 12:59:59 UTC
Ok, it's a good compromise then if 8.0 indeed does not go on (I'll believe that when it actually happens ;) ).
Comment 47 Michael Osipov 2017-03-28 09:31:39 UTC
(In reply to Violeta Georgieva from comment #45)
> Hi,
> 
> A new Connector configuration 'sendReasonPhrase' is added. When this
> attribute is set to 'true', a reason phrase will be sent with the response.
> By default a reason phrase will not be sent. This option is deprecated and
> is not available in Tomcat 9.
> 
> The functionality will be available in 8.5.13 onwards.
> 
> Regards,
> Violeta

I have left a few comments on your patch on GitHub. Please have a look.
Comment 48 Violeta Georgieva 2017-03-28 21:18:48 UTC
Hi,

(In reply to Michael Osipov from comment #47)
> (In reply to Violeta Georgieva from comment #45)
> > Hi,
> > 
> > A new Connector configuration 'sendReasonPhrase' is added. When this
> > attribute is set to 'true', a reason phrase will be sent with the response.
> > By default a reason phrase will not be sent. This option is deprecated and
> > is not available in Tomcat 9.
> > 
> > The functionality will be available in 8.5.13 onwards.
> > 
> > Regards,
> > Violeta
> 
> I have left a few comments on your patch on GitHub. Please have a look.

I replied to the comments.
Boolean#getBoolean is used for the reading of the system property as proposed.

Thanks,
Violeta
Comment 49 slavb18 2017-04-10 05:16:12 UTC
Cannot understand, why, without any reason, all legacy clients should be broken with server update.
Comment 50 Michael Osipov 2017-04-10 09:03:49 UTC
(In reply to slavb18 from comment #49)
> Cannot understand, why, without any reason, all legacy clients should be
> broken with server update.

They where already broken before. You just haven't noticed it.
Comment 51 slavb18 2017-04-10 10:25:27 UTC
I agree, but there lot of "broken" software, working everywhere in the world
Isn't too early to drop legacy support?
Even apache 2.4 proxy stops working without reason phrase

<Location /test/>
  ProxyPass http://backend/test/....
Comment 52 Blazej Bucko 2017-04-20 08:57:50 UTC
They were not broken according to the first RFC. They broke around 2014 when new RFC was published.
Comment 53 Mark Thomas 2017-04-20 09:51:25 UTC
(In reply to Blazej Bucko from comment #52)
> They were not broken according to the first RFC. They broke around 2014 when
> new RFC was published.

Nope. From RFC 2616 (dated 1999):

Reason-Phrase  = *<TEXT, excluding CR, LF>

The text around section 6.1.1 suggests that reason phrases will normally be non-zero length but the ABNF is clear that zero length reason phrases are valid (and should therefore be accepted by spec compliant user agents).
Comment 54 Blazej Bucko 2017-04-28 08:25:39 UTC
Yes, that's true. But, according to older spec, it's perfectly legal for clients to rely on this information. And removing the reason phrase breaks them.
Comment 55 Silas Smith 2017-05-01 22:23:17 UTC
I'm missing the point about why it's so important for tomcat to no longer send the reason phrase, such that even keeping it as optional is being so strongly argued against? 

I'm glad an option was added for 8.5 to enable it.

But why limit that option to only 8.5, and not keep it (optional) for 9 and beyond? Aren't we just going to have the same argument then too?
Comment 56 mgrigorov 2017-05-02 08:18:05 UTC
(In reply to Silas Smith from comment #55)
> I'm missing the point about why it's so important for tomcat to no longer
> send the reason phrase, such that even keeping it as optional is being so
> strongly argued against? 
> 
> I'm glad an option was added for 8.5 to enable it.
> 
> But why limit that option to only 8.5, and not keep it (optional) for 9 and
> beyond? Aren't we just going to have the same argument then too?

Why you can use old clients and not use old servers ?!
6.x has been just discontinued, so I expect that 8.5.x will be maintained for the next 5+ years or so.
Comment 57 Silas Smith 2017-05-02 15:01:24 UTC
(In reply to mgrigorov from comment #56)
> 
> Why you can use old clients and not use old servers ?!
> 6.x has been just discontinued, so I expect that 8.5.x will be maintained
> for the next 5+ years or so.

Because we have hundreds of thousands of distinct clients representing just about every type of device that is capable of sending an HTTP request, and we have few servers in comparison. We cannot simply break all of those clients with a shrug.

I totally get changing the default behavior, and I agree with it. But I don't get why it's so important to not allow it to be configurable going forward.
Comment 58 thorsten.meinl 2017-05-03 16:34:52 UTC
Even if this change doesn't break clients it will give a very bad impression to users. For example if you use Java to issue HTTP request and and error was returned by the server, the exception message reads "Server returned error 404: Not Found" (or similar). Even for uses not knowing the HTTP status code the "Not Found" is descriptive. Therefore I guess many Java applications displays the exception message as is. With Tomcat 8.5 the exception message is "Server returned error 404: null". Normal users don't have clue any more what's going on and even for programmers this more looks like a NullPointerException somewhere deep in the code. In order to restore a good user experience you now have to interpret the exception messages and create new ones for every possible status code.

Therefore my question is: why on earth has the reason phrase been removed in the first place? In order to save a few bytes of traffic?
Comment 59 mgrigorov 2017-05-03 18:57:29 UTC
(In reply to thorsten.meinl from comment #58)
> Even if this change doesn't break clients it will give a very bad impression
> to users. For example if you use Java to issue HTTP request and and error
> was returned by the server, the exception message reads "Server returned
> error 404: Not Found" (or similar). Even for uses not knowing the HTTP

$ http http://localhost:8080/aaa
HTTP/1.1 404 
Content-Language: en
Content-Length: 1073
Content-Type: text/html;charset=utf-8
Date: Wed, 03 May 2017 18:35:43 GMT


Those are the response headers returned by Tomcat. 
The client may add "null", emojis, unicorns, ... and Tomcat (or any server) is not to blame here!


> status code the "Not Found" is descriptive. Therefore I guess many Java
> applications displays the exception message as is. With Tomcat 8.5 the

No! It seems your client shows something it just made up!

> exception message is "Server returned error 404: null". Normal users don't
> have clue any more what's going on and even for programmers this more looks
> like a NullPointerException somewhere deep in the code. In order to restore
> a good user experience you now have to interpret the exception messages and
> create new ones for every possible status code.

Well, the HTTP protocol is pretty old! The error codes are very well known.
If a developer is confused that 404 may mean 500 then I think the time spend on debugging this problem will be very well spend time in education! This developer will learn something that will be very useful for him/her for the rest of his/her career!
If a software (i.e. non-human) makes the mistake then it is really a bug in this client software. Old clients can use old servers. Tomcat 8.5.x will be maintained for several more years. Just switch the property on and continue.

> 
> Therefore my question is: why on earth has the reason phrase been removed in
> the first place? In order to save a few bytes of traffic?

The simple answer is that the HTTP specification has been updated. HTTP2 is all about improvements and optimizations. And this is one of them.
Few bytes here, few bytes there and my web app can serve few more hundreds/thousands users more!
Comment 60 Mark Thomas 2017-05-03 21:44:07 UTC
To provide a some context / background.

7.0.x, 8.0.x always send the reason phrase
8.5.x does not send the reason phrase by default but can be configured to do so
9.0.x does not send the reason phrase and can not be configured to do so

Based on the typical lifetime of Tomcat release branches (roughly a decade or a little over and seemingly getting longer) there will be a Tomcat version (8.5.x) that can provide a reason phrase for at least 6 years, probably longer.

There were multiple reasons for dropping the reason phrase in 9.0.x:
- RFC 7230 states that clients SHOULD ignore it and therefore why bother sending it
- HTTP/2 doesn't support the reason phrase at all
- It has always (going back to at least RFC 2616) been optional - i.e. spec compliant clients should accept a zero length reason phrase
- It does save a few bytes although this will be in the noise for most users
- It allowed a little complexity to be removed (the possibility of custom reason phrases and ensuring that they were safe) from the processing of every request. This will be in the noise for most users.

Generally, the view was (and is) that the change is beneficial for the majority of spec compliant clients and for those clients that can't handle a missing reason phrase there is a viable option (8.5.x) for a relatively (in IT terms) long period.

On a related point, it is worth noting that, as a result of vulnerabilities such as CVE-2016-6816, Tomcat, and HTTP servers generally, are becoming more strict in what will be accepted. Clients that do not follow the relevant specifications and cannot be easily fixed when problems are identified are likely to become increasingly problematic.
Comment 61 Ralph Moser 2017-05-03 22:00:34 UTC
Ok. 
You have the code to add the reason phrase. It's in contrast to headers not easily fixable in most reverse proxies. Why aren't you just keeping the option to add it? We have 3000 devices out there which rely on the reason phrase. Yes we could update them but it's cumbersome. You know embedded developers? They don't read standards. They just tailor their firmware to the servers. 
Your next argument is that you are going to provide maintenance forever for 8.5. That's great but we may want to use new features. Also we regularly upgrade our spring boot version which also updates Tomcat.
Comment 62 mgrigorov 2017-05-03 22:18:45 UTC
(In reply to Ralph Moser from comment #61)
> Your next argument is that you are going to provide maintenance forever for
> 8.5. That's great but we may want to use new features. Also we regularly

How your old embedded devices will use the new features (e.g. HTTP2 features) without being upgraded ? You will need HTTP2 enabled client in the embedded device!
If you are going to update the client then it is a perfect time to relax the check for the reason phrase!

> upgrade our spring boot version which also updates Tomcat.

The version of the dependencies in Spring Boot applications is *very* easy to control with Maven/Gradle properties/settings! I.e. in your application!

Tomcat 9 won't be released until Java EE 8 / Servlet 4.0 is released. So this will take a while!
But even when Tomcat 9 is released 
1) Spring Boot won't upgrade to it immediately. They will wait for a while so any newly introduced bugs are flushed.
2) you will be still able to use 8.5.x by overriding the version
Comment 63 Christopher Schultz 2017-05-04 16:54:20 UTC
First of all, let's all settle down. Reasonable people can disagree about whether this change is a Good Thing or a Bad Thing and whether options should exist to re-enable it. With that in mind, let's continue the conversation...

(In reply to Ralph Moser from comment #61)
> You have the code to add the reason phrase. It's in contrast to headers not
> easily fixable in most reverse proxies. Why aren't you just keeping the
> option to add it?

This is a conversation. There hasn't been a "last word" on this quite yet.

> We have 3000 devices out there which rely on the reason
> phrase.

Note that there is no supported stable version of Tomcat where the reason phrase isn't available.

> Yes we could update them but it's cumbersome. You know embedded
> developers? They don't read standards. They just tailor their firmware to
> the servers. 

I see this change as motivation for embedded developers to tailor their firmware to standards-compliant servers. This is very similar to the SSL vs TLS chicken-in-an-egg scenario we had several years ago: no servers were implementing TLS because no clients supported it yet and no clients were supporting it because no servers had support, either. Someone must move first.

> Your next argument is that you are going to provide maintenance forever for
> 8.5. That's great but we may want to use new features. Also we regularly
> upgrade our spring boot version which also updates Tomcat.

If Spring Boot suddenly gives you a new major version of Tomcat without asking or being configured explicitly to do so, I think you should re-think your use of that technology.

My position has been that 8.5 should not have removed the reason phrase, so I was motivated to lobby for the introduction of a setting that would restore that feature. I have no such feelings about Tomcat 9.0 which won't be released as stable until Servlet 4.0 is finalized which could be anytime between now and when I die of old age.

We have plenty of time to discuss whether the HTTP reason phrase is something worth continuing to support in Tomcat 9 and later.

We should probably move this conversation back to the dev list, because this enhancement was filed against Tomcat 8.5 and, in fact, fixed. Let's leave this BZ issue in peace and re-locate to dev@ until we can decide whether Tomcat 9 should also be changed.
Comment 64 Christopher Schultz 2017-05-05 18:54:07 UTC
*** Bug 60183 has been marked as a duplicate of this bug. ***
Comment 65 Olivier Jaquemet 2017-06-06 13:26:35 UTC
Hi, 

Just so you know, BitKinex WebDAV client is impacted by this missing reason phrase and it can no longer access any webdav servlet configured in Tomcat unless sendReasonPhrase="true" is configured.

I just created the following bug report to make sure it gets identified if some user look for those symptoms : 
https://bz.apache.org/bugzilla/show_bug.cgi?id=61160

It might help you revise current tomcat decision to remove the phrase, if you were to consider BitKinex a widespread enough client.
Comment 66 Fred Simon 2017-11-21 09:44:24 UTC
For info Debian Apt (some may think it's also IoT - as Trash) is also hardcoded to expect a reason phrase :(
Was fixed for http: https://github.com/Debian/apt/commit/dda7233c5d3879f2580543ead0ad7cd76196a160#diff-c8367b0733401af8a29f502c70434b89

But not https: https://www.jfrog.com/jira/browse/RTFACT-15315
Comment 67 Michael Osipov 2017-11-21 22:05:32 UTC
(In reply to Fred Simon from comment #66)
> For info Debian Apt (some may think it's also IoT - as Trash) is also
> hardcoded to expect a reason phrase :(
> Was fixed for http:
> https://github.com/Debian/apt/commit/
> dda7233c5d3879f2580543ead0ad7cd76196a160#diff-
> c8367b0733401af8a29f502c70434b89
> 
> But not https: https://www.jfrog.com/jira/browse/RTFACT-15315

If I understand scanf correctly, the apt code is broken even after the call. The RFC mandates that after the status code a mandatory space (U+0020) has follow along with an optional reason phrase. The mandatory space has been removed with that commit.
Comment 68 William Watson 2018-02-22 02:57:33 UTC
I believe an option to send a reason phrase should be maintained in Tomcat 9.

The reason phrase should be ignored by RFC-compliant client software.  But RFC compliant software is not the only thing that consumes responses.

HTTP responses are also read by developers.  In this case, the reason phrase can provide useful information.  The developer experience of a REST API is improved with a reason phrase.  The reason phrase may be visible to a developer using cURL or when debugging their application.

In short, "405 METHOD NOT ALLOWED" provides a better developer experience than "405 ".

(Let me know if I should open a new bug against Tomcat 9 to progress this case).
Comment 69 Michael Osipov 2018-02-22 12:56:03 UTC
(In reply to William Watson from comment #68)
> I believe an option to send a reason phrase should be maintained in Tomcat 9.
> 
> The reason phrase should be ignored by RFC-compliant client software.  But
> RFC compliant software is not the only thing that consumes responses.
> 
> HTTP responses are also read by developers.  In this case, the reason phrase
> can provide useful information.  The developer experience of a REST API is
> improved with a reason phrase.  The reason phrase may be visible to a
> developer using cURL or when debugging their application.
> 
> In short, "405 METHOD NOT ALLOWED" provides a better developer experience
> than "405 ".


the semantics of every status code are well defined. The text is just nice gimmick. I don't think that "404 Nothing to see here" adds any value. An HTTP developer not known status codes is a bad developer.
Comment 70 Mark Thomas 2018-02-22 13:14:14 UTC
That is a view although it is perhaps a tad harsh.

Regardless, the world is moving towards HTTP/2 and HTTP/2 doesn't have a reason phrase so this is a situation developers are going to have to get used to.
Comment 71 Michael Osipov 2018-02-22 16:12:34 UTC
(In reply to Mark Thomas from comment #70)
> That is a view although it is perhaps a tad harsh.
> 
> Regardless, the world is moving towards HTTP/2 and HTTP/2 doesn't have a
> reason phrase so this is a situation developers are going to have to get
> used to.

I concur, I am quite certain that a LOT of servers and applications will remain on 1.1 for at least 10 years for a LOT of reasons. But this does not rectify not to stick to the RFCs.
Comment 72 Ralf Hauser 2018-05-16 05:24:43 UTC
First, there are many error conditions for which no precise 4xx or 5xx code is defined. So in this way, the reason might be helpful.

While I understand that an html browser can display more hints on the error in the content instead of the header, how should other http dependent clients do that such as webDav clients (winscp.net, cyberduck.io, ...)?

Just for curiosity: how should webDav clients work with Http/2 to display tailored error messages to the end-user ?
Comment 73 Michael Osipov 2018-05-16 19:39:23 UTC
(In reply to Ralf Hauser from comment #72)
> First, there are many error conditions for which no precise 4xx or 5xx code
> is defined. So in this way, the reason might be helpful.

No, use a given status code and augment it with application/problem+json or similar. The Status text cannot be set via Servlet API anyway.
Comment 74 Christopher Schultz 2018-05-17 02:02:25 UTC
(In reply to Michael Osipov from comment #73)
> (In reply to Ralf Hauser from comment #72)
> > First, there are many error conditions for which no precise 4xx or 5xx code
> > is defined. So in this way, the reason might be helpful.
> 
> No, use a given status code and augment it with application/problem+json or
> similar. The Status text cannot be set via Servlet API anyway.

https://docs.oracle.com/javaee/6/api/javax/servlet/http/HttpServletResponse.html#sendError(int,%20java.lang.String) ?

I was surprised to see that Tomcat actively strips-out the reason phrase. I had initially thought this was simply Tomcat removing reason-phrases from every response generated by Tomcat (e.g. everything coming from the DefaultServlet, various internal errors, etc.), but it's actively stripping reason phrases explicitly-set by applications. :(

At any rate, this should be discussed in dev@ and not in bugzilla at this point.
Comment 75 Ralf Hauser 2018-05-17 05:02:29 UTC
(In reply to Michael Osipov from comment #73)
> No, use a given status code and augment it with application/problem+json or
> similar. The Status text cannot be set via Servlet API anyway.

Just for completeness, Michael appears to reference

https://tools.ietf.org/html/rfc7807 (Problem Details for HTTP APIs)
Comment 76 Michael Osipov 2018-05-17 07:10:46 UTC
(In reply to Christopher Schultz from comment #74)
> (In reply to Michael Osipov from comment #73)
> > (In reply to Ralf Hauser from comment #72)
> > > First, there are many error conditions for which no precise 4xx or 5xx code
> > > is defined. So in this way, the reason might be helpful.
> > 
> > No, use a given status code and augment it with application/problem+json or
> > similar. The Status text cannot be set via Servlet API anyway.
> 
> https://docs.oracle.com/javaee/6/api/javax/servlet/http/HttpServletResponse.
> html#sendError(int,%20java.lang.String) ?

No, this does not send any reason phrase. Only the HTML error page. I know, because I have rewritten the ErrorReportValve the last time.
Comment 77 Julian Reschke 2018-05-22 11:06:12 UTC
(In reply to Ralf Hauser from comment #72)
> First, there are many error conditions for which no precise 4xx or 5xx code
> is defined. So in this way, the reason might be helpful.
> 
> While I understand that an html browser can display more hints on the error
> in the content instead of the header, how should other http dependent
> clients do that such as webDav clients (winscp.net, cyberduck.io, ...)?
> 
> Just for curiosity: how should webDav clients work with Http/2 to display
> tailored error messages to the end-user ?

FWIW, WebDAV defines a payload format for errors; see <https://www.greenbytes.de/tech/webdav/rfc4918.html#precondition.postcondition.xml.elements>
Comment 78 Ralf Hauser 2018-11-29 13:58:29 UTC
see also Bug 62964
Comment 79 Matthew Buckett 2020-05-10 16:39:52 UTC
>
> There were multiple reasons for dropping the reason phrase in 9.0.x:
> - RFC 7230 states that clients SHOULD ignore it and therefore why bother
> sending it

But it can help developers in debugging problems and it's not a MUST, so it's still acceptable.

> - HTTP/2 doesn't support the reason phrase at all

But someone moving their service from HTTP/1 to HTTP/2 will probably want to change their error handling and this is a good time to do it (not when they attempt to upgrade Tomcat).

> - It has always (going back to at least RFC 2616) been optional - i.e. spec
> compliant clients should accept a zero length reason phrase

Lots of things are optional in the spec but Tomcat still allows them.

> - It does save a few bytes although this will be in the noise for most users

So why remove it as a config option, it was already not being sent in most 8.5 installations.

> - It allowed a little complexity to be removed (the possibility of custom
> reason phrases and ensuring that they were safe) from the processing of
> every request. This will be in the noise for most users.

This to me seems to only valid reason here and it's basically causing application developers pain to simplify Tomcat. To me it seems like having the flag was a nice solution (people who needed it could enable it), newly developed applications wouldn't see it and would hopefully adopt RFC 7230 instead.