Bug 193253
| Summary: | 502 Bad Gateway Errors using Issue Tracker | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Services | Reporter: | Kubilay Kocak <koobs> | ||||
| Component: | Bug Tracker | Assignee: | Bugmeister <bugmeister> | ||||
| Status: | Closed FIXED | ||||||
| Severity: | Affects Many People | CC: | koobs, mva, peter, rhurlin, woodsb02 | ||||
| Priority: | Normal | ||||||
| Version: | unspecified | ||||||
| Hardware: | Any | ||||||
| OS: | Any | ||||||
| Attachments: |
|
||||||
|
Description
Kubilay Kocak
2014-09-02 13:05:26 UTC
Created attachment 146679 [details]
FreeBSD bugzilla/ request headers (Chrome 34.0.1847.131 on Mac OS X 10.9.4)
The issue is currently being investigated, with assistance requested from our Cluster Administration team (CC'd) May to be related to HTTP Request sizes > 4kb. After several days of not being able to login into Bugzilla, today it works. Don't know, if that is of any interest, but the request header is very short this time: Request-URL: https://bugs.freebsd.org/bugzilla/buglist.cgi?component=Individual%20Port%28s%29&list_id=21125&order=bug_id%20DESC&product=Ports%20Tree&query_based_on=&query_format=advanced&resolution=--- Request-Methode: GET Status-Code: HTTP/1.1 200 OK Request-Header 21:29:45.000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Firefox/31.0 Host: bugs.freebsd.org DNT: 1 Connection: keep-alive Cache-Control: max-age=0 Accept-Language: de-de,en-us;q=0.7,en;q=0.3 Accept-Encoding: gzip, deflate Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Gesendetes Cookie VERSION-Ports Tree: Latest TUI: attachment_text_field=0&expert_fields=1 LASTORDER: bug_id DESC Bugzilla_logincookie: 2jczaBBMj0 Bugzilla_login_request_cookie: V7c5yPO6FH Bugzilla_login: 25666 Response-Header Δ2249ms X-XSS-Protection: 1; mode=block X-Varnish: 1235694241 X-Frame-Options: SAMEORIGIN X-Firefox-Spdy: 3.1 X-Debug-In-Cookie: Bugzilla_login_request_cookie=V7c5yPO6FH; LASTORDER=bug_id%20DESC; Bugzilla_login=25666; Bugzilla_logincookie=2jczaBBMj0; TUI=attachment_text_field=0&expert_fields=1; VERSION-Ports%20Tree=Latest X-Content-Type-Options: nosniff X-Cache: MISS Via: 1.1 wfe0.ysv.FreeBSD.org Strict-Transport-Security: max-age=31536000; includeSubdomains Server: Apache Date: Tue, 02 Sep 2014 19:29:47 GMT Content-Type: multipart/x-mixed-replace;boundary="------- =_c1PeWTAJZ3e6mnbAP"; charset=UTF-8 Age: 1 clusteradm: Requesting the current value of "http_req_hdr_len" Varnish setting please. 8192 bytes. Rainer, Ben: Can you both please do the following: - Clear all the *freebsd.org cookies in your browser - play around in bugzilla, to see if the problem still/again occurs If the problem occurs again: - switch to a different browser - repeat both steps from above For the steps being done, we'd appreciate, if you could note down the results: - browser version, OS - test case (search over X, just tried to login, etc.) - cookie and request header sizes (if possible) My personal experience was/is that Chrome 36.x on Windows and FreeBSD could cause the 502, when I performed queries with large result sets. It does not occur to me with Chrome 37. Firefox 31 on both systems did not cause any problems. IE 10 sometimes caused a 502, sometimes it did not. Since the issue is not easy to be recreated and since we assume the different systems to be configured properly, we need to figure out, what could be the root cause for further investigation. Thanks! Hmmm... clearing my cookies seems to have resolved the problem (after I did that I cannot seem to get it to 502 again!) Although, I have since updated my Chrome to 37.0.2062.94 (still on OSX 10.9.4), not sure if that makes a difference though. To your suggestion I tried searches with large result sets (searched all bugs open or closed with the keyword "port", and also selected long format). No 502... Rainer: some more adjustments were made. Can you give it another try? Since Ben's not able to reproduce that issue anymore, we hope to be able to verify the issue being fixed with the latest adjustments. Thanks! This is a problem between the SSL front-end and the bugzilla instance. We have a test case where we can reliably trigger the problem. The problem, as we understand it, is that bugzilla maintains a recently viewed bugs list in cookies. This can grow fairly large, however bugzilla knows what the browser client-side limits are (4k-5k) and what the apache server limit is (8k) and makes sure to not hit either of those. However, there was a debug header being added in between the ssl front-end and the bugzilla back-end that added a cookie trace header. The ssl front-end has a limit of 4k for *http headers* when talking to a backend server and this debug trace header (which was a copy of the incoming cookie) overflowed nginx. We've made multiple changes: - removed the debug header entirely - increased the nginx proxy_buffers settings to match the worst-case size limits. I suspect this will fix it, unless bugzilla itself is not maintaining the cookie size limits correctly. I don't think any current browsers will allow this to happen though. [Contrary to #comment 6 of mva@, for me the problems occured with Firefox 31.0] I tried yesterday morning to login and had only problems, when there were "sessions before". Today it seems, that all problems went away. As far as I can see until now, no more problems to login, even after working with different bug_ids (?). So it seems to be solved for me. I am not sure, if my descriptions are sufficient. Sorry, if they are too unspecific. Many thanks for your investigations in this login problems. Thanks for letting us know. We consider the issue fixed for the moment. If the problems should occur again, please let us know. |