Nginx config should I deny requests that are missing the accept header

Question

So I block requests that have a missing / empty user agent header because there is no reason why the user agent should be not present or empty since all legit browsers and bots (googlebot etc) send that header.

##If the user agent is missing empty or contains just a single hyphen.
if ($http_user_agent ~ "^-?$") {
return 403;
}

And from my research apparently the Accept header should always be present even if it is empty it should be sent regardless. The header should not be missing.

But before going and denying requests that don't send an Accept header I wanted to verify that it won't have bad effects I see modsecurity and a few others (not so sure about OWASP) say if the accept header should be present at least. (From what I can see it should be sent and allowed access as long as the header was present in the request and not missing.)

I made the following rule for Nginx have not put it into production use since I need more information about if that header should be missing or not.

## Prevent Missing Accept header
set $block_missing_accept_header 0;
#If Accept header present
if ($http_accept) {
set $block_missing_accept_header 1;
}
#Header was missing so deny request
if ($block_missing_accept_header = 0) {
return 403;
}

Nicolas Guérinet · Answer

I think that clients with no user-agent should be blocked.

All legitimate robots requests have a HTTP user-agent field: googlebot, bing, baidu...
All legitimate known SEO robot requests have a HTTP user-agent: moz, majestic, semrush..
All legitimate browser requests have a HTTP user-agent: Chome, Edge, Brave, Mozilla...
linux commands add a user-agent by default: curl, kurly

Michael Hampton · Answer

That rule was my idea, so here is my thinking on it.

What I observed over a very large sample of requests is that actual web browsers (IE, Firefox, Chrome, Konqueror, Opera, and others) always sent the Accept header. But robots did not always do so, depending on a variety of factors such as the HTTP library, the programmer's competence, etc.

Then, robots pretending to be web browsers with a fake User-Agent string would sometimes be missing the Accept header.

So, the idea here was to catch out robots pretending to be web browsers and doing it poorly, by not sending the Accept header.

To implement this rule, you must also check that the User-Agent string starts with "Mozilla". It's not appropriate to block unless both of these conditions are met: The user agent claims to be a web browser, and the Accept header is missing.

if ($http_user_agent ~ ^Mozilla) {
    set $block_missing_accept_header "Mozilla";
}
if ($http_accept = false) {
    set $block_missing_accept_header "${block_missing_accept_header}NoAccept";
}
if ($block_missing_accept_header == "MozillaNoAccept") {
    return 403;
}

Nginx config should I deny requests that are missing the accept header

2 Answers

Add your own answers!

Ask a Question