pfSense + Nginx proxy and Real user IP
up vote
0
down vote
favorite
Ok, so I have 1 server with pfSense and many virtual servers. I'm using Nginx upstream functionality to run multiplies WEB servers on same public IP. Of course I need to know REAL users IP not Nginx proxy which is 192.168.2.2, but after switching to pfSense (recently had simple consumer router) web servers can't see real users IP.
I have tried to change various settings in System / Advanced / Firewall & NAT like:
NAT Reflection mode for port forwards
Enable automatic outbound NAT for Reflection
Also in Firewall / NAT / Outbound tried every mode, nothing helped still every user have IP of my Proxy server.
So how to disable masquarading, or how to pass real client IP.
Update
Ok, so it seams problem is with subdomains not domains. Situation now:
If client go to domain.com - everything is fine backend server can see real clinet IP
If client go to subdomain.domain.com - backend server see proxy server IP
All domains A records points to external IP, then pfSense forward 80 port to proxy, then proxy depending on domain forward to corresponding internal server.
I have 2 physical servers, 1 - pfSense router and another with virtualbox running many VM's in this example 4 VM's
Another one interesting thing, when i try to reach troublesome subdomain.domain1.com from inside local network I get this:
Again, no problems with domain1.com and domain2.com and so on...
pfsense
asked Aug 26 '16 at 17:10
RomkaLTU
1012
add a comment |
up vote
0
down vote
favorite
Ok, so I have 1 server with pfSense and many virtual servers. I'm using Nginx upstream functionality to run multiplies WEB servers on same public IP. Of course I need to know REAL users IP not Nginx proxy which is 192.168.2.2, but after switching to pfSense (recently had simple consumer router) web servers can't see real users IP.
I have tried to change various settings in System / Advanced / Firewall & NAT like:
NAT Reflection mode for port forwards
Enable automatic outbound NAT for Reflection
Also in Firewall / NAT / Outbound tried every mode, nothing helped still every user have IP of my Proxy server.
So how to disable masquarading, or how to pass real client IP.
Update
Ok, so it seams problem is with subdomains not domains. Situation now:
If client go to domain.com - everything is fine backend server can see real clinet IP
If client go to subdomain.domain.com - backend server see proxy server IP
All domains A records points to external IP, then pfSense forward 80 port to proxy, then proxy depending on domain forward to corresponding internal server.
I have 2 physical servers, 1 - pfSense router and another with virtualbox running many VM's in this example 4 VM's
Another one interesting thing, when i try to reach troublesome subdomain.domain1.com from inside local network I get this:
Again, no problems with domain1.com and domain2.com and so on...
pfsense
asked Aug 26 '16 at 17:10
RomkaLTU
1012
What does your nginx config look like?
– Paul Nordin
Aug 26 '16 at 18:42
Everything is fine with nginx, because as I said: before pfSense I was connected simple consumer router with same port forwards, but after change everything is working same just all users comming from 192.168.2.2 in server logs etc, or simple in php remote_addr
– RomkaLTU
Aug 26 '16 at 18:54
Note sure about terminology, but I think I need NAT reflection, still digging about it...
– RomkaLTU
Aug 26 '16 at 20:02
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
Ok, so I have 1 server with pfSense and many virtual servers. I'm using Nginx upstream functionality to run multiplies WEB servers on same public IP. Of course I need to know REAL users IP not Nginx proxy which is 192.168.2.2, but after switching to pfSense (recently had simple consumer router) web servers can't see real users IP.
I have tried to change various settings in System / Advanced / Firewall & NAT like:
NAT Reflection mode for port forwards
Enable automatic outbound NAT for Reflection
Also in Firewall / NAT / Outbound tried every mode, nothing helped still every user have IP of my Proxy server.
So how to disable masquarading, or how to pass real client IP.
Update
Ok, so it seams problem is with subdomains not domains. Situation now:
If client go to domain.com - everything is fine backend server can see real clinet IP
If client go to subdomain.domain.com - backend server see proxy server IP
All domains A records points to external IP, then pfSense forward 80 port to proxy, then proxy depending on domain forward to corresponding internal server.
I have 2 physical servers, 1 - pfSense router and another with virtualbox running many VM's in this example 4 VM's
Another one interesting thing, when i try to reach troublesome subdomain.domain1.com from inside local network I get this:
Again, no problems with domain1.com and domain2.com and so on...
pfsense
asked Aug 26 '16 at 17:10
RomkaLTU
1012
Ok, so I have 1 server with pfSense and many virtual servers. I'm using Nginx upstream functionality to run multiplies WEB servers on same public IP. Of course I need to know REAL users IP not Nginx proxy which is 192.168.2.2, but after switching to pfSense (recently had simple consumer router) web servers can't see real users IP.
I have tried to change various settings in System / Advanced / Firewall & NAT like:
NAT Reflection mode for port forwards
Enable automatic outbound NAT for Reflection
Also in Firewall / NAT / Outbound tried every mode, nothing helped still every user have IP of my Proxy server.
So how to disable masquarading, or how to pass real client IP.
Update
Ok, so it seams problem is with subdomains not domains. Situation now:
If client go to domain.com - everything is fine backend server can see real clinet IP
If client go to subdomain.domain.com - backend server see proxy server IP
All domains A records points to external IP, then pfSense forward 80 port to proxy, then proxy depending on domain forward to corresponding internal server.
I have 2 physical servers, 1 - pfSense router and another with virtualbox running many VM's in this example 4 VM's
Another one interesting thing, when i try to reach troublesome subdomain.domain1.com from inside local network I get this:
Again, no problems with domain1.com and domain2.com and so on...
pfsense
pfsense
asked Aug 26 '16 at 17:10
RomkaLTU
1012
asked Aug 26 '16 at 17:10
RomkaLTU
1012
edited Aug 27 '16 at 10:57
asked Aug 26 '16 at 17:10
RomkaLTU
1012
asked Aug 26 '16 at 17:10
RomkaLTU
1012
asked Aug 26 '16 at 17:10
RomkaLTU
1012
1012
What does your nginx config look like?
– Paul Nordin
Aug 26 '16 at 18:42
Everything is fine with nginx, because as I said: before pfSense I was connected simple consumer router with same port forwards, but after change everything is working same just all users comming from 192.168.2.2 in server logs etc, or simple in php remote_addr
– RomkaLTU
Aug 26 '16 at 18:54
Note sure about terminology, but I think I need NAT reflection, still digging about it...
– RomkaLTU
Aug 26 '16 at 20:02
add a comment |
What does your nginx config look like?
– Paul Nordin
Aug 26 '16 at 18:42
Everything is fine with nginx, because as I said: before pfSense I was connected simple consumer router with same port forwards, but after change everything is working same just all users comming from 192.168.2.2 in server logs etc, or simple in php remote_addr
– RomkaLTU
Aug 26 '16 at 18:54
Note sure about terminology, but I think I need NAT reflection, still digging about it...
– RomkaLTU
Aug 26 '16 at 20:02
What does your nginx config look like?
– Paul Nordin
Aug 26 '16 at 18:42
What does your nginx config look like?
– Paul Nordin
Aug 26 '16 at 18:42
Everything is fine with nginx, because as I said: before pfSense I was connected simple consumer router with same port forwards, but after change everything is working same just all users comming from 192.168.2.2 in server logs etc, or simple in php remote_addr
– RomkaLTU
Aug 26 '16 at 18:54
Everything is fine with nginx, because as I said: before pfSense I was connected simple consumer router with same port forwards, but after change everything is working same just all users comming from 192.168.2.2 in server logs etc, or simple in php remote_addr
– RomkaLTU
Aug 26 '16 at 18:54
Note sure about terminology, but I think I need NAT reflection, still digging about it...
– RomkaLTU
Aug 26 '16 at 20:02
Note sure about terminology, but I think I need NAT reflection, still digging about it...
– RomkaLTU
Aug 26 '16 at 20:02
add a comment |
2 Answers
2
active
oldest
votes
up vote
0
down vote
There basically two ways to forward ports: One is what your pfSense is doing now ("full" NAT, conntrack in Linux): When a new connection is initiated by a client, pfSense creates a new mapping in it's NAT table, swaps out the source address with it's own, changes the source port if appropriate and sends the modified packet to your webserver. Your webserver will automatically address it's answers to the pfSense machine, which can then swap out the fields again and send the packet to the client. The advantage of this approach is that your webserver doesn't need to be aware of it, it just works. As far as I remember, you can disable this in pfSense if you switch your NAT mode to "AON" and disable NAT for (webserverip, targetport).
Your consumer router did a simple port forward(DNAT in Linux): On arrival of a packet, it simply swapped the destination address and sent the packet to your webserver. Since the packet now still has the real source address, the webserver can see the real address of the client. Unfortunately, when it sends an answer, it will put it's own (private) address in the source field, which the router has to swap out against your public IP on the way out(SNAT in Linux). Since the webserver directly addresses the packet to the client, the router can only do this if it is also the default gateway! (or when you set up rather funky routing policies on your webserver)
answered Aug 27 '16 at 10:20
maxf
1,233211
I updated question. It seams that problem persist only with subdomains.
– RomkaLTU
Aug 27 '16 at 10:23
In that case we need to see your nginx.conf. I assume the domains all have the same A records?
– maxf
Aug 27 '16 at 10:24
Yes, all domains A record points to my external IP, then pfsense port forward 80 to proxy same port. Nginx config is simple, and there was no problem before pfSense. Nginx config is very simple, just upstream server 1{ server 192.168.2.12:80; } and proxy_pass server1;
– RomkaLTU
Aug 27 '16 at 10:28
If you have multiple different domains you must have multiple different server blocks to separate them? Also, do clients see the pfSense box IP or the nginx box IP?
– maxf
Aug 27 '16 at 10:33
I added picture of my infrastructure. No problems with domain1.com and domain2.com but subdomain.domain1.com receiving proxy IP...
– RomkaLTU
Aug 27 '16 at 10:43
add a comment |
up vote
0
down vote
Ok so problem was not in pfSense and not in proxy, problem was in specific backend server (green square) configuration. I mus accidentally disabled option "Use Client IP in Header" I was sure it was enabled, I know about this option, so it was backend server miss-configuration. Backend server is Litespeed.
answered Aug 28 '16 at 8:24
RomkaLTU
1012
add a comment |
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
There basically two ways to forward ports: One is what your pfSense is doing now ("full" NAT, conntrack in Linux): When a new connection is initiated by a client, pfSense creates a new mapping in it's NAT table, swaps out the source address with it's own, changes the source port if appropriate and sends the modified packet to your webserver. Your webserver will automatically address it's answers to the pfSense machine, which can then swap out the fields again and send the packet to the client. The advantage of this approach is that your webserver doesn't need to be aware of it, it just works. As far as I remember, you can disable this in pfSense if you switch your NAT mode to "AON" and disable NAT for (webserverip, targetport).
Your consumer router did a simple port forward(DNAT in Linux): On arrival of a packet, it simply swapped the destination address and sent the packet to your webserver. Since the packet now still has the real source address, the webserver can see the real address of the client. Unfortunately, when it sends an answer, it will put it's own (private) address in the source field, which the router has to swap out against your public IP on the way out(SNAT in Linux). Since the webserver directly addresses the packet to the client, the router can only do this if it is also the default gateway! (or when you set up rather funky routing policies on your webserver)
answered Aug 27 '16 at 10:20
maxf
1,233211
I updated question. It seams that problem persist only with subdomains.
– RomkaLTU
Aug 27 '16 at 10:23
In that case we need to see your nginx.conf. I assume the domains all have the same A records?
– maxf
Aug 27 '16 at 10:24
Yes, all domains A record points to my external IP, then pfsense port forward 80 to proxy same port. Nginx config is simple, and there was no problem before pfSense. Nginx config is very simple, just upstream server 1{ server 192.168.2.12:80; } and proxy_pass server1;
– RomkaLTU
Aug 27 '16 at 10:28
If you have multiple different domains you must have multiple different server blocks to separate them? Also, do clients see the pfSense box IP or the nginx box IP?
– maxf
Aug 27 '16 at 10:33
I added picture of my infrastructure. No problems with domain1.com and domain2.com but subdomain.domain1.com receiving proxy IP...
– RomkaLTU
Aug 27 '16 at 10:43
add a comment |
up vote
0
down vote
There basically two ways to forward ports: One is what your pfSense is doing now ("full" NAT, conntrack in Linux): When a new connection is initiated by a client, pfSense creates a new mapping in it's NAT table, swaps out the source address with it's own, changes the source port if appropriate and sends the modified packet to your webserver. Your webserver will automatically address it's answers to the pfSense machine, which can then swap out the fields again and send the packet to the client. The advantage of this approach is that your webserver doesn't need to be aware of it, it just works. As far as I remember, you can disable this in pfSense if you switch your NAT mode to "AON" and disable NAT for (webserverip, targetport).
Your consumer router did a simple port forward(DNAT in Linux): On arrival of a packet, it simply swapped the destination address and sent the packet to your webserver. Since the packet now still has the real source address, the webserver can see the real address of the client. Unfortunately, when it sends an answer, it will put it's own (private) address in the source field, which the router has to swap out against your public IP on the way out(SNAT in Linux). Since the webserver directly addresses the packet to the client, the router can only do this if it is also the default gateway! (or when you set up rather funky routing policies on your webserver)
answered Aug 27 '16 at 10:20
maxf
1,233211
I updated question. It seams that problem persist only with subdomains.
– RomkaLTU
Aug 27 '16 at 10:23
In that case we need to see your nginx.conf. I assume the domains all have the same A records?
– maxf
Aug 27 '16 at 10:24
Yes, all domains A record points to my external IP, then pfsense port forward 80 to proxy same port. Nginx config is simple, and there was no problem before pfSense. Nginx config is very simple, just upstream server 1{ server 192.168.2.12:80; } and proxy_pass server1;
– RomkaLTU
Aug 27 '16 at 10:28
If you have multiple different domains you must have multiple different server blocks to separate them? Also, do clients see the pfSense box IP or the nginx box IP?
– maxf
Aug 27 '16 at 10:33
I added picture of my infrastructure. No problems with domain1.com and domain2.com but subdomain.domain1.com receiving proxy IP...
– RomkaLTU
Aug 27 '16 at 10:43
add a comment |
up vote
0
down vote
up vote
0
down vote
There basically two ways to forward ports: One is what your pfSense is doing now ("full" NAT, conntrack in Linux): When a new connection is initiated by a client, pfSense creates a new mapping in it's NAT table, swaps out the source address with it's own, changes the source port if appropriate and sends the modified packet to your webserver. Your webserver will automatically address it's answers to the pfSense machine, which can then swap out the fields again and send the packet to the client. The advantage of this approach is that your webserver doesn't need to be aware of it, it just works. As far as I remember, you can disable this in pfSense if you switch your NAT mode to "AON" and disable NAT for (webserverip, targetport).
Your consumer router did a simple port forward(DNAT in Linux): On arrival of a packet, it simply swapped the destination address and sent the packet to your webserver. Since the packet now still has the real source address, the webserver can see the real address of the client. Unfortunately, when it sends an answer, it will put it's own (private) address in the source field, which the router has to swap out against your public IP on the way out(SNAT in Linux). Since the webserver directly addresses the packet to the client, the router can only do this if it is also the default gateway! (or when you set up rather funky routing policies on your webserver)
answered Aug 27 '16 at 10:20
maxf
1,233211
There basically two ways to forward ports: One is what your pfSense is doing now ("full" NAT, conntrack in Linux): When a new connection is initiated by a client, pfSense creates a new mapping in it's NAT table, swaps out the source address with it's own, changes the source port if appropriate and sends the modified packet to your webserver. Your webserver will automatically address it's answers to the pfSense machine, which can then swap out the fields again and send the packet to the client. The advantage of this approach is that your webserver doesn't need to be aware of it, it just works. As far as I remember, you can disable this in pfSense if you switch your NAT mode to "AON" and disable NAT for (webserverip, targetport).
Your consumer router did a simple port forward(DNAT in Linux): On arrival of a packet, it simply swapped the destination address and sent the packet to your webserver. Since the packet now still has the real source address, the webserver can see the real address of the client. Unfortunately, when it sends an answer, it will put it's own (private) address in the source field, which the router has to swap out against your public IP on the way out(SNAT in Linux). Since the webserver directly addresses the packet to the client, the router can only do this if it is also the default gateway! (or when you set up rather funky routing policies on your webserver)
answered Aug 27 '16 at 10:20
maxf
1,233211
answered Aug 27 '16 at 10:20
maxf
1,233211
answered Aug 27 '16 at 10:20
maxf
1,233211
answered Aug 27 '16 at 10:20
maxf
1,233211
1,233211
I updated question. It seams that problem persist only with subdomains.
– RomkaLTU
Aug 27 '16 at 10:23
In that case we need to see your nginx.conf. I assume the domains all have the same A records?
– maxf
Aug 27 '16 at 10:24
Yes, all domains A record points to my external IP, then pfsense port forward 80 to proxy same port. Nginx config is simple, and there was no problem before pfSense. Nginx config is very simple, just upstream server 1{ server 192.168.2.12:80; } and proxy_pass server1;
– RomkaLTU
Aug 27 '16 at 10:28
If you have multiple different domains you must have multiple different server blocks to separate them? Also, do clients see the pfSense box IP or the nginx box IP?
– maxf
Aug 27 '16 at 10:33
I added picture of my infrastructure. No problems with domain1.com and domain2.com but subdomain.domain1.com receiving proxy IP...
– RomkaLTU
Aug 27 '16 at 10:43
add a comment |
I updated question. It seams that problem persist only with subdomains.
– RomkaLTU
Aug 27 '16 at 10:23
In that case we need to see your nginx.conf. I assume the domains all have the same A records?
– maxf
Aug 27 '16 at 10:24
Yes, all domains A record points to my external IP, then pfsense port forward 80 to proxy same port. Nginx config is simple, and there was no problem before pfSense. Nginx config is very simple, just upstream server 1{ server 192.168.2.12:80; } and proxy_pass server1;
– RomkaLTU
Aug 27 '16 at 10:28
If you have multiple different domains you must have multiple different server blocks to separate them? Also, do clients see the pfSense box IP or the nginx box IP?
– maxf
Aug 27 '16 at 10:33
I added picture of my infrastructure. No problems with domain1.com and domain2.com but subdomain.domain1.com receiving proxy IP...
– RomkaLTU
Aug 27 '16 at 10:43
I updated question. It seams that problem persist only with subdomains.
– RomkaLTU
Aug 27 '16 at 10:23
I updated question. It seams that problem persist only with subdomains.
– RomkaLTU
Aug 27 '16 at 10:23
In that case we need to see your nginx.conf. I assume the domains all have the same A records?
– maxf
Aug 27 '16 at 10:24
In that case we need to see your nginx.conf. I assume the domains all have the same A records?
– maxf
Aug 27 '16 at 10:24
Yes, all domains A record points to my external IP, then pfsense port forward 80 to proxy same port. Nginx config is simple, and there was no problem before pfSense. Nginx config is very simple, just upstream server 1{ server 192.168.2.12:80; } and proxy_pass server1;
– RomkaLTU
Aug 27 '16 at 10:28
Yes, all domains A record points to my external IP, then pfsense port forward 80 to proxy same port. Nginx config is simple, and there was no problem before pfSense. Nginx config is very simple, just upstream server 1{ server 192.168.2.12:80; } and proxy_pass server1;
– RomkaLTU
Aug 27 '16 at 10:28
If you have multiple different domains you must have multiple different server blocks to separate them? Also, do clients see the pfSense box IP or the nginx box IP?
– maxf
Aug 27 '16 at 10:33
If you have multiple different domains you must have multiple different server blocks to separate them? Also, do clients see the pfSense box IP or the nginx box IP?
– maxf
Aug 27 '16 at 10:33
I added picture of my infrastructure. No problems with domain1.com and domain2.com but subdomain.domain1.com receiving proxy IP...
– RomkaLTU
Aug 27 '16 at 10:43
I added picture of my infrastructure. No problems with domain1.com and domain2.com but subdomain.domain1.com receiving proxy IP...
– RomkaLTU
Aug 27 '16 at 10:43
add a comment |
up vote
0
down vote
Ok so problem was not in pfSense and not in proxy, problem was in specific backend server (green square) configuration. I mus accidentally disabled option "Use Client IP in Header" I was sure it was enabled, I know about this option, so it was backend server miss-configuration. Backend server is Litespeed.
answered Aug 28 '16 at 8:24
RomkaLTU
1012
add a comment |
up vote
0
down vote
Ok so problem was not in pfSense and not in proxy, problem was in specific backend server (green square) configuration. I mus accidentally disabled option "Use Client IP in Header" I was sure it was enabled, I know about this option, so it was backend server miss-configuration. Backend server is Litespeed.
answered Aug 28 '16 at 8:24
RomkaLTU
1012
add a comment |
up vote
0
down vote
up vote
0
down vote
Ok so problem was not in pfSense and not in proxy, problem was in specific backend server (green square) configuration. I mus accidentally disabled option "Use Client IP in Header" I was sure it was enabled, I know about this option, so it was backend server miss-configuration. Backend server is Litespeed.
answered Aug 28 '16 at 8:24
RomkaLTU
1012
Ok so problem was not in pfSense and not in proxy, problem was in specific backend server (green square) configuration. I mus accidentally disabled option "Use Client IP in Header" I was sure it was enabled, I know about this option, so it was backend server miss-configuration. Backend server is Litespeed.
answered Aug 28 '16 at 8:24
RomkaLTU
1012
answered Aug 28 '16 at 8:24
RomkaLTU
1012
answered Aug 28 '16 at 8:24
RomkaLTU
1012
answered Aug 28 '16 at 8:24
RomkaLTU
1012
1012
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f305949%2fpfsense-nginx-proxy-and-real-user-ip%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What does your nginx config look like?
– Paul Nordin
Aug 26 '16 at 18:42
Everything is fine with nginx, because as I said: before pfSense I was connected simple consumer router with same port forwards, but after change everything is working same just all users comming from 192.168.2.2 in server logs etc, or simple in php remote_addr
– RomkaLTU
Aug 26 '16 at 18:54
Note sure about terminology, but I think I need NAT reflection, still digging about it...
– RomkaLTU
Aug 26 '16 at 20:02