-
Notifications
You must be signed in to change notification settings - Fork 18.8k
Docker Network bypasses Firewall, no option to disable #22054
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Note: I noticed in https://github.com/docker/docker/blob/master/vendor/src/github.com/docker/libnetwork/iptables/iptables.go that there is not even an existing option to set the source, so it's just using the iptables defaults for source ip/device. |
Duplicate of #14041? |
It is not quite #14041 as this issue is talking about exposed ports. Exposing ports is intended to make them publicly accessible, as this is how you expose services to the outside world. If you are working in a development environment, you can either disable access to ports from outside your computer with a host firewall, or simply not expose the ports and access the services directly, or from other containers on the same network. I would recommend you use the newer docker networking features to set up private networks for services that you do not want exposed at all, see https://docs.docker.com/engine/userguide/networking/ |
That's what I thought of first; but was a bit confused, because exposing a port ( If you're actually talking about publishing, then this is as designed; In your example, container B and C should not publish their ports, and container A can communicate with them through the Docker Network, e,g.
This only publishes the "web" container to the host, The web container can access the "API" and "database" containers through their name (I.e. http://api:80/, and db:3306 (assuming MySQL)) |
@justincormack so I don't think using a private network solves the issue. In my case I'm using a private network between the containers, and they're still publicly exposed because the host firewall isn't configured to limit the exposure to the private network. @thaJeztah the issue still comes down to the firewall support - there's no firewall support in docker to limit it to a specific network. You can probably still access those containers from another system as the firewall will not prevent other systems from accessing the port on the host. Now I'm running this via docker-compose; however, it's not entirely a docker-compose issue since the libnetwork functionality has no capability of limiting the network in the firewall rules - iptables rules have no source specification so regardless of how one configures the network as long as one relies on docker to create the firewall rules (which one should because it's more likely to get them right) then this becomes an issue. Consider the following in a docker-compose.yml file:
The above is an excerpt from one of my projects. While I want to be able to test all of them locally from my host, I don't want anyone else to be able to access anything but the nginx instance. I'm not sure how this translates to your nomenclature...it may be that this is part of the "publishing" aspect, and the publishing capability needs to be expanded to do what I'm saying. If this is by design, then it's a poor security model as you now expose all developers to extreme risks when on unfamiliar networks (e.g traveling). As I said, I don't expect the default to change immediately but having the option would be a good first step. |
I am a bit confused then, can you give some examples externally of what you can connect to? The backend services will be (by default) on the There is a potential issue if your external IP is also a private IP that traffic will not be dropped that is routed for the internal networks (whereas it should be from public to private) - is that the issue? |
@justincormack so I'm primarily setting up proper proxying so that some services can only be hit via the proxy (nginx - ssl termination), which then filters through an authentication proxy (repose), and finally off to another service (phoenix). I could care less if all of them are bound to the 0.0.0.0 interface; but I only want nginx to be externally accessible (or at the very least the repose portion if I didn't have nginx in place here). An easy solution, for example, would be to not have to set "127.0.0.1" in the configuration, but have a firewall section where it's easy to specify that to allow through the firewall with a base configuration of only the docker network and local host (loopback) interfaces enabled to talk - something like:
Now the situation can be mitigated somewhat by limited the network mapping on the host to 127.0.0.1 instead of the default 0.0.0.0 map. Note that this is what really mitigates it because otherwise the bridging will forward the host port into the docker network. And yes, I did verify that that limiting works; however, it still leaves potential vulnerabilities in place and the firewall rules do not match what is actually being done. As another example, there was a Linux Kernel vulnerability a little while back (having trouble finding it at the moment) that was related to ports that were marked in IPtables as being opened for use by applications, but then not actually being connected to an application - for instance, being on a local host port but not a public IP port. This potentially sets that up, and it would be better practice to limit the IPtables rules to the expected networks instead of leaving them open to connect from any where. As I said, at the very least have the option to specify. They've likely fixed that particular issue but why leave the possibility open? IOW, it's all about security. |
@BenjamenMeyer if you don't want the other services to be accessible, why do you publish their ports at all? i.e. |
One issue I have that's related to this one is that I would like to publish ports, but only allow certain IP addresses to access them. For example, I run a Jenkins environment in a couple of containers. The master node is "published", but I have to make some pretty convoluted iptables rules to lock it down so that only the 2 offices we have can access it. Is there a way around this currently built into Docker? Or at least a recommended practice? I've seen in the documentation how you might restrict access to 1 IP address; but not several. The other issue with this is that if you have a server that already has an iptables configuration, you might be resetting all of the rules before applying your rules (hence the convoluted rules I have had to set up). |
I have an issue similar to the one stated by @seeruk. There is a jarring violation of expectation when preexisting firewall rules don't apply to the published container ports. The desired behavior is as follows (for me at least)
Is there a succinct way to achieve this in iptables, or does it not easily permit such a construct. I'm particularly limited in my knowledge of iptables so bear with me. I've just recently picked up knowledge about it while trying to understand docker's interactions with it. |
What I've actually resorted to for the time being, since I am actually running these containers on a pretty powerful dedicated server, I've set up a KVM VM running Docker, then using some more standard iptables rules to restrict access from the host. The VM has it's own network interface that's only accessibly from the server, so I have to add rules to explicitly allow access to ports in iptables on the host. I have lost a little bit of performance, but not much. |
@thaJeztah I want to be able to access it from the local system, and test against it easily. For example, setting up a RESTful HTTP API that has a Health end-point and being able to reliably run For @seeruk's case, being able set an IP block (5.5.0.0/16 - a valid parameter for a source address in iptables rules) would be a very good thing. IPtables already has the capability to do the limiting, but docker is not taking advantage of it. |
@thaJeztah I set My work around right now is to turn off docker containers before I leave for the day because I can't ensure the environment I want to be external actually is external, or that the environment is properly limited for security purposes. |
@BenjamenMeyer one way to do this is running those tests in a container, e.g.
|
The issue that Ben is bringing to light is real and surprising (a bad combination). Many admins, like myself, are using the tried-and-true ufw firewall. Docker is doing and end-run around ufw and altering the iptables rules is in such a way that it 1) causes ufw to misreport the current status of the packet filtering rules, and 2) exposes seemingly private services to the public network. In order for docker to remain in the good graces of the sysadmin community, another approach must be devised. Right now there are many admins out there, who, like Ben and myself, inadvertently opened ports to the wider Internet. Unlike Ben and myself though, they have not figured it out yet. |
@thaJeztah that assumes that I am doing it via the command-line and not using another tool where I only have to set an IP address. For example, I'm working on an API. I have a tool that I can use to work with that API in production to support it; for development of the tool and the API I want to just point the tool at the dockerized API. The tool knows nothing about docker, nor should it. And I don't necessarily want to put the tool into docker just to use it - pointing it at a port exposed only to the local host should be sufficient. |
@jcheroske I agree, but I don't know that there's a good solution to that aspect. For that, That said, it would be nice if Docker could integrate with those to dump out the appropriate config files to be able to enable/disable them, or integrate with those tools such that it gets hooked in and dumps out the information appropriately, however, given there are better solutions I don't think that aspect will really be solved. Here, it's more about just limiting the scope of the |
If you are willing to do so, using iptables does make this totally possible. This is a trimmed-down example of how you can use it: https://gist.github.com/SeerUK/b583cc6f048270e0ddc0105e4b36e480 You can see that right at the bottom, 1.2.3.4 is explicitly given access to port 8000 (which is exposed by Docker), then anything else to that port is dropped. The PRE_DOCKER chain is inserted to be before the DOCKER chain so that it is hit first, meaning the DROP stops the blocked requests from ever reaching the DOCKER chain. It's a bit annoying that Docker doesn't have this functionality built-in, but it is possible work around it right now. Another alternative would be using an external firewall. Some places like AWS and Scaleway offer things like security groups where you can manage access to your boxes from outside, from there every port behaves the same way. I never actually managed to figure out how to make this work with UFW. Though for now, I'm happy with using iptables as a solution. It seems to be working out very well for me so far. Obviously, this isn't much of a great solution if you have already built a reasonably complicated set of firewall rules around UFW. Though, it does make using things like iptables-persistent quite easy. You can also use alternative ways of allowing access to this way that seem more "normal" in iptables. |
@BenjamenMeyer have you thought of using an user-defined
With this setup, myservice2 can reach myservice1 by name Also with compose 1.7, you can specify static ip address for containers and specify network subnets and ranges. |
I did figure out a simple workaround.
So simple that it really makes me wonder why the https://fralef.me/docker-and-iptables.html |
I can't get docker to stop modifying iptables to save my life. Tried updating /etc/default/docker to no avail on Ubuntu 16.04 |
@enzeart Try |
@seeruk Bless your soul |
@enzeart to configure a daemon running on a host that uses systemd, it's best to not edit the docker.unit file itself, but to use a "drop in" file. That way, you won't run into issues when upgrading docker (in case there's a newer docker.unit file). See https://docs.docker.com/engine/admin/systemd/#custom-docker-daemon-options for more info. You can also use a daemon.json configuration file, see https://docs.docker.com/engine/reference/commandline/daemon/#daemon-configuration-file |
@mavenugo There's already a docker network in place. @jcheroske that works, but as I noted it would mean that the end-user (me) would then have to make sure that all |
@msimkunas If you don't want to use a VM, look into Rootless Docker. Edit: I wonder how many downvotes I'll get this time |
@schklom Actually, I prefer running multiple VMs with Docker inside for isolation reasons. In the worst case, if something doesn’t work, I can just destroy the VM from scratch. |
Saw https://www.linkedin.com/pulse/docker-engine-v28-hardening-container-networking-default-docker-wvrse - glad this conversation contributed to the changes for v28. Certainly looking forward to trying it to see if this issue persists. |
I do not have a lot fo experience with Docker but I've setup a brand new Docker 28 and I do not see a lot of change related to network security ? Docker is still bypassing UFW rule for published ports ? |
Yes, this still holds true ... https://docs.docker.com/engine/network/packet-filtering-firewalls/#docker-and-ufw |
PSA for anyone else using carefully-crafted iptables restore scripts that avoid clobbering DOCKER rules - beware of this lovely iptables bug where user-defined tables are flushed even with
So if you specify |
Personally I've settled onto running Docker - if I cannot use something else - behind other dedicated infrastructure (whether a VM, a separate host behind a firewall, ...) precisely because I know I am not a competent enough user of iptables. I don't work with it anywhere near enough so I don't get to learn from spaced repetition. The things I inevitably re-discover those rare times I do work with iptables do not to stick in my head for rapid recall. I find the mental overhead in simply treating a machine running Docker as something that can't be secured on its own to be significantly lower than the overhead brought by implementing workarounds. EDIT: to clarify, this is not meant as a critique to @SystemParadox 's PSA, which I actually really appreciate even as someone who doesn't directly benefit from it. I'm just sharing my perspective as someone who uses containers on a daily basis, both in development and production environments, but that also, like many others, is first and foremost a developer. I do have enough sysadmin chops to get by in most cases but some (most?) of the stuff I've seen in this thread goes well above my head. |
@jacoscaz Exactly my thinking as well. I could hack around and come up with a solution if I wanted. Do I want this maintenance overhead which will inevitably break at some point? No. |
As someone who has maintained their own iptables rules alongside docker for a long time I actually completely agree with you. I consider myself an iptables expert and I'm still tripping up on this. I just cannot recommend that anyone else even tries it - it's just too dangerous. |
Are the limitations of Rootless Docker so consequential that people would rather use custom iptables or be in a VM instead? |
@schklom Personally, I've been using VMs for development for years so yes, I'd rather use a VM for simplicity so that I don't have to tinker with custom iptables rules. Custom setups increase the risk of breakage so I tend to avoid that. |
Maybe we can provide a less technical summary of the state of things that will help stick in more heads. Please correct me if I'm wrong: `If you run iptables, nftables or UFW, then you can only consider it applies to applications running on bare metal. Whatever is running in via docker (and other container systems on Linux?) should be considered by default to be running as if iptables/nftables/UFW is not enabled on the host system. |
I've tried to add this config in /etc/ufw/after.rules but published port in docker config file are still reachable from internet. My knowledge related to network config is limited, any idea ? Source : https://github.com/chaifeng/ufw-docker BEGIN UFW AND DOCKER*filter -A DOCKER-USER -j RETURN -s 10.0.0.0/8 -A DOCKER-USER -p udp -m udp --sport 53 --dport 1024:65535 -j RETURN -A DOCKER-USER -j ufw-docker-logging-deny -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 192.168.0.0/16 -A DOCKER-USER -j RETURN -A ufw-docker-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW DOCKER BLOCK] " COMMIT |
Hi @Tipiak99 - could you expand on that a bit? What does your port publishing rule look like, and what do you want it to do? As I understand it, the bad interaction between docker's rules and ufw means it's not possible to filter access to a published port from specific external hosts. (So, once the port is published to an address on a host's external interface, it can be accessed from anywhere. But, only published ports can be accessed.) If you don't want to allow any external access, "-p 127.0.0.1:8080:80" should forward from port 8080 on the host's loopback address to the container's port 80, while disabling access from outside the host. |
Hoping to clarify that ...
|
@robmry thanks for your explanations! I've done the following thing to secure my docker :
|
How does this secure your docker containers? Packets can still be routed to container IP addresses, as has already been demonstrated here. |
Hi @msimkunas - not since moby 28.0 ... https://www.docker.com/blog/docker-engine-28-hardening-container-networking-by-default/ |
Not 100% sure but in the example, they are publishing port to localhost while me i'm not publishing port of containers at all (except port of nginx proxy) |
@robmry Thanks. I don't understand what's the point in fiddling with firewall rules after v28, though. |
Sorry @msimkunas, I'm not sure what you mean ... who's fiddling with what rules? (@Tipiak99 - your setup sounds good to me!) |
@robmry Sorry, I was under the impression that people here were still trying to resolve the issue fixed in v28 by manually editing firewall rules. |
Oh, sure - got it! Thanks @msimkunas. |
@robmry It's good to hear that things have improved in v28. Personally though, I'm still much more comfortable running such trigger happy software like Docker inside a VM because editing firewall rules in the host is a privilege I don't feel like giving out lightly. Like someone else already said a few comments ago, the cognitive overhead of treating Docker as something to be contained or deployed behind dedicated infrastructure is lower than the overhead of making sure it's not misconfigured, and doesn't perform undesired firewall changes. I'm very picky about which software gets to mess with my firewall and containing Docker inside a VM gives me more peace of mind. |
We're in a situation where we have to sandbox an increasing amount of bad decision making leading to an angry "turtles all the way down" situation. Most of my docker usage is on VPSs which are already VMs and if I must then run another VM inside of those VPSs just to "sandbox" docker behavior - and I'm using docker just to easily use a vast array of images (as well as my own) - then it has just gone too far imo. |
@dm17 Yeah, it can get tricky once you start playing around with nested containment... My own use case is simpler, it's local development. To me it's infinitely easier to just run multipass launch docker and start working instead of installing Docker on bare metal and going through pages of documentation and trying to verify that Docker does not expose any of its resources in some unexpected manner (and the more complex your setup is, the higher the cognitive load). I understand though that not all use cases are like this and once you get to nested virtualization all of this can seem a bit too much. Which is why it's nice that at least something is improving with v28. |
@robmry I'm not sure v28 fully solves it yet - the underlying issue is still there AFAICT; it's just mitigated now since the default binding address is no longer So this is certainly 100% better. I'd need to do some research to see how close this is to being resolved though. |
Hi @BenjamenMeyer - the default binding address is still There are now rules in |
Output of
docker version
:Output of
docker info
:Additional environment details (AWS, VirtualBox, physical, etc.):
Rackspace Cloud Server, Ubuntu 14.04, but that shouldn't really matter
Steps to reproduce the issue:
Describe the results you received:
root@brm-pheonix-dev:~/rse# iptables --list DOCKER
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:6379
Describe the results you expected:
root@brm-pheonix-dev:~/rse# iptables --list DOCKER
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- 127.0.0.0/24 172.17.0.2 tcp dpt:6379
ACCEPT tcp -- 172.16.0.0/16 172.17.0.2 tcp dpt:6379
Additional information you deem important (e.g. issue happens only occasionally):
By default docker is munging the firewall in a way that breaks security - it allows all traffic from all network devices to access the exposed ports on containers. Consider a site that has 2 containers: Container A exposes 443 running Nginx, and Container B runs an API on port 8000. It's desirable to open Container A to the public for use, but hide Container B entirely so that it can only talk to localhost (for testing by the user) and the docker network (for talking to Container A). It might also be desirable for testing purposes to have Container C be a database used by Container B with the same kind of restrictions.
I found this because of monitoring logs on a service I had thought was not open to the public. After finding log entries from sources trying to break in, I checked the firewall rules and found there was no limit on the source addresses or interfaces. I use UFW and only allow SSH onto this particular box, and would prefer to keep it that way. This can dramatically impact using Docker containers to deploy services and lead to potential security problems if people are not careful.
The best security practice would be to by default limit the networking to work like above desired effect example, and then allow the user to add the appropriate firewall, etc rules to override such behavior, or have an option to revert to the current behavior. I know that for legacy reasons that is not likely since it would break a lot of things on up-date; so at least having an option to enable the above that can be turned on now would be a good first step, and perhaps later after much warning make it the default behavior. Assuming the default behavior is secure, having functionality to manage this (firewall->enable public port, ip) in the docker-compose yml would be a great way to visibly make it known what is going on.
I did find the --iptables=false option, however, I don't want to have to be setting all the rules myself. The only thing I am objecting to is the source setting for the rules.
While I have not verified it, I suspect all the firewalls supported by docker will have the same issue.
The text was updated successfully, but these errors were encountered: