~eduvpn/server#187: EL9 OpenVPN connect/disconnect SELinux denials

~fkooman 7 months ago

Able to reproduce on Fedora 39 as well:

Feb 13 17:06:53 vpn.example.org audit[11077]: AVC avc:  denied  { map } for  pid=11077 comm="client-connect" path=2F616E6F6E5F6875676570616765202864656C6574656429 dev="hugetlbfs" ino=100911 scontext=system_u:system_r:openvpn_t:s0 tcontext=system_u:object_r:hugetlbfs_t:s0 tclass=file permissive=0
Feb 13 17:06:53 vpn.example.org vpn-user-portal[10747]: CONNECT fkooman PROTO=openvpn (spl-o-4:BQ4vktREu3NFoh2FjFnfbjIirhzNl38Hz8xgPQ3LtTw=) [* => 10.58.175.2,fdf6:fcb1:f82c:146f::2] [AUTH_DATA=]
Feb 13 17:07:01 vpn.example.org audit[11078]: AVC avc:  denied  { map } for  pid=11078 comm="client-disconne" path=2F616E6F6E5F6875676570616765202864656C6574656429 dev="hugetlbfs" ino=100933 scontext=system_u:system_r:openvpn_t:s0 tcontext=system_u:object_r:hugetlbfs_t:s0 tclass=file permissive=0
Feb 13 17:07:01 vpn.example.org vpn-user-portal[10749]: DISCONNECT fkooman (spl-o-4:BQ4vktREu3NFoh2FjFnfbjIirhzNl38Hz8xgPQ3LtTw=) IN=2669, OUT=2540 PROTO=openvpn

~fkooman 7 months ago

Running audit2allow to see what would be required regarding policy changes:

# audit2allow < /var/log/audit/audit.log


#============= getty_t ==============
allow getty_t self:capability2 checkpoint_restore;

#============= openvpn_t ==============

#!!!! This avc can be allowed using the boolean 'domain_can_mmap_files'
allow openvpn_t hugetlbfs_t:file map;
allow openvpn_t self:netlink_generic_socket create;

~gvde 7 months ago · edit

I always have two denials for each connect or disconnect:

type=AVC msg=audit(1707841348.591:50470): avc:  denied  { read write } 
for  pid=62965 comm="client-connect" 
path=2F616E6F6E5F6875676570616765202864656C6574656429 dev="hugetlbfs" 
ino=853432 scontext=system_u:system_r:openvpn_t:s0 
tcontext=system_u:object_r:hugetlbfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1707841348.591:50470): avc:  denied  { map } for 
pid=62965 comm="client-connect" 
path=2F616E6F6E5F6875676570616765202864656C6574656429 dev="hugetlbfs" 
ino=853432 scontext=system_u:system_r:openvpn_t:s0 
tcontext=system_u:object_r:hugetlbfs_t:s0 tclass=file permissive=1

type=AVC msg=audit(1707843799.292:51572): avc:  denied  { read write } 
for  pid=64236 comm="client-disconne" 
path=2F616E6F6E5F6875676570616765202864656C6574656429 dev="hugetlbfs" 
ino=869338 scontext=system_u:system_r:openvpn_t:s0 
tcontext=system_u:object_r:hugetlbfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1707843799.292:51572): avc:  denied  { map } for 
pid=64236 comm="client-disconne" 
path=2F616E6F6E5F6875676570616765202864656C6574656429 dev="hugetlbfs" 
ino=869338 scontext=system_u:system_r:openvpn_t:s0 
tcontext=system_u:object_r:hugetlbfs_t:s0 tclass=file permissive=1

results in

#============= openvpn_t ==============

#!!!! This avc can be allowed using the boolean 'domain_can_mmap_files'
allow openvpn_t hugetlbfs_t:file map;
allow openvpn_t hugetlbfs_t:file { read write };

~fkooman 7 months ago

So, the "fix" is easy, I just don't know if this is smart, or how we can avoid OpenVPN from needing that capability.
# getsebool domain_can_mmap_files
domain_can_mmap_files --> off
Fix:
# setsebool -P domain_can_mmap_files on

~gvde 7 months ago · edit

On 13.02.24 18:15, ~fkooman wrote:
So, the "fix" is easy, I just don't know if this is smart, or how we can avoid OpenVPN from needing that capability.
# getsebool domain_can_mmap_files
domain_can_mmap_files --> off
Fix:
# setsebool -P domain_can_mmap_files on
I don't think that's a good idea, as "domain" stands for all domains, i.e. all possible processes, when all you need is only openvpn_t. Thus if it is really necessary, a specific rule for openvpn_t would be better.

More interesting to me is, why those php scripts need access to hugepages.

~fkooman 7 months ago*

I don't think that's a good idea, as "domain" stands for all domains, i.e. all possible processes, when all you need is only openvpn_t. Thus if it is really necessary, a specific rule for openvpn_t would be better.

Agreed.

More interesting to me is, why those php scripts need access to hugepages.

I have no idea, there's nothing I do that would require that (as far as I know), could also be OpenVPN itself? The lack of this "permission" doesn't seem to break any functionality, right?

~fkooman added v3.x 7 months ago

~fkooman 6 months ago

Do you have idea idea how we can trace where this supposed mmap happens?

~gvde 6 months ago · edit

On 15.02.24 15:14, ~fkooman wrote:

Do you have idea idea how we can trace where this supposed mmap happens?

Generally, you can trace all system calls with strace. That may give you an idea of the context where it happens.

From what I gather, it seems to happen in php before it even loads the script:

#runcon system_u:system_r:openvpn_t:s0 strace

/usr/libexec/vpn-server-node/client-connect ... mmap(0x41600000, 134217728, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED|MAP_ANONYMOUS|MAP_32BIT|MAP_HUGETLB, -1, 0) = -1 ENOMEM (Cannot allocate memory) ... openat(AT_FDCWD, "/usr/libexec/vpn-server-node/client-connect", O_RDONLY) = 4

And the reason why it doesn't make a difference is that it fails anyway, at least for all my tests. Thus it doesn't matter if selinux is interfering or not.

Shortly before the call it's about opcache and I think you can configure opcache with hugepages. Thus, the execution contexts seems to make some sense. As a quick test I have removed the php-opcache package and executed runcon as above again and nothing is logged anymore and the mmap above is not done, either. Thus, I am pretty sure that it's php opcache which causes the mmap.

Weird enough, but opcache.huge_code_pages=0 is in /etc/php.d/10-opcache.ini, thus I don't quite get it why it would try...

The issue underneath it all is that there is no selinux transition into a different context for the client-connect/disconnect scripts. Thus they are called by openvpn and thus executed in the openvpn_t context. openvpn generally doesn't need hugepages and thus the selinux policy doesn't allow for it.

You could transition the scripts into a separate context which allows what you actually need. That would allow you to get rid of the avc.

Otherwise, this log is probably a nuisance to live with. But I guess, some people will unnecessarily allow this to get rid of the message in the audit.log. I personally also prefer no avc to be logged into the audit log. But in this case, it's not so easy, because you don't need it and the mmap probably fails anyway...

Unless: I don't know so much about php cli scripts and which module it uses: can you specify for a script like the client-connect which module to load or better not to load? If you can run client-connect without loading the opcache module that should get rid of the avc...

I think those are the two reasonable options I could think of at the moment.

Cheers,

Gerald

~gvde 6 months ago · edit

Setting opcache.enable_cli=0 in /etc/php.d/10-opcache.ini helps, too. -Gerald

On 15.02.24 15:58, Gerald Vogt wrote:

On 15.02.24 15:14, ~fkooman wrote:

Do you have idea idea how we can trace where this supposed mmap happens?

Generally, you can trace all system calls with strace. That may give you an idea of the context where it happens.

From what I gather, it seems to happen in php before it even loads the script:

#runcon system_u:system_r:openvpn_t:s0 strace

/usr/libexec/vpn-server-node/client-connect ... mmap(0x41600000, 134217728, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED|MAP_ANONYMOUS|MAP_32BIT|MAP_HUGETLB, -1, 0) = -1 ENOMEM (Cannot allocate memory) ... openat(AT_FDCWD, "/usr/libexec/vpn-server-node/client-connect", O_RDONLY) = 4

And the reason why it doesn't make a difference is that it fails anyway, at least for all my tests. Thus it doesn't matter if selinux is interfering or not.

Shortly before the call it's about opcache and I think you can configure opcache with hugepages. Thus, the execution contexts seems to make some sense. As a quick test I have removed the php-opcache package and executed runcon as above again and nothing is logged anymore and the mmap above is not done, either. Thus, I am pretty sure that it's php opcache which causes the mmap.

Weird enough, but opcache.huge_code_pages=0 is in /etc/php.d/10-opcache.ini, thus I don't quite get it why it would try...

The issue underneath it all is that there is no selinux transition into a different context for the client-connect/disconnect scripts. Thus they are called by openvpn and thus executed in the openvpn_t context. openvpn generally doesn't need hugepages and thus the selinux policy doesn't allow for it.

You could transition the scripts into a separate context which allows what you actually need. That would allow you to get rid of the avc.

Otherwise, this log is probably a nuisance to live with. But I guess, some people will unnecessarily allow this to get rid of the message in the audit.log. I personally also prefer no avc to be logged into the audit log. But in this case, it's not so easy, because you don't need it and the mmap probably fails anyway...

Unless: I don't know so much about php cli scripts and which module it uses: can you specify for a script like the client-connect which module to load or better not to load? If you can run client-connect without loading the opcache module that should get rid of the avc...

I think those are the two reasonable options I could think of at the moment.

Cheers,

Gerald

~fkooman 6 months ago

Thanks for the extended analysis! Wow!

Otherwise, this log is probably a nuisance to live with.

This was my idea yeah...

Setting opcache.enable_cli=0 in /etc/php.d/10-opcache.ini helps, too.

Would this have a performance impact? I can imagine that it might? Of course the biggest impact would probably be with php-fpm, but still. So, I wonder whether it is a good idea to disable (cli) opcache on new installations by default, or search some other way?

~gvde 6 months ago · edit

On 15.02.24 17:23, ~fkooman wrote:

Thanks for the extended analysis! Wow!

I have looked at the source code of php opcache and it seems it's using hugetbls pretty dumb: if first tries with the option and if it fails it tries without. Thus, anything but disabling the extension completely won't help.

https://github.com/php/php-src/blob/master/ext/opcache/shared_alloc_mmap.c

Otherwise, this log is probably a nuisance to live with.

This was my idea yeah...

I don't know if that's a good idea. Generally, if there is a problem on the server, I'd like to check for SeLinux issues during troubleshooting. And those avcs are not helping. I like my servers and applications to run without interference of selinux. "ausearch -m avc" should report nothing.

As I wrote before: I think the fully "correct" solution would be to implement a selinux transition for those scripts into a context allow access.

The "shortcut" would be to allow openvpn_t to access hugetbls. Not nice, but I guess it's reasonable risk to take.

The alternative would be not to use php for those scripts in order to make they sure they perform well...

Setting opcache.enable_cli=0 in /etc/php.d/10-opcache.ini helps, too.

Would this have a performance impact? I can imagine that it might? Of course the biggest impact would probably be with php-fpm, but still. So, I wonder whether it is a good idea to disable (cli) opcache on new installations by default, or search some other way?

I can't tell you. You would have to check the performance of those scripts. You could try to run those scripts with "time" and check performance with opcache enabled or disabled.

Regards,

Gerald

~fkooman 6 months ago

I really don't know what to do. Perhaps we can document this issue on https://docs.eduvpn.org/server/v3/selinux.html and be done with it?

Where would we even file this issue upstream if we want to have it "properly" fixed?

~fkooman 6 months ago

https://todo.sr.ht/~eduvpn/server/34

Gerald Vogt 6 months ago · edit

I think the simple solution is to disable the opcache for cli. I think there is little benefit, unless you have php cli scripts which do some heavy lifting. As far as I can tell, the opcache is empty for php cli scripts in the beginning, unless you would use the file cache.

But I don't know what all those eduvpn php cli scripts do and if there are some which really benefit from the opcache for execution.

Otherwise, the "proper" fix doesn't really involve any upstream. That is all correct. openvpn does not require hugetbls this the openvpn selinux policy doesn't allow it.

The problem is that openvpn calls your script which requires hugetbls, even though it's indirectly because of php opcache installed and active and even though it may not even make a difference because php opcache cannot use hugetbls anyway, but unfortunately it tries...

As there is no transition for your script, the script is running as openvpn_t.

Easy fix 1: create and add a selinux module to allow the denied request. That means openvpn_t can use hugetbls. Obviously, that is far more than really needed.

Fix 2: set up a selinux module for those scripts to transition them into a separate type, e.g. eduvpn_ccs_t ("client connect scripts"), which has the necessary policies to allow whatever those scripts need. That way, you don't have to extend what openvpn_t can do and you can limit your scripts to exactly what they need to do. (because currently your scripts could for instance use the openvpn port 1194...)

I'll prepare a sample module for the transition. I have never done it and I am busy this week. Give me some time until I'll get to it. Then we can discuss which it the best way to solve this.

In the meantime, I guess, just put into the docs that this avc denial appears but doesn't affect the operation of the server...

Cheers,

Gerald

On 24.02.24 09:58, ~fkooman wrote:

I really don't know what to do. Perhaps we can document this issue on https://docs.eduvpn.org/server/v3/selinux.html and be done with it?

Where would we even file this issue upstream if we want to have it "properly" fixed?

~fkooman 6 months ago

François Kooman referenced this ticket in commit 2d0fc98.

Gerald Vogt 6 months ago · edit

O.K. I think I found a reasonable way to deal with this without starting your own selinux policy and transition to run those scripts:

The openvpn selinux policy has a boolean openvpn_run_unconfined which you can set. It allows openvpn to run scripts unconfined instead of running them in openvpn_t.

Your scripts need to have type openvpn_unconfined_script_exec_t to transition into the type openvpn_unconfined_script_t used by openvpn.

To test:

#setsebool openvpn_run_unconfined=1

#chcon -t openvpn_unconfined_script_exec_t

/usr/libexec/vpn-server-node/client-*

Now I don't see any avc denials regarding hugetbls whenever I connect or disconnect.

Revert:

#setsebool openvpn_run_unconfined=0

#restorecon -v /usr/libexec/vpn-server-node/client-*

And the denials are back.

Permanent:

#setsebool -P openvpn_run_unconfined=1

#semanage fcontext -a -t openvpn_unconfined_script_exec_t

/usr/libexec/vpn-server-node/client-connect

#semanage fcontext -a -t openvpn_unconfined_script_exec_t

/usr/libexec/vpn-server-node/client-disconnect

#restorecon -v /usr/libexec/vpn-server-node/client-*connect

which should probably go into the spec file for vpn-server-node...

As this is a way implemented by openvpn to handle this kind of situation it seems the best way to go. Definitively much simpler than implementing your own selinux policy.

I hope this helps.

Regards,

Gerald

On 26.02.24 11:31, ~Gerald Vogt wrote:

I think the simple solution is to disable the opcache for cli. I think there is little benefit, unless you have php cli scripts which do some heavy lifting. As far as I can tell, the opcache is empty for php cli scripts in the beginning, unless you would use the file cache.

But I don't know what all those eduvpn php cli scripts do and if there are some which really benefit from the opcache for execution.

Otherwise, the "proper" fix doesn't really involve any upstream. That is all correct. openvpn does not require hugetbls this the openvpn selinux policy doesn't allow it.

The problem is that openvpn calls your script which requires hugetbls, even though it's indirectly because of php opcache installed and active and even though it may not even make a difference because php opcache cannot use hugetbls anyway, but unfortunately it tries...

As there is no transition for your script, the script is running as openvpn_t.

Easy fix 1: create and add a selinux module to allow the denied request. That means openvpn_t can use hugetbls. Obviously, that is far more than really needed.

Fix 2: set up a selinux module for those scripts to transition them into a separate type, e.g. eduvpn_ccs_t ("client connect scripts"), which has the necessary policies to allow whatever those scripts need. That way, you don't have to extend what openvpn_t can do and you can limit your scripts to exactly what they need to do. (because currently your scripts could for instance use the openvpn port 1194...)

I'll prepare a sample module for the transition. I have never done it and I am busy this week. Give me some time until I'll get to it. Then we can discuss which it the best way to solve this.

In the meantime, I guess, just put into the docs that this avc denial appears but doesn't affect the operation of the server...

Cheers,

Gerald

On 24.02.24 09:58, ~fkooman wrote:

I really don't know what to do. Perhaps we can document this issue on https://docs.eduvpn.org/server/v3/selinux.html and be done with it?

Where would we even file this issue upstream if we want to have it "properly" fixed?

~fkooman 6 months ago

Thanks for investigating!

It doesn't look like a good idea to add this to the package directly, it would pretty much result in removing all SELinux protection (if there's actually any for OpenVPN) only to silence a warning. So for now I think the documentation here is sufficient.

~fkooman 6 months ago

I guess if we want to file a bug upstream, it could be for opcache.huge_code_pages being ignored?

Gerald Vogt 6 months ago · edit

On 27.02.24 11:05, ~fkooman wrote:

It doesn't look like a good idea to add this to the package directly, it would pretty much result in removing all SELinux protection (if there's actually any for OpenVPN) only to silence a warning. So for now I think the documentation here is sufficient.

I don't think it's really that bad. It only affects the scripts openvpn calls (which you define in the profile) and only if those scripts are labeled accordingly. Other scripts still run as openvpn_t meaning they can do whatever openvpn can do which is still a lot more. But the boolean is for the purpose that a script called from openvpn can do things that openvpn normally doesn't do.

That is in a way also more future-proof. For instance, if at some point you need to do something more in those scripts or openvpn tightens the selinux policy blocking some operations of your script you'll have to adjust anyway. Currently you can do what openvpn_t can do thus you cannot know how this may change with the next openvpn version...

If you concerned about what the script can do you would have to define your own selinux policy which can be a lot of work depending on what those scripts need to do. With a custom policy you are safe but you need to adjust your policy whenever you change some access which is controlled by selinux...

I don't know if a bug report with php or php module opcache will really be considered. In most cases, php scripts run in context which allow it anyway, e.g.

allow httpd_sys_script_t hugetlbfs_t:file { append getattr ioctl lock map open read write };

I guess that's why hardly anyone notices this...

Regards,

Gerald

~fkooman 4 months ago

It seems on Fedora 40 this is really breaking OpenVPN connections now...

Apr 24 18:15:11 vpn-next.tuxed.net audit[13256]: AVC avc:  denied  { map } for  pid=13256 comm="client-connect" path=2F616E6F6E5F6875676570616765202864656C6574656429 dev="hugetlbfs" ino=176487 scontext=system_u:system_r:openvpn_t:s0 tcontext=system_u:object_r:hugetlbfs_t:s0 tclass=file permissive=0
Apr 24 18:15:11 vpn-next.tuxed.net audit[13256]: AVC avc:  denied  { lock } for  pid=13256 comm="client-connect" path=2F6D656D66643A6F7063616368655F6C6F636B202864656C6574656429 dev="tmpfs" ino=3482 scontext=system_u:system_r:openvpn_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file permissive=0
Apr 24 18:15:11 vpn-next.tuxed.net audit[13257]: AVC avc:  denied  { map } for  pid=13257 comm="client-disconne" path=2F616E6F6E5F6875676570616765202864656C6574656429 dev="hugetlbfs" ino=176490 scontext=system_u:system_r:openvpn_t:s0 tcontext=system_u:object_r:hugetlbfs_t:s0 tclass=file permissive=0
Apr 24 18:15:11 vpn-next.tuxed.net audit[13257]: AVC avc:  denied  { lock } for  pid=13257 comm="client-disconne" path=2F6D656D66643A6F7063616368655F6C6F636B202864656C6574656429 dev="tmpfs" ino=3484 scontext=system_u:system_r:openvpn_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file permissive=0
Apr 24 18:15:12 vpn-next.tuxed.net audit[13258]: AVC avc:  denied  { map } for  pid=13258 comm="client-connect" path=2F616E6F6E5F6875676570616765202864656C6574656429 dev="hugetlbfs" ino=176496 scontext=system_u:system_r:openvpn_t:s0 tcontext=system_u:object_r:hugetlbfs_t:s0 tclass=file permissive=0
Apr 24 18:15:12 vpn-next.tuxed.net audit[13258]: AVC avc:  denied  { lock } for  pid=13258 comm="client-connect" path=2F6D656D66643A6F7063616368655F6C6F636B202864656C6574656429 dev="tmpfs" ino=3486 scontext=system_u:system_r:openvpn_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file permissive=0
Apr 24 18:15:12 vpn-next.tuxed.net audit[13259]: AVC avc:  denied  { map } for  pid=13259 comm="client-disconne" path=2F616E6F6E5F6875676570616765202864656C6574656429 dev="hugetlbfs" ino=176499 scontext=system_u:system_r:openvpn_t:s0 tcontext=system_u:object_r:hugetlbfs_t:s0 tclass=file permissive=0
Apr 24 18:15:12 vpn-next.tuxed.net audit[13259]: AVC avc:  denied  { lock } for  pid=13259 comm="client-disconne" path=2F6D656D66643A6F7063616368655F6C6F636B202864656C6574656429 dev="tmpfs" ino=3488 scontext=system_u:system_r:openvpn_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file permissive=0

~fkooman 4 months ago*

Steps:

setenforce 0

Connect VPN client using OpenVPN

run `cat /var/log/audit/audit.log | audit2allow -M local

edit local.te and remove the stuff that is not relevant

You end up with something like this:
module local 1.0;

require {
	type openvpn_t;
	type tmpfs_t;
	type hugetlbfs_t;
	class netlink_generic_socket { bind create getattr setopt };
	class file { getattr lock map read write };
}

#============= openvpn_t ==============

#!!!! This avc can be allowed using the boolean 'domain_can_mmap_files'
allow openvpn_t hugetlbfs_t:file map;
allow openvpn_t hugetlbfs_t:file { read write };
allow openvpn_t self:netlink_generic_socket { bind create getattr setopt };
allow openvpn_t tmpfs_t:file lock;
Then, continue:

checkmodule -M -m -o local.mod local.te

semodule_package -o local.pp -m local.mod

semodule -i local.pp

This should be enough, you can go back to enforcing, e.g. reboot, and everything should still work.

see audit2allow manpage for more information, "Using audit2allow to generate and build module policy".

TODO: figure out how to make audit2allow -M only add the OpenVPN specific stuff to the .te file so we don't have to modify it anymore.

It is all still a big mess!

~fkooman 4 months ago

Perhaps we can rewrite client-connect and client-disconnect in a simpler language, e.g. posix shell :P

Gerald Vogt 4 months ago · edit

On 24.04.24 20:15, ~fkooman wrote:
It seems on Fedora 40 this is really breaking OpenVPN connections now...
Apr 24 18:15:11 vpn-next.tuxed.net audit[13256]: AVC avc:  denied  { map } for  pid=13256 comm="client-connect" path=2F616E6F6E5F6875676570616765202864656C6574656429 dev="hugetlbfs" ino=176487 scontext=system_u:system_r:openvpn_t:s0 tcontext=system_u:object_r:hugetlbfs_t:s0 tclass=file permissive=0
That's the map on /anon_hugepage (deleted) like before.

Apr 24 18:15:11 vpn-next.tuxed.net audit[13256]: AVC avc: denied { lock } for pid=13256 comm="client-connect" path=2F6D656D66643A6F7063616368655F6C6F636B202864656C6574656429 dev="tmpfs" ino=3482 scontext=system_u:system_r:openvpn_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file permissive=0

That's a lock on /memfd:opcache_lock (deleted)

On EL9 that's not possible either:

#sesearch -A -s openvpn_t -t tmpfs_t -c file

allow domain file_type:file map; [ domain_can_mmap_files ]:True

-Gerald

~gvde 4 months ago

On 24.04.24 20:48, ~fkooman wrote:

Steps:

setenforce 0

Connect VPN client using OpenVPN

run `cat /var/log/audit/audit.log | audit2allow -M local

edit local.te and remove the stuff that is not relevant

There is another problem here: you basically have to make sure that you cover all scenarios for the "Connect VPN client using OpenVPN". Basically you have to check every branch of code in your script, including error/exception branches, to find really everything which may happen.

In addition, there is the problem, that currently you rely on the openvpn selinux policy to let you do whatever you need to do. Your scripts can do what openvpn_t can do. If, in the future, they remove some rules from the openvpn selinux policy because they found they don't need it anymore but your scripts use it, then it'll suddenly fail.

Did you try to set the scripts to openvpn_unconfined_script_exec_t? Does it work then?

As I wrote sometime before: either you set it to openvpn_unconfined_script_exec_t or you need to set up a transition to your own type for your scripts and write a full selinux policy for that.

Using audit2allow you are depending on the specific policy set which is currently active and only find, what's missing. But the currently active policy set is something which is changing over time...

-Gerald

~gvde 4 months ago

For testing, I have set up a eduvpn_client_connect_t type and a transition from openvpn and collected the denials during a normal connect and disconnect on AlmaLinux 9.3. This results in this module:

policy_module(eduvpn_client_connect, 1.0.0)

gen_require(`
	type openvpn_t;
        type bin_t;
	type cert_t;
	type http_port_t;
	type hugetlbfs_t;
	type net_conf_t;
	type tmp_t;
')

########################################
#
# Declarations
#

type eduvpn_client_connect_t;
type eduvpn_client_connect_exec_t;

#application_type(eduvpn_client_connect_t)
#application_executable_file(eduvpn_client_connect_exec_t)

domain_type(eduvpn_client_connect_t)
domain_entry_file(eduvpn_client_connect_t, eduvpn_client_connect_exec_t)

role system_r types eduvpn_client_connect_t;

domtrans_pattern(openvpn_t, eduvpn_client_connect_exec_t, eduvpn_client_connect_t)

permissive eduvpn_client_connect_t;

########################################
#
# eduvpn_client_connect_script local policy
#

allow eduvpn_client_connect_t bin_t:file execute;

allow eduvpn_client_connect_t bin_t:file map;
allow eduvpn_client_connect_t cert_t:dir search;
allow eduvpn_client_connect_t cert_t:file { getattr open read };
allow eduvpn_client_connect_t http_port_t:tcp_socket name_connect;
allow eduvpn_client_connect_t hugetlbfs_t:file { read write };
allow eduvpn_client_connect_t hugetlbfs_t:file map;
allow eduvpn_client_connect_t net_conf_t:file { getattr open read };
allow eduvpn_client_connect_t self:netlink_route_socket { bind create getattr nlmsg_read };
allow eduvpn_client_connect_t self:tcp_socket { connect create getattr getopt setopt };
allow eduvpn_client_connect_t self:udp_socket { connect create getattr };
allow eduvpn_client_connect_t tmp_t:dir { add_name remove_name write };
allow eduvpn_client_connect_t tmp_t:file { create open setattr unlink write };

This doesn't mean it's complete nor that it works the same way on other operating systems. And I am no expert in selinux thus there may be some other macros which would cover it better...

~fkooman 4 months ago*

So yeah, exactly what you warned for happened... SELinux got more strict. I think this approach (trying to fix the policies) will be quite brittle as you mentioned, no matter what, it might make sense to focus on improving connect/disconnect. Other options:

rewrite in shell script (using e.g. curl);

rewrite in Go

try to tweak PHP to not load any extensions, only the ones we need, e.g. #/usr/bin/php -n -d json -d curl -d libsodium (or something like this)

The first two are difficult because /etc/vpn-server-node/config.php is a PHP file, so we'd need to be able to parse PHP. One option could be to rewrite the configuration file to JSON/ini/some key value format on vpn-server-node package upgrade and then make the connect/disconnect tool use that.

~fkooman 4 months ago*

Setting opcache.enable_cli=0 in /etc/php.d/10-opcache.ini helps, too. -Gerald

Perhaps we can use this setting in the top of the connect/disconnect script?
$ php -i | grep opcache.enable_cli
opcache.enable_cli => On => On
$ php -d opcache.enable_cli=0 -i | grep opcache.enable_cli
opcache.enable_cli => Off => Off
So, we can modify client-connect and client-disconnect scripts in /usr/libexec/vpn-server-node and set this as the first line:
#!/usr/bin/php -d opcache.enable_cli=0
This seems to silence all SELinux stuff and make things work again... I'd be more comfortable to do this on a per script basis than "globally" disabling it. What do you think?

~gvde 4 months ago

I don't think SeLinux got more strict. I guess they changed where php/opcache keeps it lock file and moved it into memfd. openvpn doesn't use memfd for lock files.

I think the biggest obstacle is that php is not really considered for cli and even less for scripts called from something else but httpd. There are macros for apache and httpd in selinux but nothing for php cli.

And I don't think using a different script language will really help permanently. You are just trying to avoid making calls which openvpn usually doesn't do.

The real issue remains: you are running as openvpn_t, i.e. with selinux rules designed to run openvpn but not arbitrary scripts within openvpn.

Therefore the real solution is not to run as openvpn_t. The solution built into openvpn is to use openvpn_unconfined_script_exec_t.

The alternative is to build your own selinux module, make a transition into your own type and keep your policy up-to-date yourself.

I still think, unless you have the time to do the alternative, openvpn_unconfined_script_exec_t is the way to go.

~gvde 4 months ago

#!/usr/bin/php -d opcache.enable_cli=0`
This again tries to mitigate/avoid issues and doesn't solve it. Whenever php changes something in paths, it may break your script. It may even break now if a user changes some paths in configuration putting locks/config/whatever in paths with contexts which httpd/php can read but not openvpn.

The basic problems remains that you are running php with openvpn_t, and php doesn't consider openvpn and openvpn doesn't consider php. Thus the result is always somewhat "random".

~fkooman 4 months ago

Okay... I guess it makes most sense than to set the context of the binaries during %post of the RPM install. I also already do something similar for the vpn-user-portal data directory:

https://git.sr.ht/~fkooman/vpn-user-portal.rpm/tree/v3/item/SPECS/vpn-user-portal.spec#L206

~fkooman 4 months ago

$ ls -lZ /usr/libexec/vpn-server-node/
total 16
-rwxr-xr-x. 1 root root system_u:object_r:openvpn_unconfined_script_exec_t:s0 1181 May 11  2023 client-connect
-rwxr-xr-x. 1 root root system_u:object_r:openvpn_unconfined_script_exec_t:s0 1236 May 11  2023 client-disconnect
-rwxr-xr-x. 1 root root system_u:object_r:bin_t:s0                             748 May 11  2023 generate-secrets
-rwxr-xr-x. 1 root root system_u:object_r:bin_t:s0                            1084 May 11  2023 server-config

This doesn't work, still spews the same SELinux warnings, errors.

Commands used:

$ sudo semanage fcontext -a -t openvpn_unconfined_script_exec_t /usr/libexec/vpn-server-node/client-connect 
$ sudo semanage fcontext -a -t openvpn_unconfined_script_exec_t /usr/libexec/vpn-server-node/client-disconnect 
$ sudo restorecon -R /usr/libexec/vpn-server-node

~fkooman 4 months ago

Hm... maybe it DOES work, I restarted OpenVPN services and now the log stays silent...

~fkooman 4 months ago

Or not, broken again after reboot.

~fkooman 4 months ago

@gvde do you have any ideas what would fix this (permanently) what am I missing?

~gvde 4 months ago

Did you enable the boolean to enable the whole thing?

# setsebool -P openvpn_run_unconfined=1

# semanage boolean -l -C
SELinux boolean                State  Default Description

httpd_can_network_connect      (on   ,   on)  Allow HTTPD scripts and modules to connect to the network using TCP.
openvpn_run_unconfined         (on   ,   on)  Allow openvpn to run unconfined scripts

~fkooman 4 months ago

Did you enable the boolean to enable the whole thing?

Okay, that was it, thanks!

~fkooman 4 months ago

Fix for deploy scripts: https://codeberg.org/eduVPN/deploy/commit/1f8fb93d29c565e14e42a8f4e98cf44fbff15574

~fkooman 4 months ago

Fix RPM package: https://git.sr.ht/~fkooman/vpn-server-node.rpm/commit/6432abfbff59c15e6fdec2844bf6ecf24d188953

~fkooman 4 months ago

Will build development RPM packages now and deploy on a new Alma 9 and Fedora 40 test system before building production packages.

~fkooman 4 months ago

will test later new deploys, it works fine when updating existing systems.

~gvde 4 months ago

Looks good. Maybe you want to remove that comment # XXX is this still needed? from the deploy scripts as well?

I have quickly tested it and it's still necessary:
#============= httpd_t ==============

#!!!! This avc can be allowed using one of the these booleans:
#     httpd_can_network_connect, httpd_can_network_relay, httpd_can_connect_ftp, httpd_use_openstack, nis_enabled
allow httpd_t ephemeral_port_t:tcp_socket name_connect;

#!!!! This avc can be allowed using one of the these booleans:
#     httpd_can_network_connect, httpd_graceful_shutdown, httpd_can_network_relay, nis_enabled
allow httpd_t http_port_t:tcp_socket name_connect;
httpd connects to port 80 as we have OSCP stapling enabled and php-fpm (running as httpd_t) connects to the vpn-daemon on port 41194. Thus, httpd_can_network_connect=1 is necessary, though not for apache to connect to php-fpm.

~fkooman 4 months ago

Maybe you want to remove that comment # XXX is this still needed?

Yeah, I did that in the same patch as when I added the openvpn unconfined bool :)

Thanks for your persistence and help! I'll test a bit more in the next days and will push a prod update then if all works.

~gvde 4 months ago

We are running our eduVPN test and production servers like that for a while now and haven‘t seen any issues or denials.

~fkooman REPORTED CLOSED 4 months ago

https://docs.eduvpn.org/server/v3/selinux.html#policy-updates

https://lists.geant.org/sympa/arc/eduvpn-deploy/2024-04/msg00006.html