Support OpenSSL 3 FIPS

[junaruga]

I would like to use this ticket to manage all the issues related to OpenSSL 3 FIPS. The goal of this ticket is to pass all the unit tests with the OpenSSL 3.2, 3.1 and 3.0 FIPS cases on the CI.

FIPS related issues/pull-requests

I update the list regularly. The issues are sorted by newest first.

• 17: hmac: enable tests in the FIPS mode #1042
• 16: Fix test_pkcs12.rb in FIPS. #997
• 15: Fix test_digest.rb in FIPS. #995
• 14: Fix test_cipher.rb in FIPS. #994
• 13: Update keys used in tests #953
• 12: Fix test_ssl.rb in FIPS. #937
• 11: Rakefile: Manage test files by excluding test files in the test_fips task. #820
• 10: Add passing test files in FIPS. #819
• 9: Fix test_provider.rb in FIPS. #794
• 8: Fix test_pkey_rsa.rb in FIPS. #790
• 7: Fix test_pkey_dsa.rb in FIPS. #729
• 6: Fix test_pkey_dh.rb in FIPS. #694
• 5: Fixed test/openssl/test_pkey_ec.rb. test_pkey_ec.rb test failures in OpenSSL FIPS #671 fixed by test/openssl/test_pkey_ec.rb: refactor tests for EC.builtin_curves #675, and Fix test_pkey_ec.rb on FIPS. #681.
• 4: Fixed the second issue in OpenSSL::PKey.read and test/openssl/test_pkey.rb. The issue was fixed by the the workaround ossl_pkey.c: Workaround: Decode with non-zero selections. #669.
  • ed25519: OpenSSL 3.0.8: ed25519 a decode from and then encode to a pem file corrupts the key if fips+base provider is used openssl/openssl#20758
  • x25519: OpenSSL 3: x25519 a decode from and then encode to a pem file corrupts the key if fips+base provider is used openssl/openssl#21493
  • Fixed PR in OpenSSL Decoder key export fixes openssl/openssl#21519
• 3: OpenSSL 3 FIPS mode - creating encrypted RSA key pair fails with PEM_write_bio_PrivateKey_traditional: initialization error (OpenSSL::PKey::PKeyError) #643
• 2: Fixed an issue in OpenSSL::PKey.read. Applied the workaround PR Fix OpenSSL::PKey.read that cannot parse PKey in the FIPS mode. #615 to avoid the OpenSSL issue OSSL_DECODER_CTX_set_selection doesn't apply the selection value properly openssl/openssl#20657.
• 1: Fixed the OpenSSL.fips_mode in OpenSSL 3.0+. The issue: OpenSSL 3: OpenSSL.fips_mode returns false in FIPS enabled environment #605, the PR: Implement FIPS functions, adding OpenSSL FIPS mode case on CI. #608

Remaining tasks to the goal

Fix other test files test/**/test_*.rb to pass in CI. The fixed test files 18/33.

$ date
Wed Oct  4 04:54:22 PM CEST 2023

$ find test/ -name "test_*.rb" | wc -l
33

Documents

• How to debug Ruby OpenSSL binding with OpenSSL 3 FIPS: Contribution is welcome!
• Check openssl-head fips, openssl-3.1.z fips or openssl-3.0 fips cases in this repository's GitHub Actions YAML file to know how to compile Ruby OpenSSL binding with OpenSSL FIPS.

Original comment

I deleted to simplify this issue ticket. I think we don't need the original comment's information any more.

[jackorp]

I'll add what we learned on our side in more detail.

Downstream bug report: https://bugzilla.redhat.com/show_bug.cgi?id=2170105

Some words and what they refer to throughout this report:
rubygem-openssl = this repo, the OpenSSL bindings gem shipped with Ruby
OpenSSL / C OpenSSL / libopenssl = the system OpenSSL

Problem description:
rubygem-openssl cannot load keys when the system is in FIPS mode. This was found on RHEL 9 and applies to RHEL 9.0 up to the package set available in C9S at the time of this report, which translates to RHEL 9.2.

Reproducer on RHEL 9:
0. have FIPS enabled system
1.a $ sudo dnf install ruby -- this will install Ruby 3.0 with backported patches to work with OpenSSL 3.0
1.b Optionally, you can use Ruby 3.1 which ships with rubygem-openssl 3.0 on RHEL 9.1 and later:

$ sudo dnf module enable ruby -y
$ sudo dnf install ruby -y

2. $ openssl genrsa -out key.pem 4096 -- create a key for reproducer purposes
3. ruby -ropenssl -e "OpenSSL::PKey.read(File.read('key.pem'))"

Actual results:

-e:1:in `read': Could not parse PKey (OpenSSL::PKey::PKeyError)
	from -e:1:in `<main>'

Expected results:

No errors

This was observed with OpenSSL versions 3.0.1 through 3.0.7 and with rubygem-openssl 2.2.1, 3.0.0, 3.1.0 up to the more recent commits on the master branch of this repo: 9ea934a .

The problem is that ossl_pkey_read_generic fails to correctly parse the key.

I have distilled Ruby's code flow into a simple C example to illustrate better what is happening. See the comment on the downstream ticket: https://bugzilla.redhat.com/show_bug.cgi?id=2170105#c7

I found that initializing OSSL_DECODER_CTX_new_for_pkey(...) with EVP_PKEY_KEYPAIR as the value for the selection argument of the function allows for the decoding to succeed. However, if I reuse the OSSL_DECODER_CTX structure, where selection is 0 and use OSSL_DECODER_CTX_set_selection before decoding with EVP_PKEY_KEYPAIR, decoding fails unless there is a proper call to OpenSSL_init_ssl() at the beginning of the main function of the C example. Currently these OpenSSL flags work for the example:

OPENSSL_INIT_NO_ADD_ALL_DIGESTS
OPENSSL_INIT_NO_LOAD_CONFIG
OPENSSL_INIT_ENGINE_RDRAND
PENSSL_INIT_ENGINE_CRYPTODEV
OPENSSL_INIT_ENGINE_PADLOCK
OPENSSL_INIT_ENGINE_ALL_BUILTIN

I'll emphasize that it is important that OSSL_DECODER_CTX_new_for_pkey with different selection arguments works when decoding a key in BIO struct multiple times.

I have also used system's python (python 3.9), where loading pem key simply works:

$ sudo dnf install -y python3-cryptography
$ cat script.py
from cryptography.hazmat.primitives import serialization

file = open("key.pem", "rb")
pem_bytes = file.read()
file.close()
print(serialization.load_pem_private_key(pem_bytes, password=None))
$ python script.py
<cryptography.hazmat.backends.openssl.rsa._RSAPrivateKey object at 0x7ff5ec2a7940>

They seem to initialize OpenSSL similarly to Ruby

From inspecting ltraces of both languages I noticed that python uses EVP_MD* function family from openssl unlike rubygem-openssl, where OSSL_DECODER* function family is used.

Traces for python and Ruby:
python was ran as follows: python script.py
python_crypto_ltrace.txt
python_crypto_strace.txt
Ruby was ran as follows: ruby -ropenssl -e "OpenSSL::PKey.read(File.read('key.pem'))"
rubygem_openssl_ltrace.txt
rubygem_openssl_ltrace.txt

To me it seems something isn't being loaded properly in Ruby when system is in FIPS mode as calling OSSL_PROVIDER_load(NULL, "fips") fixes the situation when inserted into rubygem-openssl code before decoding the key. However, from what I understand from Dmitry Belyavskiy on the downstream ticket, that call should not be needed, as correct providers should be loaded automatically, assuming OpenSSL was initialized: https://bugzilla.redhat.com/show_bug.cgi?id=2170105#c29

[junaruga]

First, good news is the Ruby project has a RHEL 9.1 server that is one of the servers used in Ruby CI, where maintainers can access by SSH. And I chat with hsbt, asking him if we can have the RHEL 9 server with FIPS mode enabled. After that, maintainers can access to the server by SSH.

Note that now the maintainers can try the FIPS mode enabled environment on the RHEL 9.1 server in the Ruby project. For the steps, see #601 (comment) . I was able to reproduce this issue ticket' error on the environment.

[junaruga]

$ sudo dnf install -y python3-cryptography
...

I noted the the mentioned RPM package "python3-cryptography" is not available on Fedora: https://src.fedoraproject.org/rpms/python3-cryptography .

So, just installing the PyPI package "cryptography" directly is convenient. The GitHub is here.

$ pip3 install --user cryptography

$ pip3 list | grep cryptography
cryptography         37.0.2

[junaruga]

We can reproduce this issue on GitHub Actions after this PR #608 is merged!

[jackorp]

I noted the the mentioned RPM package "python3-cryptography" is not available on

I am not sure you are right in this. I was able to find this via both the dnf info and dnf install on containers of Fedora 36 up to Rawhide.

it is actually coming from this repo: https://src.fedoraproject.org/rpms/python-cryptography

[junaruga]

@jackorp You are right. Thanks for the info. I also can see the installed package on my environment. What was I seeing at that time... Sorry.

$ cat /etc/fedora-release 
Fedora release 37 (Thirty Seven)

$ rpm -q python3-cryptography
python3-cryptography-37.0.2-4.fc37.x86_64

It seems that the python-cryptography repository's python-cryptography.spec (recipe file) manages the entry of the python3-cryptography package.

[junaruga]

I am debugging this issue on Fedora 37. Let me share my current status. I think my small program: fips_mode command is helpful for you to check if you are running the correct FIPS mode on your environment. You can run like this.

$ cat /home/jaruga/.local/openssl-3.0.8-fips/ssl/openssl_fips.cnf
config_diagnostics = 1
openssl_conf = openssl_init

.include /home/jaruga/.local/openssl-3.0.8-fips/ssl/fipsmodule.cnf
#.include ./fipsmodule.cnf

[openssl_init]
providers = provider_sect
alg_section = algorithm_sect

[provider_sect]
fips = fips_sect
base = base_sect

[base_sect]
activate = 1

[algorithm_sect]
default_properties = fips=yes

$ LD_LIBRARY_PATH=/home/jaruga/.local/openssl-3.0.8-fips/lib/ \
  OPENSSL_CONF=/home/jaruga/.local/openssl-3.0.8-fips/ssl/openssl_fips.cnf \
  ~/git/openssl-test/fips_mode
FIPS mode provider available: 1
FIPS mode enabled: 1

I am debugging this issue with GDB on the commits in the #608 on Fedora. There is an issue gdb with LD_LIBRARY_PATH=/path/to/openssl_lib (this issue). So, I am setting the LD_LIBRARY_PATH in the GDB prompt.

$ OPENSSL_CONF=/home/jaruga/.local/openssl-3.0.8-fips/ssl/openssl_fips.cnf \
  gdb --args ruby -I lib -e "require 'openssl'; OpenSSL::PKey.read(File.read('key.pem'))"
(gdb) set environment LD_LIBRARY_PATH /home/jaruga/.local/openssl-3.0.8-fips/lib/

Some of the content might repeat what jarkorp said.
The error is happening on the code below, because the pkey is NULL.

openssl/ext/openssl/ossl_pkey.c

Lines 215 to 218 in a14055a

	pkey = ossl_pkey_read_generic(bio, ossl_pem_passwd_value(pass));
	BIO_free(bio);
	if (!pkey)
	ossl_raise(ePKeyError, "Could not parse PKey");

Here is the backtrace.

(gdb) f
#0  ossl_pkey_new_from_data (argc=<optimized out>, argv=<optimized out>, 
    self=<optimized out>) at ../../../../ext/openssl/ossl_pkey.c:218
218		ossl_raise(ePKeyError, "Could not parse PKey");
(gdb) bt
#0  ossl_pkey_new_from_data (argc=<optimized out>, argv=<optimized out>, 
    self=<optimized out>) at ../../../../ext/openssl/ossl_pkey.c:218
#1  0x00007ffff7b309f7 in vm_call_cfunc_with_frame (ec=0x40a0c0, reg_cfp=0x7ffff7542f90, 
    calling=<optimized out>) at /home/jaruga/src/ruby-3.2.1/vm_insnhelper.c:3268
#2  0x00007ffff7b35d44 in vm_sendish (method_explorer=<optimized out>, 
    block_handler=<optimized out>, cd=<optimized out>, reg_cfp=<optimized out>, 
    ec=<optimized out>) at /home/jaruga/src/ruby-3.2.1/vm_callinfo.h:367
#3  vm_exec_core (ec=0x7, initial=4214800, initial@entry=0)
    at /home/jaruga/src/ruby-3.2.1/insns.def:820
#4  0x00007ffff7b3bdf9 in rb_vm_exec (ec=0x40a0c0, jit_enable_p=jit_enable_p@entry=true)
    at vm.c:2383
#5  0x00007ffff7b3cde8 in rb_iseq_eval_main (iseq=<optimized out>) at vm.c:2633
#6  0x00007ffff7951755 in rb_ec_exec_node (ec=ec@entry=0x40a0c0, n=n@entry=0x7ffff7e7bab8)
    at eval.c:289
#7  0x00007ffff7957c7b in ruby_run_node (n=0x7ffff7e7bab8) at eval.c:330
#8  0x0000000000401102 in rb_main (argv=0x7fffffffda58, argc=5) at ./main.c:38
#9  main (argc=<optimized out>, argv=<optimized out>) at ./main.c:57
(gdb) bt
#0  ossl_pkey_new_from_data (argc=<optimized out>, argv=<optimized out>, self=<optimized out>) at ../../../../ext/openssl/ossl_pkey.c:218
#1  0x00007ffff7b309f7 in vm_call_cfunc_with_frame (ec=0x40a0c0, reg_cfp=0x7ffff7542f90, calling=<optimized out>) at /home/jaruga/src/ruby-3.2.1/vm_insnhelper.c:3268
#2  0x00007ffff7b35d44 in vm_sendish (method_explorer=<optimized out>, block_handler=<optimized out>, cd=<optimized out>, reg_cfp=<optimized out>, ec=<optimized out>)
    at /home/jaruga/src/ruby-3.2.1/vm_callinfo.h:367
#3  vm_exec_core (ec=0x7, initial=4214800, initial@entry=0) at /home/jaruga/src/ruby-3.2.1/insns.def:820
#4  0x00007ffff7b3bdf9 in rb_vm_exec (ec=0x40a0c0, jit_enable_p=jit_enable_p@entry=true) at vm.c:2383
#5  0x00007ffff7b3cde8 in rb_iseq_eval_main (iseq=<optimized out>) at vm.c:2633
#6  0x00007ffff7951755 in rb_ec_exec_node (ec=ec@entry=0x40a0c0, n=n@entry=0x7ffff7e7bab8) at eval.c:289
#7  0x00007ffff7957c7b in ruby_run_node (n=0x7ffff7e7bab8) at eval.c:330
#8  0x0000000000401102 in rb_main (argv=0x7fffffffda58, argc=5) at ./main.c:38
#9  main (argc=<optimized out>, argv=<optimized out>) at ./main.c:57

In the process, the ossl_pkey_read_generic is called from the ext/openssl/ossl_pkey_dh.c ossl_dh_initialize, and also called from ext/openssl/ossl_pkey.c ossl_pkey_new_from_data.

When the ossl_pkey_read_generic called from the ossl_dh_initialize, the OSSL_DECODER_from_bio(dctx, bio) returns 1, and set the value to the poiner pkey in this timing. But the ossl_pkey_read_generic called from the ossl_pkey_new_from_data, OSSL_DECODER_from_bio(dctx, bio) doesn't return 1, and the variable pkey is still NULL.

openssl/ext/openssl/ossl_pkey.c

Lines 144 to 145 in a14055a

	if (OSSL_DECODER_from_bio(dctx, bio) == 1)
	goto out;

[junaruga]

I opened the issue ticket openssl/openssl#20657 on the OpenSSL project to ask a queston.

[junaruga]

In the issue ticket on the openssl/openssl above, the root cause was found. In short summary, the OSSL_DECODER_CTX_set_selection has a bug in the FIPS mode case. And I confirmed this issue was fixed with my patch on openssl/openssl#20657 (comment). As a note, I am working for that on my forked repository's branch https://github.com/junaruga/openssl/tree/wip/fips-read.

By applying this patch, the unit test result in the FIPS mode changes in a better way from

https://github.com/junaruga/openssl/actions/runs/4681990779/jobs/8295272065

513 tests, 1963 assertions, 14 failures, 331 errors, 0 pendings, 1 omissions, 0 notifications
32.6172% passed

to

https://github.com/junaruga/openssl/actions/runs/4690199157/jobs/8312977212

513 tests, 3082 assertions, 17 failures, 135 errors, 0 pendings, 1 omissions, 0 notifications
70.3125% passed

.

I am still working to fix other issues on the FIPS mode. Seeing the code below in the test, it seems that the FIPS mode case was not tested before. And I would like to remove this logic to run the unit tests in the FIPS mode.

openssl/test/openssl/utils.rb

Lines 5 to 8 in 6182ac0

	# Disable FIPS mode for tests for installations
	# where FIPS mode would be enabled by default.
	# Has no effect on all other installations.
	OpenSSL.fips_mode=false