eyaml decrypt adds newline

[bifek]

% echo -n nonl > nonl; od -c < nonl
0000000   n   o   n   l
% eyaml encrypt -f nonl -o string > nonl.eyaml
% eyaml decrypt -f nonl | od -c
0000000   n   o   n   l  \n

Pretty sure that "eyaml decrypt" is wrongly emitting newline.

if we take base64-encoded encrypted value and decode it with openssl, we get correct result:

% cut -c11- nonl.eyaml | base64 -d > nonl.pkcs7.der
% openssl smime -decrypt -inkey keys/private_key.pkcs7.pem \
    -signer keys/public_key.pkcs7.pem -inform der -in nonl.pkcs7.der  | od -c
0000000   n   o   n   l

no newline, as expected.

Even more clear example:

% { echo -n foo; eyaml encrypt -f nonl -o string | tr -d '\n';  echo -n bar; } |
    eyaml decrypt --stdin  | od -c
0000000   f   o   o   n   o   n   l   b   a   r  \n
0000013

so it decrypts properly w/o \n, it just adds \n when emitting, even if input does not contain ending \n
unfortunately, it does that also when output is not terminal, ie. to the file, which breaks binary files.

[danielparks]

Here's an even simpler case:

❯ echo -n nonl | od -c
0000000    n   o   n   l                                                
0000004
❯ echo -n nonl | eyaml decrypt --stdin | od -c
0000000    n   o   n   l  \n                                            
0000005

[jms1voalte]

For what it's worth, the Puppet lookup() function appears to be removing the final newlines from encrypted values.

It kinda feels like somebody noticed extra newlines being added at some point in the past, and "fixed" it in the wrong place?

# eyaml encrypt -o string -l xyzzy -f server.key >> /etc/hieradata/common.yaml
#
# od -c server.key | tail -3
0003160   -   -   E   N   D       R   S   A       P   R   I   V   A   T
0003200   E       K   E   Y   -   -   -   -   -  \n
0003213
# grep '^xyzzy:' /etc/hieradata/common.yaml | sed '{s/^.*PKCS7,//;s/]$//;}' | base64 -d | openssl smime -decrypt -inkey /etc/puppetlabs/keys/private_key.pkcs7.pem -signer /etc/puppetlabs/keys/public_key.pkcs7.pem -inform der | od -c | tail -3
0003160   -   -   E   N   D       R   S   A       P   R   I   V   A   T
0003200   E       K   E   Y   -   -   -   -   -  \n
0003213
# puppet apply --color=false -e '$xyzzy=lookup("xyzzy");notice("<[${xyzzy}]>")' | grep '^-----END' | od -c
0000000   -   -   -   -   -   E   N   D       R   S   A       P   R   I
0000020   V   A   T   E       K   E   Y   -   -   -   -   -   ]   >  \n
0000040

The original file ends with a newline, and the encrypted value ends with a newline, but the string returned by the lookup() function does NOT end with a newline.

Other examples I found while testing this:

Normal string WITHOUT trailing newline. This looks more or less like what I was expecting, with no newline between testing and ]>.

# echo -n testing | eyaml encrypt -o string -l xyzzy2 --stdin >> /etc/hieradata/common.yaml
#
# echo -n testing | od -c
0000000   t   e   s   t   i   n   g
0000007
# grep '^xyzzy2:' /etc/hieradata/common.yaml | sed '{s/^.*PKCS7,//;s/]$//;}' | base64 -d | openssl smime -decrypt -inkey /etc/puppetlabs/keys/private_key.pkcs7.pem -signer /etc/puppetlabs/keys/public_key.pkcs7.pem -inform der | od -c | tail -3
0000000   t   e   s   t   i   n   g
0000007
# puppet apply --color=false -e '$xyzzy=lookup("xyzzy2");notice("<[${xyzzy}]>")' | grep ']>'
Notice: Scope(Class[main]): <[testing]>

Normal string WITH trailing newline. Here I was expecting to see a newline between testing and ]>, because the last character of the encrypted value is a newline.

# echo testing | eyaml encrypt -o string -l xyzzy3 --stdin >> /etc/hieradata/common.yaml
#
# echo testing | od -c
0000000   t   e   s   t   i   n   g  \n
0000010
# grep '^xyzzy3:' /etc/hieradata/common.yaml | sed '{s/^.*PKCS7,//;s/]$//;}' | base64 -d | openssl smime -decrypt -inkey /etc/puppetlabs/keys/private_key.pkcs7.pem -signer /etc/puppetlabs/keys/public_key.pkcs7.pem -inform der | od -c | tail -3
0000000   t   e   s   t   i   n   g  \n
0000010
# puppet apply --color=false -e '$xyzzy=lookup("xyzzy3");notice("<[${xyzzy}]>")' | grep ']>'
Notice: Scope(Class[main]): <[testing]>

[AndrewLipscomb]

Necroing this as at least on my installation this still happens and has caused some hair pulling this morning - but this seems like the last place this is discussed by my googling.

The somewhat terrible but functional way I've been dealing with the inconsistency (and the fact that SSH private keys have to have that newline) is to force eyaml files to use the YAML block format

my_sensitive_ssh_private_key_data: ENC[PKCS7,....]

becomes

my_sensitive_ssh_private_key_data: >
  ENC[PKCS7,....]

I can't say I've dug into the rules for how newlines come across from Puppet lookup - but I'm guessing its stripped the trailing newline by some default rules way-way back in the parser. This block format forces that newline to appear - regardless of the content type (ie: encrypted Sensitive strings or just regular old strings)