Trailing newline is removed during encryption/decryption

[danielparks]

Hiera eyaml 2.1.0 removes the trailing new line either during encryption or decryption.

❯ eyaml createkeys
❯ eyaml encrypt -s "test" -o string | tr -d '\n' | eyaml decrypt --stdin ; echo EOF
test
EOF
❯ echo -n test | eyaml encrypt --stdin -o string | tr -d '\n' | eyaml decrypt --stdin ; echo EOF
test
EOF
❯ echo test | eyaml encrypt --stdin -o string | tr -d '\n' | eyaml decrypt --stdin ; echo EOF 
test
EOF

Either the first two examples should output "testEOF", or the final one should be:

❯ echo test | eyaml encrypt --stdin -o string | tr -d '\n' | eyaml decrypt --stdin ; echo EOF 
test

EOF

The first two examples should encrypt the string "test", and the final example should encrypt the string "test\n".

[rnelson0]

@danielparks The issue you are observing is that eyaml decrypt parses the ENC[...] portion only, and the shell tries to be nice by attaching a newline to the end of the output. The trim command in the middle changes ENC[...]\n to ENC[...] but that's a step too early. See this difference in output when the newline is stripped from the output of eyaml with echo -n or `printf '%s':

[rn7284@chi-build06 ~]$ eyaml encrypt -s "test" -o string | echo $(eyaml decrypt --stdin)
test
[rn7284@chi-build06 ~]$

[rn7284@chi-build06 ~]$ eyaml encrypt -s "test" -o string | echo -n $(eyaml decrypt --stdin)
test[rn7284@chi-build06 ~]$

[rn7284@chi-build06 ~]$ eyaml encrypt -s "test" -o string | printf '%s' $(eyaml decrypt --stdin)
test[rn7284@chi-build06 ~]$

You can also move tr -d '\n' to the end, or numerous other methods, but if you're piping this into another system as I imagine, it may not be a problem if you avoid the shell entirely.

[danielparks]

I'm sorry, my example is confusing. The issue is that it drops the trailing newline:

❯ echo test | eyaml encrypt --stdin -o string | tr -d '\n' | echo -n $(eyaml decrypt --stdin) ; echo EOF 
testEOF

It should be encrypting "test\n".

[rnelson0]

I think that's still a shell issue. If I create a file containing that string and a newline, it behaves as you would expect:

[rn7284@chi-build06 ~]$ cat test
test
[rn7284@chi-build06 ~]$ cat test | eyaml encrypt --stdin -o string | eyaml decrypt --stdin
[hiera-eyaml-core] Loaded config from /etc/eyaml/config.yaml
[hiera-eyaml-core] Loaded config from /etc/eyaml/config.yaml
test

[rn7284@chi-build06 ~]$

[danielparks]

eyaml encrypt --stdin -o string appends a newline to its output; that newline is passed through directly. That's why there is a tr -d '\n' in my example.

[rnelson0]

That is the output on the terminal. The encrypted string remains test\n. I performed encryption, output it to a file, and the encrypted content includes a newline - it's preserved during encryption, and when decrypted by eyaml edit.

[rn7284@chi-build06 ~]$ echo test | eyaml encrypt --stdin -o string > test
[hiera-eyaml-core] Loaded config from /etc/eyaml/config.yaml
[rn7284@chi-build06 ~]$ eyaml edit test
[hiera-eyaml-core] Loaded config from /etc/eyaml/config.yaml
#| This is eyaml edit mode. This text (lines starting with #| at the top of the
#| file) will be removed when you save and exit.
#|  - To edit encrypted values, change the content of the DEC(<num>)::PKCS7[]!
#|    block.
#|    WARNING: DO NOT change the number in the parentheses.
#|  - To add a new encrypted value copy and paste a new block from the
#|    appropriate example below. Note that:
#|     * the text to encrypt goes in the square brackets
#|     * ensure you include the exclamation mark when you copy and paste
#|     * you must not include a number when adding a new block
#|    e.g. DEC::PKCS7[]!
DEC(0)::PKCS7[test
]!

[danielparks]

Then the issue is decryption — the encrypted string is "test\n", but the decrypted string is "test".

FWIW, this originally came up because it's an issue in Puppet code. I have bunch of encrypted files, all with trailing newlines, and those newlines are lost when they're passed into my Puppet class. The files were encrypted with -f.

[danielparks]

Can this be reopened? There's a real issue here. Alternatively I can write up a clearer description and submit a new issue — as long as you actually want it.

I looked at the code and couldn't figure out where the issue was arising.

[rnelson0]

How about a new issue with a clearer description? I have no idea where it's happening either, but with a different description and/or some examples, maybe we can find it.

[bifek]

% echo -n nonl > nonl; od -c < nonl
0000000   n   o   n   l
% eyaml encrypt -f nonl -o string | eyaml decrypt --stdin | od -c
0000000   n   o   n   l  \n

I would not expect a newline here, would you?

Pretty sure that "eyaml decrypt" is wrongly emitting newline. Does this example help?

[danielparks]

Thanks @bifek!

One little thing: eyaml passes everything through that isn't part of the ENC[...] block. That includes the trailing newline that eyaml encrypt adds. (Its output is ENC[...]\n.) So in your example you would expect a trailing new line.

However, stripping the newline still shows the problem:

% echo -n nonl > nonl; od -c < nonl
0000000 n o n l
% eyaml encrypt -f nonl -o string | tr -d '\n' | eyaml decrypt --stdin | od -c
0000000 n o n l \n

(You can examine the output of eyaml encrypt -f nonl -o string to see the trailing newline.)

[danielparks]

Filed #272 as requested.

[bifek]

I actually already opened #270 to track this as rnelson suggested.

The \n after ENC[] doesn't matter since all yaml properties have that and getting a non-encrypted value from eyaml does not bring you extra \n. But you proved it yourself by stripping this \n before decrypt, and the result was the same.

Even more fun example:

% { echo -n foo; eyaml encrypt -f nonl -o string | tr -d '\n';  echo -n bar; } | eyaml decrypt --stdin  | od -c
0000000   f   o   o   n   o   n   l   b   a   r  \n
0000013

so it decrypts properly w/o \n, it just adds \n when emitting, even if input does not contain ending \n

[danielparks]

Hm. It looks like the \n does matter:

❯ enc=$(eyaml encrypt -s nonl -o string)
❯ printf "%s" "$enc" ; echo %
ENC[PKCS7,...]%
❯ printf "%s" "$enc" | eyaml decrypt --stdin ; echo %
nonl
%
❯ printf "one\n%s\nthree\n" "$enc" ; echo %
one
ENC[PKCS7,...]
three
%
❯ printf "one\n%s\nthree\n" "$enc" | eyaml decrypt --stdin ; echo %
one
nonl
three
%