WIP: implement case switching as key action

[fischerling]

Implement case switching internally as key action.
This does no longer rely on piping to external tools and breaking new lines.

This is an quick draft. I have not tested it thoroughly.
Especially since I am no encoding expert, I am not sure if the assertion that case switched wchars need the same amount of bytes holds.

Fixes #1320, #1238.
Supersedes #1381, #1382.

[rnpnr]

Thanks for this! I will have to test with the previous non-ASCII examples a little later.

This is an quick draft. I have not tested it thoroughly. Especially since I am no encoding expert, I am not sure if the assertion that case switched wchars need the same amount of bytes holds.

I'm not sure either but I suspect its fine. I would just leave it as you wrote until someone reports otherwise. Here are a few other points:

• I do know that the number of wide characters will be strictly less than the number of multibyte characters. Therefore I would prefer if you just did a single allocation for both the wide character buffer and the text buffer with that assumption (i.e. allocate N wide characters for a selection containing N bytes). To avoid an alignment issue put the wide char buffer at the start of the allocation. The NUL termination is not needed in either case, we know both lengths. You don't need to separately measure the wide char string the conversion result will give you the resulting count of wide characters.
• You forgot to free the wide character buffer in the normal case (moot if you do the above suggestion).
• No need to abort for all selections just because one selection contains an illegal sequence.

[fischerling]

I'm not sure either but I suspect its fine. I would just leave it as you wrote until someone reports otherwise. Here are a few other points:

• I do know that the number of wide characters will be strictly less than the number of multibyte characters. Therefore I would prefer if you just did a single allocation for both the wide character buffer and the text buffer with that assumption (i.e. allocate N wide characters for a selection containing N bytes). To avoid an alignment issue put the wide char buffer at the start of the allocation. The NUL termination is not needed in either case, we know both lengths. You don't need to separately measure the wide char string the conversion result will give you the resulting count of wide characters.

I do not understand what you mean. How can we reuse the multibyte buffer for our wide-character string?

The wide-character string may need more bytes.
Example: The multibyte string: Grüße! needs 8 bytes and has 6 wide-characters. However a wide-character string with 6 characters needs 4 * 6 = 24 bytes.
I see no reliable way storing both in the same 24 byte object.

We could skip the initial counting, if we always allocate N wide-characters for a N byte selection.
Not sure if this is a worthy optimization/tradeoff.

• [...] You don't need to separately measure the wide char string the conversion result will give you the resulting count of wide characters.

Measuring the actual amount of bytes required for the modified wide-character string is indeed unnecessary if your assumption (#bytes == #modified bytes) holds.

• You forgot to free the wide character buffer in the normal case (moot if you do the above suggestion).
  Thanks!

• No need to abort for all selections just because one selection contains an illegal sequence.
  Ok then I will skip the selection.

[rnpnr]

I do not understand what you mean. How can we reuse the multibyte buffer for our wide-character string?

The wide-character string may need more bytes. Example: The multibyte string: Grüße! needs 8 bytes and has 6 wide-characters. However a wide-character string with 6 characters needs 4 * 6 = 24 bytes. I see no reliable way storing both in the same 24 byte object

Currently you do an allocation in text_bytes_alloc0() and another manually with calloc(). I'm just saying add the two sizes together and do a single allocation. Do the math with the sizes you already have to get the pointers then use text_bytes_get() to read the data prior to converting it.

Even if you want to ignore the performance implications, doing what I'm suggesting is less error prone because you only need to free a single allocation.

[fischerling]

I do not understand what you mean. How can we reuse the multibyte buffer for our wide-character string?
The wide-character string may need more bytes. Example: The multibyte string: Grüße! needs 8 bytes and has 6 wide-characters. However a wide-character string with 6 characters needs 4 * 6 = 24 bytes. I see no reliable way storing both in the same 24 byte object

Currently you do an allocation in text_bytes_alloc0() and another manually with calloc(). I'm just saying add the two sizes together and do a single allocation. Do the math with the sizes you already have to get the pointers then use text_bytes_get() to read the data prior to converting it.

Even if you want to ignore the performance implications, doing what I'm suggesting is less error prone because you only need to free a single allocation.

A now I get it :)
Using a single big enough allocation for both disjunct memory regions seam reasonable.

[rnpnr]

I have tested with some non-ASCII and everything seemed fine to me (I can't read the languages I was testing). This is good to go on my end!