Implement language dependent casing support for std.uni

[rikkimax]

@LightBender @atilaneves

Rationale

Casing support needs to handle the language specific rules of SpecialCasing.txt, this hurts our Turkish users not to have.

Feature request or issue tracking

I didn't spot a bug report, but this is a known issue that has been bugging me for ages.

Pre-review checklist

• I have performed a self-review of my code.
• If my PR fixes a bug or introduces a new feature, I have added thorough tests.
• If my changes are non-trivial and do not concern a reported issue, I have added a changelog entry.

LLM/AI disclosure

This PR was written entirely by an LLM/AI.

MiMo V2.5

Casing support in std.uni is incomplete.
It only includes simple casing, not Special casing.

I have given you an updated unicode database file, for the table generator.

You will need to figure out how to build std.uni along with the tables to verify the implementation.

Craft a plan that is based upon the current state of casing support of std.uni and the table generator. You have the relevant specification.

Casing locality is a runtime variable.
Do not modify existing casing support, instead add new functions and tables for it.
I have given you access to another casing implementation in @documents/basic_memory

There are differences between ProjectSidero's tables approach and std.uni's. You should do it the way std.uni does it, since that is the target library.

Ignore build_v3.d it has nothing to do with PhobosV2 that you are working on.
Also change of plans, we'll go back to ucd-17, you don't need to update the directory.

Go

CasingLanguage needs to be public, with each entry in the enum to have a ddoc comment on it.
Move specialIsCaseIgnorable, specialIsCased, specialIsSoftDotted, specialGetCCC into the same block as specialTitleIndex_AzeriTrie and remove the private as it already is private.
Move toLowerSpecial, toUpperSpecial to a place above the private: area so that it is public along with their unittest. Include in their ddoc's why the special version exists.

SpecialCasingCondition and SpecialCasing should not be public, put them back behind private.

std/uni/package.d(10896): Error: none of the overloads of template std.utf.encode are callable using argument types !()(Appender!(char[]), dchar)
encode(result, mc);
std/utf.d(2324): Candidates are: encode(Flag useReplacementDchar = No.useReplacementDchar)(out char[4] buf, dchar c)
std/utf.d(2458): encode(Flag useReplacementDchar = No.useReplacementDchar)(out wchar[2] buf, dchar c)
std/utf.d(2511): encode(Flag useReplacementDchar = No.useReplacementDchar)(out dchar[1] buf, dchar c)
std/utf.d(2554): encode(Flag useReplacementDchar = No.useReplacementDchar)(ref scope char[] str, dchar c)
std/utf.d(2676): encode(Flag useReplacementDchar = No.useReplacementDchar)(ref scope wchar[] str, dchar c)
std/utf.d(2733): encode(Flag useReplacementDchar = No.useReplacementDchar)(ref scope dchar[] str, dchar c)

std/utf.d(2324): Candidates are: encode(Flag useReplacementDchar = No.useReplacementDchar)(out char[4] buf, dchar c)
std/utf.d(2458): encode(Flag useReplacementDchar = No.useReplacementDchar)(out wchar[2] buf, dchar c)
std/utf.d(2511): encode(Flag useReplacementDchar = No.useReplacementDchar)(out dchar[1] buf, dchar c)
std/utf.d(2554): encode(Flag useReplacementDchar = No.useReplacementDchar)(ref scope char[] str, dchar c)
std/utf.d(2676): encode(Flag useReplacementDchar = No.useReplacementDchar)(ref scope wchar[] str, dchar c)
std/utf.d(2733): encode(Flag useReplacementDchar = No.useReplacementDchar)(ref scope dchar[] str, dchar c)

Build std.uni to verify there are not more compiler errors.

std/uni/package.d(10834): Error: function toLowerTab is not callable using argument types ()
auto val = toLowerTab[idx];
std/uni/package.d(10834): too few arguments, expected 1, got 0
auto val = toLowerTab[idx];
std/uni/package.d(9154): std.uni.toLowerTab(ulong idx) declared here
dchar toLowerTab(size_t idx) { return toLowerTable[idx]; }
std/uni/package.d(11221): Error: function tryDir is not callable using argument types (immutable(Trie!(ushort, dchar, 1114112LU, sliceBits!(13LU, 21LU), sliceBits!(6LU, 13LU), sliceBits!(0LU, 6LU))), immutable(uint)[], immutable(ubyte)[])
tryDir(specialLowerIndex_LithuanianTrie(), specialLowerTable_Lithuanian(), specialLowerCond_Lithuanian());

Parameters cannot be typed as auto. It sounds like you wanted a template.
For tryDir.

std/uni/package.d(10849): Error: return value result.data() of type char[] does not match return type string, and cannot be implicitly converted
return result.data;
std/uni/package.d(10886): Error: undefined identifier decode, did you mean template decoder(C)(C[] s, size_t offset = 0) if (is(C : wchar) || is(C : char))?
dchar c = decode(s, i);
std/uni/package.d(10919): Error: return value result.data() of type char[] does not match return type string, and cannot be implicitly converted
return result.data;

core.exception.AssertError@generated/linux/debug/64/publictests/std_uni_package.d(10929): unittest failure

That doesn't make sense.
You need to build and run the unittests of std.uni. Patch whatever module you need so that it compiles.

mappingFromTable still contains your replacement for the len == 0 return null;

core.exception.AssertError@../dmd/druntime/import/core/internal/array/casting.d(68): immutable(char)[] of length 1 cannot be cast to const(dchar)[] as its length in bytes (1) is not a multiple of const(dchar).sizeof (4).
../dmd/druntime/import/core/internal/array/casting.d:68 [0x5b9f7f1e6d60]
../dmd/druntime/import/core/internal/array/casting.d:88 [0x5b9f7f22368d]
std/uni/package.d:10822 [0x5b9f7f217feb]
generated/linux/debug/64/publictests/std_uni_package.d:10931 [0x5b9f7f1c2acf]

std/uni/package.d(10811): Error: cannot implicitly convert expression s of type string to dstring
dstring ds = s;
std/uni/package.d(10886): Error: cannot implicitly convert expression s of type string to dstring
dstring ds = s;

dchar[] ds = new dchar[s.length];
In toUpperSpecial/toLowerSpecial.
Is this predecoding to array even necessary ? In other words - have you tested performance of this vs simple casefolding?
Also observe that in common case there is maybe one or two such characters in the string decoding everything and allocating dchar[] is overkill. Let evalCondition decode (forward and backward) for the majority of real cases this is going to be minimal overhead.

Build and run the unittests without passing -I. You must provide the unicode table file.
Without the -I it shouldn't see the trait problem.

You have introduced a bunch of new utility functions:
utf8StrideBack, decodePrevUTF8, decodeNextUTF8, utf8StrideForward
Look in std.utf and see if you can replace them with existing functions.

We upper and lower special, but no title special, implement it.

[LightBender]

It's frustrating that we have to do this, but it's a well-known problem in Turkic.

[0xEAB]

On Discord the question came up why we are doing this all by ourselves instead of relying on OS-provided functionality.

[Herringway]

On Discord the question came up why we are doing this all by ourselves instead of relying on OS-provided functionality.

Does it exist on all OSes? Are you counting that locale-infested abomination that is libc?

Addressed

[DmitryOlshansky]

LGTM, do not understand how do I do review in Github mobile