better char/string handling functions

[trillllian]

char() returns utf-8 bytes of a string. even setting aside the bytes vs codepoints thing, this is different from character literals in that it ignores the current table mapping. char("X",0) being different from 'X' feels really bad to me.

dunno what we should do. i guess one relatively simple solution is making char() return the Nth unicode codepoint of the string, looked up through the table mapping. this does make it quadratic time to iterate through a string, but asar has some stupid O(n^2) places already, we can optimize it if it actually becomes a problem (probably via python-style "upgrade string to utf32 if necessary").

i was going to insert some rambling here about how this could lead to surprises if the table mapping is not injective and some string processing code tries to compare characters for equality, but this is probably rare enough to ignore and i can't think of any good solution for that case anyways.

[p4plus2]

To me reading bytes makes it easier to write custom parsers. Maybe not everybody's optimal use case.

I think two function might be a decent idea, thoughts?

[trillllian]

I think two function might be a decent idea, thoughts?

and then we need to add a second syntax for character literals too? or do you want to write char("x",0) everywhere?

[trillllian]

selicre wants a function to make a string from a char.

<trillian> yeah i think we might need to add a character type
<trillian> that gets autoconverted to int via table lookup when doing math on it, but is still a separate type for equality comparisons and maybe a separate "rawchar()" function or something that returns the codepoint without table lookup
<selicre> as long as char("str",0) == 's' I'm fine with any solution

then char() can return the character type, and the char->string function can take a character type as input. maybe also add a function for integer unicode codepoint -> char?

[trillllian]

selicre's suggestion:

Introduce a new primitive type, char, which is equivalent to one unicode codepoint ranging from 0x0 to 0x10FFFF [editor's note: rust's char type excludes utf16 surrogates, we should do the same i think]. Casts to number in operands and d[bwld], remains utf-8 in print. Retains type in functions.

Functions

• char(s: string, i: number) -> char: obtains the Nth unicode codepoint.
• strslice(s: string, from: number, to: number) -> string: returns a substring between the from and to codepoint indices.
  • [ed: start inclusive, end exclusive i assume?]
• strfind(s: string, needle: string) -> number: returns the codepoint index that matches needle.
  • [ed: in case of no match, throw an error or return -1?]
• strlen(s: string): stringlength() but with a better name
• strcat(l: string|char, ...) -> string: concatenates variable amounts of strings or chars
  • [ed: this can be done by +, but that's O(n²) with n arguments...]
• charvalue(c: char) -> number: table value of c
• charidx(c: char) -> number: unicode codepoint index of c
• charfromidx(n: number) -> char: convert unicode codepoint index to char

[ed: not sure i like the unicode codepoint functions being called "idx", but idk what else to call them. "charunicode" is a little long]