A community in which webmasters can ask for help with topics such as PHP coding , MySQL , IT jobs, web design, IT security.
Current location:homephp forumphp talk in 2008 yearHow to get code point number for a given character in a utf-8 string? - page 1
User InfoPosts
How to get code point number for a given character in a utf-8 string?#1
I want to get the UCS-2 code points for a given UTF-8 string. For example the word "hello" should become something like "0068 0065 006C 006C 006F". Please note that the characters could be from any language including complex scripts like the east asian languages.

So, the problem comes down to "convert a given character to its UCS-2 code point"

But how? Please, any kind of help will be very very much appreciated since I am in a great hurry.

Thanks in advance



Transcription of questioner's response posted as an answer

Thanks for your reply, but it needs to be done in PHP v 4 or 5 but not 6.

The string will be a user input, from a form field.

I want to implement a PHP version of utf8to16 or utf8decode like

function get_ucs2_codepoint($char)
{
// calculation of ucs2 codepoint value and assign it to $hex_codepoint
return $hex_codepoint;
}


Can you help me with PHP or can it be done with PHP with version mentioned above?

Thank you again.

posted date: 2008-12-27 20:38:00


Re: How to get code point number for a given character in a utf-8 string?#2
I had made out the solution of this problem. click to view my topic...

hope that hepls.

posted date: 2008-12-27 20:38:01


Re: How to get code point number for a given character in a utf-8 string?#3
Which programming language?

posted date: 2008-12-27 20:57:00


Re: How to get code point number for a given character in a utf-8 string?#4
Use an existing utility such as iconv, or whatever libraries come with the language you're using.If you insist on rolling your own solution, read up on the UTF-8 format. Basically, each code point is stored as 1-4 bytes, depending on the value of the code point. The ranges are as follows:U+0000 — U+007F: 1 byte: 0xxxxxxxU+0080 — U+07FF: 2 bytes: 110xxxxx 10xxxxxxU+0800 — U+FFFF: 3 bytes: 1110xxxx 10xxxxxx 10xxxxxxU+10000 — U+10FFFF: 4 bytes: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxxWhere each x is a data bit. Thus, you can tell how many bytes compose each code point by looking at the first byte: if it begins with a 0, it's a 1-byte character. If it begins with 110, it's a 2-byte character. If it begins with 1110, it's a 3-byte character. If it begins with 11110, it's a 4-byte character. If it begins with 10, it's a non-initial byte of a multibyte character. If it begins with 11111, it's an invalid character.Once you figure out how many bytes are in the character, it's just a matter if bit twiddling. Also note that UCS-2 cannot represent characters above U+FFFF.Since you didn't specify a language, here's some sample C code (error checking omitted):wchar_t utf8_char_to_ucs2(const unsigned char *utf8){ if(!(utf8[0] & 0x80)) // 0xxxxxxx return (wchar_t)utf8[0]; else if((utf8[0] & 0xE0) == 0xC0) // 110xxxxx return (wchar_t)(((utf8[0] & 0x1F) << 6) | (utf8[1] & 0x3F)); else if((utf8[0] & 0xF0) == 0xE0) // 1110xxxx return (wchar_t)(((utf8[0] & 0x0F) << 12) | ((utf8[1] & 0x3F) << 6) | (utf8[2] & 0x3F)); else return ERROR; // uh-oh, UCS-2 can't handle code points this high}

posted date: 2008-12-27 21:04:00


Re: How to get code point number for a given character in a utf-8 string?#5
I'm amused because I just gave this problem to students on a final exam. Here's a sketch of UTF-8:hex binary UTF-8 binary0000-007F 00000000 0abcdefg => 0abcdefg0080-07FF 00000abc defghijk => 110abcde 10fghijk0800-FFFF abcdefgh ijklmnop => 1110abcd 10efghij 10klmnopAnd here's some C99 code:static void check(char c) { if ((c & 0xc0) != 0xc0) RAISE(Bad_UTF8);}uint16_t Utf8_decode(char **p) { // return code point and advance *p char *s = *p; if ((s[0] & 0x80) == 0) { (*p)++; return s[0]; } else if ((s[0] & 0x40) == 0) { RAISE (Bad_UTF8); return ~0; // prevent compiler warning } else if ((s[0] & 0x20) == 0) { if ((s[0] & 0xf0) != 0xe0) RAISE (Bad_UTF8); check(s[1]); check(s[2]); (*p) += 3; return ((s[0] & 0x0f) << 12) + ((s[1] & 0x3f) << 6) + ((s[2] & 0x3f)); } else { check(s[1]); (*p) += 2; return ((s[0] & 0x1f) << 6) + ((s[1] & 0x3f)); }}

posted date: 2008-12-27 22:18:00


Re: How to get code point number for a given character in a utf-8 string?#6
Scott Reynen wrote a function to convert UTF-8 into Unicode. I found it looking at the PHP documentation.function utf8_to_unicode( $str ) { $unicode = array(); $values = array(); $lookingFor = 1; for ($i = 0; $i < strlen( $str ); $i++ ) { $thisValue = ord( $str[ $i ] ); if ( $thisValue < ord('A') ) { // exclude 0-9 if ($thisValue >= ord('0') && $thisValue <= ord('9')) { // number $unicode[] = chr($thisValue); } else { $unicode[] = '%'.dechex($thisValue); } } else { if ( $thisValue < 128) $unicode[] = $str[ $i ]; else { if ( count( $values ) == 0 ) $lookingFor = ( $thisValue < 224 ) ? 2 : 3; $values[] = $thisValue; if ( count( $values ) == $lookingFor ) { $number = ( $lookingFor == 3 ) ? ( ( $values[0] % 16 ) * 4096 ) + ( ( $values[1] % 64 ) * 64 ) + ( $values[2] % 64 ): ( ( $values[0] % 32 ) * 64 ) + ( $values[1] % 64 ); $number = dechex($number); $unicode[] = (strlen($number)==3)?"%u0".$number:"%u".$number; $values = array(); $lookingFor = 1; } // if } // if } } // for return implode("",$unicode);} // utf8_to_unicode

posted date: 2008-12-27 22:59:00


Re: How to get code point number for a given character in a utf-8 string?#7
Whoops, sorry for wasting your time with C code. But I hope you find the little diagram helpful.

posted date: 2008-12-27 23:15:00


Re: How to get code point number for a given character in a utf-8 string?#8
Isn't the check() function broken? Shouldn't it be testing: if ((c & 0xC0) != 0x80)? Also, what's the RAISE macro?

posted date: 2008-12-28 00:02:00


Re: How to get code point number for a given character in a utf-8 string?#9
I wrote that function. The original is here, along with some other functions you might find useful: randomchaos.com/documents/?source=php_and_unicode

posted date: 2008-12-28 08:47:00


Re: How to get code point number for a given character in a utf-8 string?#10
@Reynen, Small world, eh? I've edited my post to give you credit and some advertisement. ;D

posted date: 2008-12-28 12:56:00


Re: How to get code point number for a given character in a utf-8 string?#11
Thanks for bug fix in check() function. Serves me right for writing code the night before a final. RAISE is from cs.princeton.edu/software/cii.

posted date: 2008-12-28 18:03:00


Re: How to get code point number for a given character in a utf-8 string?#12
PHP code (which assumes valid utf-8, no check for non-valid utf-8):function ord_utf8($c) { $b0 = ord($c[0]); if ( $b0 < 0x10 ) { return $b0; } $b1 = ord($c[1]); if ( $b0 < 0xE0 ) { return (($b0 & 0x1F) << 6) + ($b1 & 0x3F); } return (($b0 & 0x0F) << 12) + (($b1 & 0x3F) << 6) + (ord($c[2]) & 0x3F); }

posted date: 2012-07-29 03:28:00


select page: « 1 2 »
Copyright ©2008-2017 www.momige.com, all rights reserved.