Emoji Encoding

[Montmorency]

Something I'm ready to complain about: many browser (Chrome, IE, but displays on Mozilla) forms of emoji (https://unicode.org/emoji/charts/full-emoji-list.html) do not render on that page, or when pasted* in text. For example:

��
��
��
��

The middle two are not rendered anywhere for me. This is especially noticeable on Youtube, where comments look full of the (non-represented) rectangles.
What do?

EDIT: While some displayed in the Org text prompt, none of them do in the post.


*How do you type these unicodes? I can never seem to get it.

[Husar]

I'm not entirely sure about the science, but I think you just don't.
They will mostly work in personal correspondence such as messengers and email, but not on most webpages.
On webpages, you usually have to go with the set provided by the webpage or hope that the other person is using a (compatible) phone themselves. Some phone manufacturers seem to offer emojis that don't work on other manufacturers' phones and the implementation can obviously differ as well. I can't say that I ever noticed working smileys in youtube comments and I read them quite frequently.

If I'm not mistaken, that is because the browser web engines usually do not contain any emojis and the parsing of the text (that replaces the signs with the picture of the emoji) is done by the webserver itself and then sent to the browser in completed form. The browser and the operating system will then not really replace anything anymore (outside of the replacement instructions as sent by the webserver, which do not include unicode emojis). There used to be some exceptions IIRC, but that was long ago.

It's different in apps and the like, but apps like whatsapp don't use the smileys of the operating system and have their own set instead. Windows 10 also has emojis, but I doubt they will work here for said reasons. They can be found either in the on-screen keyboard or by using a special key combination (Windows key + Ü in Germany, obviously different elsewhere, I think windows key + . ).

That's my take on it, if anyone knows how this mess actually works, I'm interested as well. :sweatdrop:

����‍������ <- these actually work for me, who else can see them?....aaaaand after editing the post to add that they work, they're gone :laugh4: at least that saves my theory above :sweatdrop:

[Csargo]

From googling this is apparently an issue with fonts not having the appropriate Unicode support for the vast array of different symbols associated with them. So whatever font your phone, PC, and I'm guessing that extends to webpages as well, are incapable of encompassing all of the Unicode symbols/emoji's that exist. From what I read it's not possible to have a font that's capable of displaying every Unicode symbol, because they can't fit them all into one font, and the number of symbols continues to grow. I'm not sure how it relates to webpages and browsers. I'm guessing the webpage would supersede the browser when displaying stuff on the page, but I'm not sure if that's true. I.e. it wouldn't matter what fonts you were using in the browser if a webpage had it's own specific font it was using, so your browser would potentially be able to display the Unicode, but the webpage wouldn't. I don't quite know how it all works though, so this could be completely wrong.

[Husar]

If it's the fonts, then I wonder how Smartphones do it since they can also mix letters and symbols. Same for email or even MS Word. I just tried it and even Excel displays very basic black and white emojis when I enter them there. It's almost only on webpages/internet forums that you run into display issues. Or when two smartphones/messenger versions have incompatible sets. E.g. when a friend had a version of WhatsApp with more smileys than my version (Android vs WindowsPhone), I couldn't see those additional ones.

Obviously in an app there could be a special implementation, like here in the forums, but pretty much every modern OS implements smileys in a native way, which is why they work in Word, Excel etc. The question would be why those aren't shown in the forum here? My guess is that the OS is not parsing the text shown in the browser, so the OS implementation does not matter. They do work upon entering, which is because the code that I enter is executed on my computer (local JavaScript I guess), but once I send the post to the server, they are gone, because the server cannot handle that code.

Example: ������

[Husar]

Now the screenshots, maybe therother can explain the exact details.

22058

Remember that before I post, the text is only on my computer and after I posted, the browser displays the text that the server sends to my browser. Apparently the emojis are lost when the server creates the post for displaying in the browser. It could still be about fonts, but then there seems to be a fundamental difference in how fonts are handled on websites and in apps, etc. :shrug:

[Montmorency]

Good stuff, but as I noted - for Youtube at least - the emojis display in Firefox but not Chrome or IE.

And clearly a lot of people can see and use them, because they show up frequently enough.

Still can't figure out key commands. For example, how do I type out U+1F600? These instructions (https://support.office.com/en-us/article/insert-ascii-or-unicode-latin-based-symbols-and-characters-d13f58d3-7bcb-44a7-a4d5-972ee12e50e0) don't cut it (not that there's much practical reason to be typing out emojis):



Inserting Unicode characters
To insert a Unicode character, type the character code, press ALT, and then press X. For example, to type a dollar symbol ($), type 0024, press ALT, and then press X. For more Unicode character codes, see Unicode character code charts by script.

Important: Some of the Microsoft Office programs, such as PowerPoint and InfoPath, cannot convert Unicode codes to characters. If you need a Unicode character and are using one of the programs that doesn't support Unicode characters, use the Character Map to enter the character(s) that you need.

Notes:

If ALT+X converts the wrong character code into Unicode, select the correct character code before pressing ALT+X.

Alternatively, precede the correct character code with the text "U+". For example, typing "1U+B5" and pressing ALT+X will always return the text "1µ", while typing "1B5" and pressing ALT+X will return the text

[Husar]

Good stuff, but as I noted - for Youtube at least - the emojis display in Firefox but not Chrome or IE.

And clearly a lot of people can see and use them, because they show up frequently enough.

Still can't figure out key commands. For example, how do I type out U+1F600? These instructions (https://support.office.com/en-us/article/insert-ascii-or-unicode-latin-based-symbols-and-characters-d13f58d3-7bcb-44a7-a4d5-972ee12e50e0) don't cut it (not that there's much practical reason to be typing out emojis):

You're right, I forgot that you mentioned the browsers (shame on me). They do show up for me on Youtube, in the app and in Edge on my phone and also in my Vivaldi browser, which is based on Chromium. Edge on the desktop has them as well.
They still don't show up here on the Org though. Which makes it really strange because now it seems like in some cases it depends on the website and in others on the browser.

What is your OS? Do you use Windows 10 or an older one? Since you mentioned IE but not Edge. There could be an element of OS support as well.

[therother]

I'm afraid the more exotic characters have always been a black box for me as well.

As to why some won't work on the Org, I suspect this is due to the database using the (then) default MySQL database collation latin1 (https://dev.mysql.com/doc/refman/5.6/en/charset-charsets.html), which is a single byte character set. latin1 doesn't support as wide an array of characters that a multibyte character set such as utf8, which is I believe the current default. We moved to vBulletin 15 years ago, which was when the decision was made. IIRC, converting the database to utf8 isn't straightforward, particularly such a large and old database as the Org's.

[Montmorency]

You're right, I forgot that you mentioned the browsers (shame on me). They do show up for me on Youtube, in the app and in Edge on my phone and also in my Vivaldi browser, which is based on Chromium. Edge on the desktop has them as well.
They still don't show up here on the Org though. Which makes it really strange because now it seems like in some cases it depends on the website and in others on the browser.

What is your OS? Do you use Windows 10 or an older one? Since you mentioned IE but not Edge. There could be an element of OS support as well.

W7


I'm afraid the more exotic characters have always been a black box for me as well.

As to why some won't work on the Org, I suspect this is due to the database using the (then) default MySQL database collation latin1 (https://dev.mysql.com/doc/refman/5.6/en/charset-charsets.html), which is a single byte character set. latin1 doesn't support as wide an array of characters that a multibyte character set such as utf8, which is I believe the current default. We moved to vBulletin 15 years ago, which was when the decision was made. IIRC, converting the database to utf8 isn't straightforward, particularly such a large and old database as the Org's.

Raises the specter of the Org shutting down as maintenance becomes unmanageable with time. :shame:

[Husar]

W7

That could easily be part of your problem. Perhaps Firefox has some kind of built-in emoji support while IE and Chrome rely on the OS for that. I don't think W7 has native emoji support like Win10 does. Does not seem like the sort of thing they would patch in shortly before the end of official support. :sweatdrop:

edit with further research: https://blog.getemoji.com/post/57054354336/which-browsers-support-emoji


Windows — Windows 8 and 8.1

Internet Explorer — Yes!
Firefox — Yes!
Safari — Yes!
Chrome — No ☹
☹ If using Windows 7, Windows Vista, or any other early version of Windows, you will not see Emoji correctly in any browser.

And to finally nail the Firefox thing as well: https://www.mozilla.org/en-US/firefox/50.0/releasenotes/


Added a built-in Emoji set for operating systems without native Emoji fonts (Windows 8.0 and lower and Linux)

So it's basically that your Windows does not support emojis, but Firefox has special built-in emoji support since version 50. That's why you only see them in Firefox and nowhere else.

You could have a look at this, but not sure what exactly could go wrong or what it does: https://answers.microsoft.com/en-us/windows/forum/windows_7-start/color-emojifont-support-on-windows-7/8010df8b-c628-4d2d-8cad-24b7bec8c444

[nicholascn]

great work