Terminal shows non-ascii? characters in UTF-16 hex codes











up vote
0
down vote

favorite












todoroki@todoroki-VJZ13B ~>printf "än"
echo "ä"
ä
ä
ä
udcc3udca4: u30b3u30deu30f3u30c9u304cu898bu3064u304bu308au307eu305bu3093


according to a UTF-16 decode tool, u30b3u30deu30f3u30c9u304cu898bu3064u304bu308au307eu305bu3093 is コマンドが見つかりません (= "command not found"), which is the correct Japanese output I expect.



From the printf and echo result, UTF-8 seems working correctly.



This happens in all shell outputs, such as ls (Japanese characters in filenames shows up in UTF-16 hex format)
EDIT: ls output was not utf-16, but something called "Octal Escape Sequence" (where becomes 346234210)



ls in a directory which contains 3 folders named C, あいう, and outputs:



todoroki@todoroki-VJZ13B ~/test> ls -l
total 12
drwxr-xr-x 3 todoroki todoroki 4096 10月 4 15:02 C/
drwxr-xr-x 2 todoroki todoroki 4096 10月 11 09:04 ''$'343201202343201204343201206'/
drwxr-xr-x 2 todoroki todoroki 4096 10月 11 09:05 ''$'346234210'/


(and this is weird because of the file creation dates are shown correctly, while the directory name one isn't)



less vi nano behaves more strange; a file (a.txt, created with gedit) like below






ä


will show as



in less (it complains "a.txt" may be a binary file. See it anyway?):



<E3><81><82>
<E3><81><84>
<E3><81><86>
<C3><A4>


in vi:



�~A~B
�~A~D
�~A~F
ä


and in nano:



 ^a^b
^a^d
^a^f


I don't remember what I had done, but it was correctly showing Japanese letters at least two days ago (and for more than 6 months).



What could be the problem and the way to recover from this?










share|improve this question




















  • 2




    Just a shot in the dark - did you change any locale settings?
    – RudiC
    Oct 11 at 9:11















up vote
0
down vote

favorite












todoroki@todoroki-VJZ13B ~>printf "än"
echo "ä"
ä
ä
ä
udcc3udca4: u30b3u30deu30f3u30c9u304cu898bu3064u304bu308au307eu305bu3093


according to a UTF-16 decode tool, u30b3u30deu30f3u30c9u304cu898bu3064u304bu308au307eu305bu3093 is コマンドが見つかりません (= "command not found"), which is the correct Japanese output I expect.



From the printf and echo result, UTF-8 seems working correctly.



This happens in all shell outputs, such as ls (Japanese characters in filenames shows up in UTF-16 hex format)
EDIT: ls output was not utf-16, but something called "Octal Escape Sequence" (where becomes 346234210)



ls in a directory which contains 3 folders named C, あいう, and outputs:



todoroki@todoroki-VJZ13B ~/test> ls -l
total 12
drwxr-xr-x 3 todoroki todoroki 4096 10月 4 15:02 C/
drwxr-xr-x 2 todoroki todoroki 4096 10月 11 09:04 ''$'343201202343201204343201206'/
drwxr-xr-x 2 todoroki todoroki 4096 10月 11 09:05 ''$'346234210'/


(and this is weird because of the file creation dates are shown correctly, while the directory name one isn't)



less vi nano behaves more strange; a file (a.txt, created with gedit) like below






ä


will show as



in less (it complains "a.txt" may be a binary file. See it anyway?):



<E3><81><82>
<E3><81><84>
<E3><81><86>
<C3><A4>


in vi:



�~A~B
�~A~D
�~A~F
ä


and in nano:



 ^a^b
^a^d
^a^f


I don't remember what I had done, but it was correctly showing Japanese letters at least two days ago (and for more than 6 months).



What could be the problem and the way to recover from this?










share|improve this question




















  • 2




    Just a shot in the dark - did you change any locale settings?
    – RudiC
    Oct 11 at 9:11













up vote
0
down vote

favorite









up vote
0
down vote

favorite











todoroki@todoroki-VJZ13B ~>printf "än"
echo "ä"
ä
ä
ä
udcc3udca4: u30b3u30deu30f3u30c9u304cu898bu3064u304bu308au307eu305bu3093


according to a UTF-16 decode tool, u30b3u30deu30f3u30c9u304cu898bu3064u304bu308au307eu305bu3093 is コマンドが見つかりません (= "command not found"), which is the correct Japanese output I expect.



From the printf and echo result, UTF-8 seems working correctly.



This happens in all shell outputs, such as ls (Japanese characters in filenames shows up in UTF-16 hex format)
EDIT: ls output was not utf-16, but something called "Octal Escape Sequence" (where becomes 346234210)



ls in a directory which contains 3 folders named C, あいう, and outputs:



todoroki@todoroki-VJZ13B ~/test> ls -l
total 12
drwxr-xr-x 3 todoroki todoroki 4096 10月 4 15:02 C/
drwxr-xr-x 2 todoroki todoroki 4096 10月 11 09:04 ''$'343201202343201204343201206'/
drwxr-xr-x 2 todoroki todoroki 4096 10月 11 09:05 ''$'346234210'/


(and this is weird because of the file creation dates are shown correctly, while the directory name one isn't)



less vi nano behaves more strange; a file (a.txt, created with gedit) like below






ä


will show as



in less (it complains "a.txt" may be a binary file. See it anyway?):



<E3><81><82>
<E3><81><84>
<E3><81><86>
<C3><A4>


in vi:



�~A~B
�~A~D
�~A~F
ä


and in nano:



 ^a^b
^a^d
^a^f


I don't remember what I had done, but it was correctly showing Japanese letters at least two days ago (and for more than 6 months).



What could be the problem and the way to recover from this?










share|improve this question















todoroki@todoroki-VJZ13B ~>printf "än"
echo "ä"
ä
ä
ä
udcc3udca4: u30b3u30deu30f3u30c9u304cu898bu3064u304bu308au307eu305bu3093


according to a UTF-16 decode tool, u30b3u30deu30f3u30c9u304cu898bu3064u304bu308au307eu305bu3093 is コマンドが見つかりません (= "command not found"), which is the correct Japanese output I expect.



From the printf and echo result, UTF-8 seems working correctly.



This happens in all shell outputs, such as ls (Japanese characters in filenames shows up in UTF-16 hex format)
EDIT: ls output was not utf-16, but something called "Octal Escape Sequence" (where becomes 346234210)



ls in a directory which contains 3 folders named C, あいう, and outputs:



todoroki@todoroki-VJZ13B ~/test> ls -l
total 12
drwxr-xr-x 3 todoroki todoroki 4096 10月 4 15:02 C/
drwxr-xr-x 2 todoroki todoroki 4096 10月 11 09:04 ''$'343201202343201204343201206'/
drwxr-xr-x 2 todoroki todoroki 4096 10月 11 09:05 ''$'346234210'/


(and this is weird because of the file creation dates are shown correctly, while the directory name one isn't)



less vi nano behaves more strange; a file (a.txt, created with gedit) like below






ä


will show as



in less (it complains "a.txt" may be a binary file. See it anyway?):



<E3><81><82>
<E3><81><84>
<E3><81><86>
<C3><A4>


in vi:



�~A~B
�~A~D
�~A~F
ä


and in nano:



 ^a^b
^a^d
^a^f


I don't remember what I had done, but it was correctly showing Japanese letters at least two days ago (and for more than 6 months).



What could be the problem and the way to recover from this?







terminal locale unicode fish






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited yesterday









Jeff Schaller

37.5k1052121




37.5k1052121










asked Oct 10 at 23:50









Todoroki

114




114








  • 2




    Just a shot in the dark - did you change any locale settings?
    – RudiC
    Oct 11 at 9:11














  • 2




    Just a shot in the dark - did you change any locale settings?
    – RudiC
    Oct 11 at 9:11








2




2




Just a shot in the dark - did you change any locale settings?
– RudiC
Oct 11 at 9:11




Just a shot in the dark - did you change any locale settings?
– RudiC
Oct 11 at 9:11










1 Answer
1






active

oldest

votes

















up vote
1
down vote



accepted










I had accidentally updated my fish config file to read ~/.profile,

which included a line saying locale=C.



I changed this to locale=C_UTF8 and everything recovered.






share|improve this answer





















    Your Answer








    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "106"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f474695%2fterminal-shows-non-ascii-characters-in-utf-16-hex-codes%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes








    up vote
    1
    down vote



    accepted










    I had accidentally updated my fish config file to read ~/.profile,

    which included a line saying locale=C.



    I changed this to locale=C_UTF8 and everything recovered.






    share|improve this answer

























      up vote
      1
      down vote



      accepted










      I had accidentally updated my fish config file to read ~/.profile,

      which included a line saying locale=C.



      I changed this to locale=C_UTF8 and everything recovered.






      share|improve this answer























        up vote
        1
        down vote



        accepted







        up vote
        1
        down vote



        accepted






        I had accidentally updated my fish config file to read ~/.profile,

        which included a line saying locale=C.



        I changed this to locale=C_UTF8 and everything recovered.






        share|improve this answer












        I had accidentally updated my fish config file to read ~/.profile,

        which included a line saying locale=C.



        I changed this to locale=C_UTF8 and everything recovered.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered 2 days ago









        Todoroki

        114




        114






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Unix & Linux Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.





            Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


            Please pay close attention to the following guidance:


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f474695%2fterminal-shows-non-ascii-characters-in-utf-16-hex-codes%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            サソリ

            広島県道265号伴広島線

            Setup Asymptote in Texstudio