Remove non-alphas while ignoring certain unicode chars












0














I have a text like the following: 



përgjithshme [" është "] 28.748 [< kilometra katrorë.>]


This text contains cyrillic characters in the range U400 - U4FF.



I want to strip out the non-alphas such as: "<> but when I do this in Perl using Matt Mahoney's script: http://mattmahoney.net/dc/textdata.html it also applies the following Perl line:



tr/a-z/ /cs;


This remove also the cyrillic chars above like ë.



Is there a way to do this in Perl, namely remove non-alphas whilst ignoring certain unicode chars that fall under a range like the one I mentioned above?






perl unicode







share|improve this question



























    0














    I have a text like the following: 



    përgjithshme [" është "] 28.748 [< kilometra katrorë.>]


    This text contains cyrillic characters in the range U400 - U4FF.



    I want to strip out the non-alphas such as: "<> but when I do this in Perl using Matt Mahoney's script: http://mattmahoney.net/dc/textdata.html it also applies the following Perl line:



    tr/a-z/ /cs;


    This remove also the cyrillic chars above like ë.



    Is there a way to do this in Perl, namely remove non-alphas whilst ignoring certain unicode chars that fall under a range like the one I mentioned above?






    perl unicode







    share|improve this question

























      0












      0








      0







      I have a text like the following: 



      përgjithshme [" është "] 28.748 [< kilometra katrorë.>]


      This text contains cyrillic characters in the range U400 - U4FF.



      I want to strip out the non-alphas such as: "<> but when I do this in Perl using Matt Mahoney's script: http://mattmahoney.net/dc/textdata.html it also applies the following Perl line:



      tr/a-z/ /cs;


      This remove also the cyrillic chars above like ë.



      Is there a way to do this in Perl, namely remove non-alphas whilst ignoring certain unicode chars that fall under a range like the one I mentioned above?






      perl unicode







      share|improve this question













      I have a text like the following: 



      përgjithshme [" është "] 28.748 [< kilometra katrorë.>]


      This text contains cyrillic characters in the range U400 - U4FF.



      I want to strip out the non-alphas such as: "<> but when I do this in Perl using Matt Mahoney's script: http://mattmahoney.net/dc/textdata.html it also applies the following Perl line:



      tr/a-z/ /cs;


      This remove also the cyrillic chars above like ë.



      Is there a way to do this in Perl, namely remove non-alphas whilst ignoring certain unicode chars that fall under a range like the one I mentioned above?






      perl unicode




      perl unicode






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked 20 mins ago









      bytebiscuit

      1435




      1435






















          1 Answer
          1






          active

          oldest

          votes


















          0














          tr/a-z/ /cs; would ask to replace with spaces all characters that are not in the set a-z. (The trailing c is to complement the set, otherwise it would replace all characters from a to z.)



          You'll have to be more specific. s/[[:punct:]]+/ /g would replace all sequences of punctuation characters with single spaces, and s/[^[:alpha:]]+/ /g would replace all sequences of non-alphabetic characters with single spaces.



          Note that at least [:alpha:] will be locale-specific, and you'll need to tell Perl to use UTF-8 with -C or such. Also note that [^[:alpha:]] removes digits and the trailing newline as well. This is in the en_US.UTF-8 locale on Debian:



          $ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[^[:alpha:]]+/ /g'
          
 përgjithshme është kilometra katrorë $ 
          

          
$ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[[:punct:]]+/ /g'
          
 përgjithshme   është   28 748   kilometra katrorë 
          
$




          share





















            Your Answer








            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "106"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: false,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: null,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f491464%2fremove-non-alphas-while-ignoring-certain-unicode-chars%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            0














            tr/a-z/ /cs; would ask to replace with spaces all characters that are not in the set a-z. (The trailing c is to complement the set, otherwise it would replace all characters from a to z.)



            You'll have to be more specific. s/[[:punct:]]+/ /g would replace all sequences of punctuation characters with single spaces, and s/[^[:alpha:]]+/ /g would replace all sequences of non-alphabetic characters with single spaces.



            Note that at least [:alpha:] will be locale-specific, and you'll need to tell Perl to use UTF-8 with -C or such. Also note that [^[:alpha:]] removes digits and the trailing newline as well. This is in the en_US.UTF-8 locale on Debian:



            $ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[^[:alpha:]]+/ /g'
            
 përgjithshme është kilometra katrorë $ 
            

            
$ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[[:punct:]]+/ /g'
            
 përgjithshme   është   28 748   kilometra katrorë 
            
$




            share


























              0














              tr/a-z/ /cs; would ask to replace with spaces all characters that are not in the set a-z. (The trailing c is to complement the set, otherwise it would replace all characters from a to z.)



              You'll have to be more specific. s/[[:punct:]]+/ /g would replace all sequences of punctuation characters with single spaces, and s/[^[:alpha:]]+/ /g would replace all sequences of non-alphabetic characters with single spaces.



              Note that at least [:alpha:] will be locale-specific, and you'll need to tell Perl to use UTF-8 with -C or such. Also note that [^[:alpha:]] removes digits and the trailing newline as well. This is in the en_US.UTF-8 locale on Debian:



              $ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[^[:alpha:]]+/ /g'
              
 përgjithshme është kilometra katrorë $ 
              

              
$ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[[:punct:]]+/ /g'
              
 përgjithshme   është   28 748   kilometra katrorë 
              
$




              share
























                0












                0








                0






                tr/a-z/ /cs; would ask to replace with spaces all characters that are not in the set a-z. (The trailing c is to complement the set, otherwise it would replace all characters from a to z.)



                You'll have to be more specific. s/[[:punct:]]+/ /g would replace all sequences of punctuation characters with single spaces, and s/[^[:alpha:]]+/ /g would replace all sequences of non-alphabetic characters with single spaces.



                Note that at least [:alpha:] will be locale-specific, and you'll need to tell Perl to use UTF-8 with -C or such. Also note that [^[:alpha:]] removes digits and the trailing newline as well. This is in the en_US.UTF-8 locale on Debian:



                $ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[^[:alpha:]]+/ /g'
                
 përgjithshme është kilometra katrorë $ 
                

                
$ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[[:punct:]]+/ /g'
                
 përgjithshme   është   28 748   kilometra katrorë 
                
$




                share












                tr/a-z/ /cs; would ask to replace with spaces all characters that are not in the set a-z. (The trailing c is to complement the set, otherwise it would replace all characters from a to z.)



                You'll have to be more specific. s/[[:punct:]]+/ /g would replace all sequences of punctuation characters with single spaces, and s/[^[:alpha:]]+/ /g would replace all sequences of non-alphabetic characters with single spaces.



                Note that at least [:alpha:] will be locale-specific, and you'll need to tell Perl to use UTF-8 with -C or such. Also note that [^[:alpha:]] removes digits and the trailing newline as well. This is in the en_US.UTF-8 locale on Debian:



                $ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[^[:alpha:]]+/ /g'
                
 përgjithshme është kilometra katrorë $ 
                

                
$ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[[:punct:]]+/ /g'
                
 përgjithshme   është   28 748   kilometra katrorë 
                
$





                share











                share


                share










                answered 1 min ago









                ilkkachu

                55.5k783151




                55.5k783151






























                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to Unix & Linux Stack Exchange!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid …



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.





                    Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


                    Please pay close attention to the following guidance:


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid …



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f491464%2fremove-non-alphas-while-ignoring-certain-unicode-chars%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    -

                    Popular posts from this blog

                    サソリ

                    Image
                    この記事は検証可能な参考文献や出典が全く示されていないか、不十分です。 出典を追加して記事の信頼性向上にご協力ください。 （ 2019年1月 ） 「 サソリ 」のその他の用法については「サソリ (曖昧さ回避)」をご覧ください。 「 スコーピオン 」と「 スコルピオン 」はこの項目へ転送されています。その他の用法については「スコーピオン」をご覧ください。 サソリ目 生息年代: 430–0 Ma PreЄ Є O S D C P T J K Pg N シルル紀前期-現世 ダイオウサソリ（ Pandinus Imperator ） 分類 界 : 動物界 Animalia 門 : 節足動物門 Arthropoda 亜門 : 鋏角亜門 Chelicerata 綱 : クモガタ綱 Arachnida 目 : サソリ目 Scorpiones 学名 Scorpiones C. L. Koch, 1837 英名 Scorpion 上科 Pseudochactoidea キョクトウサソリ上科 Buthoidea Chaeriloidea Chactoidea Iuroidea Scorpionoidea サソリ （ サソリ類 、 蠍 、 蝎 、 scorpion ）は、鋏角亜門・クモガタ綱の サソリ目 （ Scorpiones ）に属する節足動物の総称である。体の前端に鋏型の触肢、後端に毒針を有する捕食者である。 1700種以上を含め、最古の化石記録は4億3千万年前のシルル紀まで遡る。有毒生物として広く知られるが、人間の命にかかわる毒を持つものはごく一部の種類に過ぎない。 目次 1 形態 1.1 前体 1.2 後体 1.2.1 中体 1.2.2 終体 2 雌雄 3 生態 3.1 天敵 3.2 生殖 4 分布 5 毒 6 蛍光現象 7 分類
                    Read more

                    広島県道265号伴広島線

                    Image
                    一般県道 広島県道265号伴広島線 路線延長 km 起点 広島県広島市安佐南区伴東7丁目 終点 広島県広島市中区榎町 接続する 主な道路 （記法） 広島県道38号広島豊平線 広島県道71号広島湯来線 山陽自動車道五日市IC 国道183号（ 国道261号重複） ■テンプレート（■ノート ■使い方） ■PJ道路 広島県道265号伴広島線 （ひろしまけんどう265ごう ともひろしません）は広島県広島市安佐南区から広島県広島市中区を結ぶ一般県道である。 目次 1 路線データ 2 通過する自治体 3 見どころ 4 接続道路 4.1 本線 4.2 旧道 5 別名 6 関連項目 路線データ 本線 起点：広島市安佐南区伴東7丁目・大原下橋南詰交差点（広島県道38号広島豊平線（本線・旧道）交点） 終点：広島市中区榎町・十日市交差点（国道183号（国道261号重用）交点） 総延長： 旧道 起点：広島市安佐南区伴東4丁目・新細坂交差点（広島県道38号広島豊平線交点） 終点：広島市安佐南区伴中央4丁目・寺組橋(南)交差点（広島県道265号伴広島線（本線）交点） 総延長： 通過する自治体 広島県 広島市（安佐南区、佐伯区、西区、中区） 見どころ 広島高速交通広島新交通1号線（アストラムライン） 広島広域公園 広島広域公園陸上競技場（広島ビッグアーチ） 己斐峠 己斐橋（太田川放水路） 天満橋（天満川） 接続道路 本線 広島県道38号広島豊平線（本線・旧道）（広島市安佐南区伴東7丁目・大原下橋南詰交差点〔起点〕） 広島県道71号広島湯来線（本線）（広島市安佐南区伴中央2丁目・伴交番入口交差点 - 広島市安佐南区伴中央4丁目・寺組橋(南)交差点間で重用） 広島県道71号広島湯来線（旧道）（広島県道265号伴広島線（旧道）重用）（広島市安佐南区伴中央4丁目・寺組橋(南)交差点） 広島県道71号広島湯来線（広島市安佐南区伴中央4丁目・寺組橋(南)交差点 - 広島市佐伯区五日市町石内・五月が丘(南)交差点間で重用）
                    Read more

                    Accessing regular linux commands in Huawei's Dopra Linux

                    Image
                    4 Backstory: I have a Huawei HG8245 router in my house and I want to change the default username/password. I've tried following this guide but the config file always gets reset. My ISP isn't helping either so I'm looking for other ways to accomplish this. The router has telnet access so this seems like a way to change the credentials. The first shell I gain access to looks like this: WAP> Pressing "?" gives me a list of available commands which I've pasted here, the two interesting ones are "su" and "shell". I can't seem to get su access, even after logging in as root (challenge password prompt, and then nothing happens), but shell gives me a proper linux shell. WAP>shell BusyBox v1.18.4 (2017-12-26 17:06:34 CST) built-in shell (ash) Enter 'hel
                    Read more