Remove non-alphas while ignoring certain unicode chars












0














I have a text like the following: 



përgjithshme [" është "] 28.748 [< kilometra katrorë.>]


This text contains cyrillic characters in the range U400 - U4FF.



I want to strip out the non-alphas such as: "<> but when I do this in Perl using Matt Mahoney's script: http://mattmahoney.net/dc/textdata.html it also applies the following Perl line:



tr/a-z/ /cs;


This remove also the cyrillic chars above like ë.



Is there a way to do this in Perl, namely remove non-alphas whilst ignoring certain unicode chars that fall under a range like the one I mentioned above?






perl unicode







share|improve this question



























    0














    I have a text like the following: 



    përgjithshme [" është "] 28.748 [< kilometra katrorë.>]


    This text contains cyrillic characters in the range U400 - U4FF.



    I want to strip out the non-alphas such as: "<> but when I do this in Perl using Matt Mahoney's script: http://mattmahoney.net/dc/textdata.html it also applies the following Perl line:



    tr/a-z/ /cs;


    This remove also the cyrillic chars above like ë.



    Is there a way to do this in Perl, namely remove non-alphas whilst ignoring certain unicode chars that fall under a range like the one I mentioned above?






    perl unicode







    share|improve this question

























      0












      0








      0







      I have a text like the following: 



      përgjithshme [" është "] 28.748 [< kilometra katrorë.>]


      This text contains cyrillic characters in the range U400 - U4FF.



      I want to strip out the non-alphas such as: "<> but when I do this in Perl using Matt Mahoney's script: http://mattmahoney.net/dc/textdata.html it also applies the following Perl line:



      tr/a-z/ /cs;


      This remove also the cyrillic chars above like ë.



      Is there a way to do this in Perl, namely remove non-alphas whilst ignoring certain unicode chars that fall under a range like the one I mentioned above?






      perl unicode







      share|improve this question













      I have a text like the following: 



      përgjithshme [" është "] 28.748 [< kilometra katrorë.>]


      This text contains cyrillic characters in the range U400 - U4FF.



      I want to strip out the non-alphas such as: "<> but when I do this in Perl using Matt Mahoney's script: http://mattmahoney.net/dc/textdata.html it also applies the following Perl line:



      tr/a-z/ /cs;


      This remove also the cyrillic chars above like ë.



      Is there a way to do this in Perl, namely remove non-alphas whilst ignoring certain unicode chars that fall under a range like the one I mentioned above?






      perl unicode




      perl unicode






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked 20 mins ago









      bytebiscuit

      1435




      1435






















          1 Answer
          1






          active

          oldest

          votes


















          0














          tr/a-z/ /cs; would ask to replace with spaces all characters that are not in the set a-z. (The trailing c is to complement the set, otherwise it would replace all characters from a to z.)



          You'll have to be more specific. s/[[:punct:]]+/ /g would replace all sequences of punctuation characters with single spaces, and s/[^[:alpha:]]+/ /g would replace all sequences of non-alphabetic characters with single spaces.



          Note that at least [:alpha:] will be locale-specific, and you'll need to tell Perl to use UTF-8 with -C or such. Also note that [^[:alpha:]] removes digits and the trailing newline as well. This is in the en_US.UTF-8 locale on Debian:



          $ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[^[:alpha:]]+/ /g'
          
 përgjithshme është kilometra katrorë $ 
          

          
$ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[[:punct:]]+/ /g'
          
 përgjithshme   është   28 748   kilometra katrorë 
          
$




          share





















            Your Answer








            StackExchange.ready(function() {
            var channelOptions = {
            tags: "".split(" "),
            id: "106"
            };
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function() {
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled) {
            StackExchange.using("snippets", function() {
            createEditor();
            });
            }
            else {
            createEditor();
            }
            });

            function createEditor() {
            StackExchange.prepareEditor({
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: false,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: null,
            bindNavPrevention: true,
            postfix: "",
            imageUploader: {
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            },
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            });


            }
            });














            draft saved

            draft discarded


















            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f491464%2fremove-non-alphas-while-ignoring-certain-unicode-chars%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown

























            1 Answer
            1






            active

            oldest

            votes








            1 Answer
            1






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            0














            tr/a-z/ /cs; would ask to replace with spaces all characters that are not in the set a-z. (The trailing c is to complement the set, otherwise it would replace all characters from a to z.)



            You'll have to be more specific. s/[[:punct:]]+/ /g would replace all sequences of punctuation characters with single spaces, and s/[^[:alpha:]]+/ /g would replace all sequences of non-alphabetic characters with single spaces.



            Note that at least [:alpha:] will be locale-specific, and you'll need to tell Perl to use UTF-8 with -C or such. Also note that [^[:alpha:]] removes digits and the trailing newline as well. This is in the en_US.UTF-8 locale on Debian:



            $ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[^[:alpha:]]+/ /g'
            
 përgjithshme është kilometra katrorë $ 
            

            
$ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[[:punct:]]+/ /g'
            
 përgjithshme   është   28 748   kilometra katrorë 
            
$




            share


























              0














              tr/a-z/ /cs; would ask to replace with spaces all characters that are not in the set a-z. (The trailing c is to complement the set, otherwise it would replace all characters from a to z.)



              You'll have to be more specific. s/[[:punct:]]+/ /g would replace all sequences of punctuation characters with single spaces, and s/[^[:alpha:]]+/ /g would replace all sequences of non-alphabetic characters with single spaces.



              Note that at least [:alpha:] will be locale-specific, and you'll need to tell Perl to use UTF-8 with -C or such. Also note that [^[:alpha:]] removes digits and the trailing newline as well. This is in the en_US.UTF-8 locale on Debian:



              $ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[^[:alpha:]]+/ /g'
              
 përgjithshme është kilometra katrorë $ 
              

              
$ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[[:punct:]]+/ /g'
              
 përgjithshme   është   28 748   kilometra katrorë 
              
$




              share
























                0












                0








                0






                tr/a-z/ /cs; would ask to replace with spaces all characters that are not in the set a-z. (The trailing c is to complement the set, otherwise it would replace all characters from a to z.)



                You'll have to be more specific. s/[[:punct:]]+/ /g would replace all sequences of punctuation characters with single spaces, and s/[^[:alpha:]]+/ /g would replace all sequences of non-alphabetic characters with single spaces.



                Note that at least [:alpha:] will be locale-specific, and you'll need to tell Perl to use UTF-8 with -C or such. Also note that [^[:alpha:]] removes digits and the trailing newline as well. This is in the en_US.UTF-8 locale on Debian:



                $ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[^[:alpha:]]+/ /g'
                
 përgjithshme është kilometra katrorë $ 
                

                
$ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[[:punct:]]+/ /g'
                
 përgjithshme   është   28 748   kilometra katrorë 
                
$




                share












                tr/a-z/ /cs; would ask to replace with spaces all characters that are not in the set a-z. (The trailing c is to complement the set, otherwise it would replace all characters from a to z.)



                You'll have to be more specific. s/[[:punct:]]+/ /g would replace all sequences of punctuation characters with single spaces, and s/[^[:alpha:]]+/ /g would replace all sequences of non-alphabetic characters with single spaces.



                Note that at least [:alpha:] will be locale-specific, and you'll need to tell Perl to use UTF-8 with -C or such. Also note that [^[:alpha:]] removes digits and the trailing newline as well. This is in the en_US.UTF-8 locale on Debian:



                $ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[^[:alpha:]]+/ /g'
                
 përgjithshme është kilometra katrorë $ 
                

                
$ echo 'përgjithshme [" është "] 28.748 [< kilometra katrorë.>]' | perl -C -pe 's/[[:punct:]]+/ /g'
                
 përgjithshme   është   28 748   kilometra katrorë 
                
$





                share











                share


                share










                answered 1 min ago









                ilkkachu

                55.5k783151




                55.5k783151






























                    draft saved

                    draft discarded




















































                    Thanks for contributing an answer to Unix & Linux Stack Exchange!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid …



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.





                    Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


                    Please pay close attention to the following guidance:


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid …



                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function () {
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f491464%2fremove-non-alphas-while-ignoring-certain-unicode-chars%23new-answer', 'question_page');
                    }
                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    -

                    Popular posts from this blog

                    Accessing regular linux commands in Huawei's Dopra Linux

                    Image
                    4 Backstory: I have a Huawei HG8245 router in my house and I want to change the default username/password. I've tried following this guide but the config file always gets reset. My ISP isn't helping either so I'm looking for other ways to accomplish this. The router has telnet access so this seems like a way to change the credentials. The first shell I gain access to looks like this: WAP> Pressing "?" gives me a list of available commands which I've pasted here, the two interesting ones are "su" and "shell". I can't seem to get su access, even after logging in as root (challenge password prompt, and then nothing happens), but shell gives me a proper linux shell. WAP>shell BusyBox v1.18.4 (2017-12-26 17:06:34 CST) built-in shell (ash) Enter 'hel
                    Read more

                    Can't connect RFCOMM socket: Host is down

                    Image
                    up vote 3 down vote favorite I have bluez 5.4.3 and I am attempting to communicate with a ble device through minicom. These two deceives (my computer and a bluno nano) are alreadt paired and this has been my method so far. #bluetoothctl # pair xx:xx:xx:xx:xx:xx # exit Everything is working well so far then... #sudo rfcomm connect /dev/rfcomm0 xx:xx:xx:xx:xx:xx 1 & [1] 3665 Can't connect RFCOMM socket: Host is down Does anyone know how to resolve this issue? Any help would be very much appreciated. bluetooth bluez share | improve this question asked Feb 9 '17 at 18:46 Brandon Knape 46 2
                    Read more

                    Kernel panic - not syncing: Fatal Exception in Interrupt

                    Image
                    0 Every day this error occured on my pc, kernel panic error on interrupt, what will be the problem? linux-mint share asked 6 mins ago Vinod Selvin Vinod Selvin 1 1 New contributor Vinod Selvin is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct. add a comment
                    Read more