Sstrhsrtj

Morning All,

I'm new to UNIX so looking to do something I can do in VB but don't have the experience to do in UNIX.

I have an xml spec on a share which requires regular deletion and updates as new reuters RIC codes come online. Two items to achieve:

A. Remove a RIC entry

1. Open the file


2. Find a specific string


3. Delete this found line and the 21 lines below it


4. Save file

I though this might work:

sed –e '/<ric id="AUG03250639E=YBAU">/,+21d'  a.xml >a.xml

B. Add a new RIC entry

1. Open a file


2. Find the last occurrence of the sting </source>
   


3. Move up 29 lines to the last RIC entry block


4. Copy this line and the 21 lines below (the ric block)


5. Insert a new line 22 lines below and paste this bloc (a new block) i.e. directly below the block you copied


6. Change the ric on line 1 of the new block to a new Ric string i.e. <ricid="AAAAA=YBAU" to <ricid="BBBBB=YBAU"
   


7. Save file

How can I do this?

This is the last section of the file. Note the end of the ric blocks (which I want to manipulate) is when the following strings appear
.
.
.

            <ric id="AUG03250639E=YBAU">

                <securities>

                    <security>

                        <issueid>178117</issueid>

                        <quote-type>YIELD</quote-type>

                        <complex-logic>

                            <calculations>

                                <yield-type>

                                    <type>BID_YIELD</type>

                                    <calculation name="AB" field="RT_YIELD_1" />

                                </yield-type>

                                <yield-type>

                                    <type>OFFER_YIELD</type>

                                    <calculation name="AB" field="SEC_YLD_1" />

                                </yield-type>

                            </calculations>

                        </complex-logic>

                        <derived-type name="PRICE" baseValue="100.0" />

                    </security>

                </securities>

            </ric>

            <ric id="AUG03250640E=YBAU">

                <securities>

                    <security>

                        <issueid>178117</issueid>

                        <quote-type>YIELD</quote-type>

                        <complex-logic>

                            <calculations>

                                <yield-type>

                                    <type>BID_YIELD</type>

                                    <calculation name="AB" field="RT_YIELD_1" />

                                </yield-type>

                                <yield-type>

                                    <type>OFFER_YIELD</type>

                                    <calculation name="AB" field="SEC_YLD_1" />

                                </yield-type>

                            </calculations>

                        </complex-logic>

                        <derived-type name="PRICE" baseValue="100.0" />

                    </security>

                </securities>

            </ric>

        </rics>

        <topics>

            <topic>

                <id>default</id>

                <type>rmds</type>

                <value>IDN_SELECTFEED.ANY.%s.NaE</value>

            </topic>

        </topics>

    </source>



    <transformers>

        <!-- Name of transformer -->

        <transformer></transformer>

    </transformers>



    <processors>

        <!-- Enricher to add additional fields from source query result while 

            publishing -->

        <processor></processor>

    </processors>

    <endpoints>

        <!-- Order of post processor is important. First topic, then mapper -->



        <endpoint id="rmds" topic="FI.ANY.%s.YBAU" multicast="true">



            <postprocessor>reuters-topic-builder</postprocessor>

            <postprocessor>reuters-message-mapper</postprocessor>

            <!-- <multitopic id="solace" topic="LN/FI/IP/APS/SSHEET/YIELD/BATS_%s" 

                /> -->

            <multitopic id="solace-credit" topic="LN/FI/EP/CREDITBPS/SSHEET/BATS_%s" />

            <multitopic id="solace-credit" topic="LN/FI/EP/CREDITBPS/YIELDBROKER/BATS_%s" />

        </endpoint>



    </endpoints>

    <other-properties>

        <!-- common formatting of price/yield -->

        <property name="formattor-1">(math:pow(INPUT/100+1,0.5)-1)*200</property>

        <property name="handle_negative_values">false</property>

        <property name="handle_negative_values_output">0.001</property>

    </other-properties>

</specification>

So for A. Remove a RIC entry where I want to remove AUG03250640E=YBAU, the file would show:

            <ric id="AUG03250639E=YBAU">

                <securities>

                    <security>

                        <issueid>178117</issueid>

                        <quote-type>YIELD</quote-type>

                        <complex-logic>

                            <calculations>

                                <yield-type>

                                    <type>BID_YIELD</type>

                                    <calculation name="AB" field="RT_YIELD_1" />

                                </yield-type>

                                <yield-type>

                                    <type>OFFER_YIELD</type>

                                    <calculation name="AB" field="SEC_YLD_1" />

                                </yield-type>

                            </calculations>

                        </complex-logic>

                        <derived-type name="PRICE" baseValue="100.0" />

                    </security>

                </securities>

            </ric>

        </rics>

        <topics>

            <topic>

                <id>default</id>

                <type>rmds</type>

                <value>IDN_SELECTFEED.ANY.%s.NaE</value>

            </topic>

        </topics>

    </source>



    <transformers>

        <!-- Name of transformer -->

        <transformer></transformer>

    </transformers>



    <processors>

        <!-- Enricher to add additional fields from source query result while 

            publishing -->

        <processor></processor>

    </processors>

    <endpoints>

        <!-- Order of post processor is important. First topic, then mapper -->



        <endpoint id="rmds" topic="FI.ANY.%s.YBAU" multicast="true">



            <postprocessor>reuters-topic-builder</postprocessor>

            <postprocessor>reuters-message-mapper</postprocessor>

            <!-- <multitopic id="solace" topic="LN/FI/IP/APS/SSHEET/YIELD/BATS_%s" 

                /> -->

            <multitopic id="solace-credit" topic="LN/FI/EP/CREDITBPS/SSHEET/BATS_%s" />

            <multitopic id="solace-credit" topic="LN/FI/EP/CREDITBPS/YIELDBROKER/BATS_%s" />

        </endpoint>



    </endpoints>

    <other-properties>

        <!-- common formatting of price/yield -->

        <property name="formattor-1">(math:pow(INPUT/100+1,0.5)-1)*200</property>

        <property name="handle_negative_values">false</property>

        <property name="handle_negative_values_output">0.001</property>

    </other-properties>

</specification>

For B. Add a new RIC entry assuming I want to add the new ric AUG03250641E=YBAU, the file would show:

            <ric id="AUG03250639E=YBAU">

                <securities>

                    <security>

                        <issueid>178117</issueid>

                        <quote-type>YIELD</quote-type>

                        <complex-logic>

                            <calculations>

                                <yield-type>

                                    <type>BID_YIELD</type>

                                    <calculation name="AB" field="RT_YIELD_1" />

                                </yield-type>

                                <yield-type>

                                    <type>OFFER_YIELD</type>

                                    <calculation name="AB" field="SEC_YLD_1" />

                                </yield-type>

                            </calculations>

                        </complex-logic>

                        <derived-type name="PRICE" baseValue="100.0" />

                    </security>

                </securities>

            </ric>

            <ric id="AUG03250640E=YBAU">

                <securities>

                    <security>

                        <issueid>178117</issueid>

                        <quote-type>YIELD</quote-type>

                        <complex-logic>

                            <calculations>

                                <yield-type>

                                    <type>BID_YIELD</type>

                                    <calculation name="AB" field="RT_YIELD_1" />

                                </yield-type>

                                <yield-type>

                                    <type>OFFER_YIELD</type>

                                    <calculation name="AB" field="SEC_YLD_1" />

                                </yield-type>

                            </calculations>

                        </complex-logic>

                        <derived-type name="PRICE" baseValue="100.0" />

                    </security>

                </securities>

            </ric>

            <ric id="AUG03250641E=YBAU">

                <securities>

                    <security>

                        <issueid>178117</issueid>

                        <quote-type>YIELD</quote-type>

                        <complex-logic>

                            <calculations>

                                <yield-type>

                                    <type>BID_YIELD</type>

                                    <calculation name="AB" field="RT_YIELD_1" />

                                </yield-type>

                                <yield-type>

                                    <type>OFFER_YIELD</type>

                                    <calculation name="AB" field="SEC_YLD_1" />

                                </yield-type>

                            </calculations>

                        </complex-logic>

                        <derived-type name="PRICE" baseValue="100.0" />

                    </security>

                </securities>

            </ric>

        </rics>

        <topics>

            <topic>

                <id>default</id>

                <type>rmds</type>

                <value>IDN_SELECTFEED.ANY.%s.NaE</value>

            </topic>

        </topics>

    </source>



    <transformers>

        <!-- Name of transformer -->

        <transformer></transformer>

    </transformers>



    <processors>

        <!-- Enricher to add additional fields from source query result while 

            publishing -->

        <processor></processor>

    </processors>

    <endpoints>

        <!-- Order of post processor is important. First topic, then mapper -->



        <endpoint id="rmds" topic="FI.ANY.%s.YBAU" multicast="true">



            <postprocessor>reuters-topic-builder</postprocessor>

            <postprocessor>reuters-message-mapper</postprocessor>

            <!-- <multitopic id="solace" topic="LN/FI/IP/APS/SSHEET/YIELD/BATS_%s" 

                /> -->

            <multitopic id="solace-credit" topic="LN/FI/EP/CREDITBPS/SSHEET/BATS_%s" />

            <multitopic id="solace-credit" topic="LN/FI/EP/CREDITBPS/YIELDBROKER/BATS_%s" />

        </endpoint>



    </endpoints>

    <other-properties>

        <!-- common formatting of price/yield -->

        <property name="formattor-1">(math:pow(INPUT/100+1,0.5)-1)*200</property>

        <property name="handle_negative_values">false</property>

        <property name="handle_negative_values_output">0.001</property>

    </other-properties>

</specification>

For A with POSIX sed and head utilities and a regular in file:

{   sed  -ne'/^<ric id="AUG03250639E=YBAU">$/q;p'

    head -n21 >/dev/null

    cat

}   <in >out

Except for the fact that parsing and/or manipulating xml with regular expressions is misguided at best (although simple things like this can work if you're careful), you almost had it right:

Using GNU sed:

sed –i -e '/<ric id="AUG03250639E=YBAU">/,+21d'  a.xml

If your sed doesn't support the -i (--in-place) option, you can do it with a temporary file (which is how sed does it behind the scenes):

TF=$(mktemp)

sed -e '/<ric id="AUG03250639E=YBAU">/,+21d'  a.xml > "$TF" && mv "$TF" a.xml

You can't read from a file and redirect output to it in the shell like you tried - the first thing the shell will do is overwrite the file so it is empty, and that happens before the sed script runs.

For more complex XML parsing tasks, use an XML parser like xmlstarlet in shell scripts or one of the many XML parsing libraries for perl or python (or almost any other language you can think of)

Don't use regular expressions to parse XML. It's dirty - prone to breaking and creates brittle code. There's a bunch of things that can trip you up trivially, like line counts - it is perfectly valid within XML to format an element:

<calculation name="AB" field="SEC_YLD_1" />

Or:

<calculation

    field="SEC_YLD_1"

    name="AB"

/>

And a variety of other options - all of which are semantically identical, but ... won't match the same regex.

For your sample, this is shockingly easy if you use a parser. perl has XML::Twig which can do this handily:

Delete:

#!/usr/bin/env perl

use strict;

use warnings;

use XML::Twig;



my $twig = XML::Twig -> parsefile ( 'your_file.xml' );



$_ -> delete for $twig -> get_xpath('//ric[@id="AUG03250639E=YBAU"]');



$twig -> set_pretty_print('indented');

$twig -> print;

Note - does delete duplicates if any exist.

Now, creating a new one - it looks like you're trying to copy and amend - so:

#!/usr/bin/env perl

use strict;

use warnings;

use XML::Twig;



my $twig = XML::Twig -> parsefile ( 'your_file.xml' );



#find one to copy - this will just get the first 'ric' element. 

my $ric_to_copy = $twig -> get_xpath('//ric',0); 

#copy it

my $new_ric = $ric_to_copy -> copy;

#alter the new one

$new_ric -> set_att('id', 'BBBBB=YBAU' );

#paste it

$new_ric -> paste ( 'last_child', $ric_to_copy->parent);

$twig -> set_pretty_print('indented');

$twig -> print;

Now this reads, and prints to STDOUT - you can print to a particular file:

my ( $output, '>', 'new.xml') or die $!;

print {$output} $twig -> sprint;

close ( $output );