Sstrhsrtj

I have a number of mirrors of Redhat YUM repositories, which are updated on a daily basis. The commands that are used to accomplish this are:

reposync --repoid=${i} --download_path=${destdir}  --gpgcheck -l --download-metadata --downloadcomps --newest --delete



createrepo -s sha256 --checkts --update --workers=4 -g $destdir/$fn/comps.xml

The variables (i, destdir and fn) are set in the script that issues the commands. This works all very well, and the team has been using the mirrors to good effect.

The problem is that, after a year or so, one of the repositories has accumulated an impressive stack of updateinfo xml files, with names of the pattern <hash>-updateinfo.xml.gz: 456MB in the top directory and 28.45GB in the repodata subdirectory. The repository contains only 4GB of package files.

Clients that do a yum makecache on this repo end up with a 4GB repmod.xml file.

My questions are

1. Why do these files accumulate, even though I have --delete
   specified.. ?


2. Can I remove them without breaking the repository?


3. Are the parameters that I use the most optimal? We want to mirror a
   complete repo, but only the latest version of every package.

EDIT 4/6/2018

After deeper digging I found some more hints that these files are in fact not required.

The <hash>updateinfo.xml.gz files in the top directory in the repository are all more or less the same size, about 3.8M. The files in the repodata directory (which is created/updated by createrepo) constantly grow in size due to all files in the top directory being concatenated.

e.g.: in this repodata directory, I've got 129 gzipped files. The first file has the same average size as the ones in the top directory, the last is huge and has 129 updates tags, against the first one only 1.

# l -tr

total 29G

-rw-r--r-- 1 root root 3.5M Sep 28  2016 6f9c8bca09bb360b0ac2c18231168d45aa6ef51254fee7b791c6d09693677f4c-updateinfo.xml.gz

...

-rw-r--r-- 1 root root 465M May 17 03:21 1696bec0516791660751bb4a319b287f2a3a5ecfee086aefb73285f07cad3ac5-updateinfo.xml.gz

drwxr-xr-x 3 root root  20K May 22 12:37 ../

# gzip -dc  1696bec0516791660751bb4a319b287f2a3a5ecfee086aefb73285f07cad3ac5-updateinfo.xml.gz >updateinfo-big.xml

# gzip -dc  6f9c8bca09bb360b0ac2c18231168d45aa6ef51254fee7b791c6d09693677f4c-updateinfo.xml.gz >updateinfo.xml

# grep '<updates>' updateinfo.xml |wc -l

1

# grep '<updates>' updateinfo-big.xml |wc -l

129

# ls -1 *updateinfo.xml.gz|wc -l

129

# l updateinfo*

-rw-r--r-- 1 root root 2.4G Jun  4 17:09 updateinfo-big.xml

-rw-r--r-- 1 root root  18M Jun  4 17:10 updateinfo.xml

I guess that the reposync should remove any existing updateinfo.xml.gz files in the top directory before the createrepo runs over it. The client gets the latest gzipped file from the repodata directory when it does a makecache, and unzips it.

I've moved the stack to a backup directory after posting the question above and saw no adverse effects on the clients.

Answering my own question, to document this for others.

We are now practically sure that the old updateinfo.xml files are superfluous to needs. Apparently, they accumulate only because the of the hash value that is prepended to the file name. Based on this, I made some changes and now the repository remains mostly static in size.

In its original form, after the reposync and createrepo commands cited in the question, the script runs gunzip followed by a modifyrepo command which creates a new updateinfo.xml.gz file in the ../repodata directory:

  if  [ -n "$(/bin/ls -t $destdir/$fn/*updateinfo.xml.gz 2>/dev/null)" ]; then

     gunzip -c $(/bin/ls -t $destdir/$fn/*updateinfo.xml.gz) > $destdir/$fn/updateinfo.xml 2>> $LOGFILE

     modifyrepo $destdir/$fn/updateinfo.xml $destdir/$fn/repodata  >> $LOGFILE 2>&1

  fi

I changed this section to:

  if  [ -n "$(/bin/ls -t $destdir/$fn/*updateinfo.xml.gz 2>/dev/null)" ]; then

     gunzip -c $(/bin/ls -tr $destdir/$fn/*updateinfo.xml.gz|tail -1) > $destdir/$fn/updateinfo.xml 2>> $LOGFILE

     modifyrepo $destdir/$fn/updateinfo.xml $destdir/$fn/repodata  >> $LOGFILE 2>&1



     # clean up old update info - keeping only the 2 most recent files.

     for i in $destdir/$fn $destdir/$fn/repodata; do

         for j in `/bin/ls -t ${i}/*updateinfo.xml.gz|tail -n +3`; do

            echo "removing security file "$(ls -l ${j}) >> $LOGFILE

            /bin/rm -f ${j} >> $LOGFILE 2>&1

         done

     done

  fi

The gunzip command only unzips the latest updateinfo.xml due to the reverse sort on the timestamp and the tail command. As a result, the new file in the repodata directory will only contain one version. The second change is to delete all older updateinfo.xml files, bar 2 (just in case).

We have been running with this version for several months and have not noticed any unwanted side effects.