Break a large file into smaller pieces











up vote
60
down vote

favorite
17












How do I break a large, +4GB file into smaller files of about 500MB each.



And how do I re-assemble them again to get the original file?






command-line split







share|improve this question




















  • 2




    text line-wise version: stackoverflow.com/questions/2016894/…
    – Ciro Santilli 新疆改造中心 六四事件 法轮功
    Apr 26 '16 at 12:22










  • Related: How to split larger files into smaller parts?
    – kenorb
    Oct 22 '17 at 13:40















up vote
60
down vote

favorite
17












How do I break a large, +4GB file into smaller files of about 500MB each.



And how do I re-assemble them again to get the original file?






command-line split







share|improve this question




















  • 2




    text line-wise version: stackoverflow.com/questions/2016894/…
    – Ciro Santilli 新疆改造中心 六四事件 法轮功
    Apr 26 '16 at 12:22










  • Related: How to split larger files into smaller parts?
    – kenorb
    Oct 22 '17 at 13:40













up vote
60
down vote

favorite
17









up vote
60
down vote

favorite
17






17





How do I break a large, +4GB file into smaller files of about 500MB each.



And how do I re-assemble them again to get the original file?






command-line split







share|improve this question















How do I break a large, +4GB file into smaller files of about 500MB each.



And how do I re-assemble them again to get the original file?






command-line split




command-line split






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Feb 20 '11 at 19:13









Gilles

523k12610421575




523k12610421575










asked Sep 4 '10 at 18:51









Stefan

11.4k3182123




11.4k3182123








  • 2




    text line-wise version: stackoverflow.com/questions/2016894/…
    – Ciro Santilli 新疆改造中心 六四事件 法轮功
    Apr 26 '16 at 12:22










  • Related: How to split larger files into smaller parts?
    – kenorb
    Oct 22 '17 at 13:40














  • 2




    text line-wise version: stackoverflow.com/questions/2016894/…
    – Ciro Santilli 新疆改造中心 六四事件 法轮功
    Apr 26 '16 at 12:22










  • Related: How to split larger files into smaller parts?
    – kenorb
    Oct 22 '17 at 13:40








2




2




text line-wise version: stackoverflow.com/questions/2016894/…
– Ciro Santilli 新疆改造中心 六四事件 法轮功
Apr 26 '16 at 12:22




text line-wise version: stackoverflow.com/questions/2016894/…
– Ciro Santilli 新疆改造中心 六四事件 法轮功
Apr 26 '16 at 12:22












Related: How to split larger files into smaller parts?
– kenorb
Oct 22 '17 at 13:40




Related: How to split larger files into smaller parts?
– kenorb
Oct 22 '17 at 13:40










2 Answers
2






active

oldest

votes

















up vote
71
down vote



accepted










You can use split and cat.



For example something like



$ split --bytes 500M --numeric-suffixes --suffix-length=3 foo foo.


(where the input filename is foo and the last argument is the output prefix). This will create files like foo.000 foo.001 ...



The same command with short options:



$ split -b 100k -d -a 3 foo foo


You can also specify "--line-bytes" if you wish it to split on line boundaries instead of just exact number of bytes.



For re-assembling the generated pieces again you can use e.g.:



$ cat foo.* > foo_2


(assuming that the shell sorts the results of shell globbing - and the number of parts does not exceed the system dependent limit of arguments)



You can compare the result via:



$ cmp foo foo_2

$ echo $?


(which should output 0)



Alternatively, you can use a combination of find/sort/xargs to re-assemble the pieces:



$ find -maxdepth 1 -type f -name 'foo.*'  | sort | xargs cat > foo_3





share|improve this answer



















  • 2




    Try this command: man split cat md5sum
    – Kevin M
    Sep 4 '10 at 19:13






  • 5




    When assembling, I recommend cat foo.{000..NNN} where NNN is the last expected piece. That way you get an error message if one of the pieces is missing. But note that -d to get numeric suffixes is specific to GNU split; on other platforms you have to make do with foo.aaa, foo.aab, etc.
    – Gilles
    Oct 17 '10 at 11:16






  • 1




    And bear in mind that, for split, KB = 1000, K = 1024, MB = 1000*1000, M = 1024*1024 etc.
    – Zorawar
    Nov 29 '12 at 18:05










  • Shouldn't this ... cat > foo_3 be ... cat >>foo_3?
    – alk
    Jul 8 '15 at 12:10








  • 1




    If you decide to ease pain by using a utility. rar and 7zip are often used in making such splits easier to reassemble cross-platform
    – infixed
    Jun 3 '16 at 18:05


















up vote
4
down vote













You can also do this with Archive Manager if you prefer a GUI. Look under 'Save->Other Options->Split into volumes of'.






share|improve this answer

















  • 5




    i tagged it 'command-line', but thanks for the answer :)
    – Stefan
    Sep 5 '10 at 19:25











Your Answer








StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f1588%2fbreak-a-large-file-into-smaller-pieces%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























2 Answers
2






active

oldest

votes








2 Answers
2






active

oldest

votes









active

oldest

votes






active

oldest

votes








up vote
71
down vote



accepted










You can use split and cat.



For example something like



$ split --bytes 500M --numeric-suffixes --suffix-length=3 foo foo.


(where the input filename is foo and the last argument is the output prefix). This will create files like foo.000 foo.001 ...



The same command with short options:



$ split -b 100k -d -a 3 foo foo


You can also specify "--line-bytes" if you wish it to split on line boundaries instead of just exact number of bytes.



For re-assembling the generated pieces again you can use e.g.:



$ cat foo.* > foo_2


(assuming that the shell sorts the results of shell globbing - and the number of parts does not exceed the system dependent limit of arguments)



You can compare the result via:



$ cmp foo foo_2

$ echo $?


(which should output 0)



Alternatively, you can use a combination of find/sort/xargs to re-assemble the pieces:



$ find -maxdepth 1 -type f -name 'foo.*'  | sort | xargs cat > foo_3





share|improve this answer



















  • 2




    Try this command: man split cat md5sum
    – Kevin M
    Sep 4 '10 at 19:13






  • 5




    When assembling, I recommend cat foo.{000..NNN} where NNN is the last expected piece. That way you get an error message if one of the pieces is missing. But note that -d to get numeric suffixes is specific to GNU split; on other platforms you have to make do with foo.aaa, foo.aab, etc.
    – Gilles
    Oct 17 '10 at 11:16






  • 1




    And bear in mind that, for split, KB = 1000, K = 1024, MB = 1000*1000, M = 1024*1024 etc.
    – Zorawar
    Nov 29 '12 at 18:05










  • Shouldn't this ... cat > foo_3 be ... cat >>foo_3?
    – alk
    Jul 8 '15 at 12:10








  • 1




    If you decide to ease pain by using a utility. rar and 7zip are often used in making such splits easier to reassemble cross-platform
    – infixed
    Jun 3 '16 at 18:05















up vote
71
down vote



accepted










You can use split and cat.



For example something like



$ split --bytes 500M --numeric-suffixes --suffix-length=3 foo foo.


(where the input filename is foo and the last argument is the output prefix). This will create files like foo.000 foo.001 ...



The same command with short options:



$ split -b 100k -d -a 3 foo foo


You can also specify "--line-bytes" if you wish it to split on line boundaries instead of just exact number of bytes.



For re-assembling the generated pieces again you can use e.g.:



$ cat foo.* > foo_2


(assuming that the shell sorts the results of shell globbing - and the number of parts does not exceed the system dependent limit of arguments)



You can compare the result via:



$ cmp foo foo_2

$ echo $?


(which should output 0)



Alternatively, you can use a combination of find/sort/xargs to re-assemble the pieces:



$ find -maxdepth 1 -type f -name 'foo.*'  | sort | xargs cat > foo_3





share|improve this answer



















  • 2




    Try this command: man split cat md5sum
    – Kevin M
    Sep 4 '10 at 19:13






  • 5




    When assembling, I recommend cat foo.{000..NNN} where NNN is the last expected piece. That way you get an error message if one of the pieces is missing. But note that -d to get numeric suffixes is specific to GNU split; on other platforms you have to make do with foo.aaa, foo.aab, etc.
    – Gilles
    Oct 17 '10 at 11:16






  • 1




    And bear in mind that, for split, KB = 1000, K = 1024, MB = 1000*1000, M = 1024*1024 etc.
    – Zorawar
    Nov 29 '12 at 18:05










  • Shouldn't this ... cat > foo_3 be ... cat >>foo_3?
    – alk
    Jul 8 '15 at 12:10








  • 1




    If you decide to ease pain by using a utility. rar and 7zip are often used in making such splits easier to reassemble cross-platform
    – infixed
    Jun 3 '16 at 18:05













up vote
71
down vote



accepted







up vote
71
down vote



accepted






You can use split and cat.



For example something like



$ split --bytes 500M --numeric-suffixes --suffix-length=3 foo foo.


(where the input filename is foo and the last argument is the output prefix). This will create files like foo.000 foo.001 ...



The same command with short options:



$ split -b 100k -d -a 3 foo foo


You can also specify "--line-bytes" if you wish it to split on line boundaries instead of just exact number of bytes.



For re-assembling the generated pieces again you can use e.g.:



$ cat foo.* > foo_2


(assuming that the shell sorts the results of shell globbing - and the number of parts does not exceed the system dependent limit of arguments)



You can compare the result via:



$ cmp foo foo_2

$ echo $?


(which should output 0)



Alternatively, you can use a combination of find/sort/xargs to re-assemble the pieces:



$ find -maxdepth 1 -type f -name 'foo.*'  | sort | xargs cat > foo_3





share|improve this answer














You can use split and cat.



For example something like



$ split --bytes 500M --numeric-suffixes --suffix-length=3 foo foo.


(where the input filename is foo and the last argument is the output prefix). This will create files like foo.000 foo.001 ...



The same command with short options:



$ split -b 100k -d -a 3 foo foo


You can also specify "--line-bytes" if you wish it to split on line boundaries instead of just exact number of bytes.



For re-assembling the generated pieces again you can use e.g.:



$ cat foo.* > foo_2


(assuming that the shell sorts the results of shell globbing - and the number of parts does not exceed the system dependent limit of arguments)



You can compare the result via:



$ cmp foo foo_2

$ echo $?


(which should output 0)



Alternatively, you can use a combination of find/sort/xargs to re-assemble the pieces:



$ find -maxdepth 1 -type f -name 'foo.*'  | sort | xargs cat > foo_3






share|improve this answer














share|improve this answer



share|improve this answer








edited Nov 27 at 6:29









rogerdpack

3231312




3231312










answered Sep 4 '10 at 19:00









maxschlepzig

33.1k32135208




33.1k32135208








  • 2




    Try this command: man split cat md5sum
    – Kevin M
    Sep 4 '10 at 19:13






  • 5




    When assembling, I recommend cat foo.{000..NNN} where NNN is the last expected piece. That way you get an error message if one of the pieces is missing. But note that -d to get numeric suffixes is specific to GNU split; on other platforms you have to make do with foo.aaa, foo.aab, etc.
    – Gilles
    Oct 17 '10 at 11:16






  • 1




    And bear in mind that, for split, KB = 1000, K = 1024, MB = 1000*1000, M = 1024*1024 etc.
    – Zorawar
    Nov 29 '12 at 18:05










  • Shouldn't this ... cat > foo_3 be ... cat >>foo_3?
    – alk
    Jul 8 '15 at 12:10








  • 1




    If you decide to ease pain by using a utility. rar and 7zip are often used in making such splits easier to reassemble cross-platform
    – infixed
    Jun 3 '16 at 18:05














  • 2




    Try this command: man split cat md5sum
    – Kevin M
    Sep 4 '10 at 19:13






  • 5




    When assembling, I recommend cat foo.{000..NNN} where NNN is the last expected piece. That way you get an error message if one of the pieces is missing. But note that -d to get numeric suffixes is specific to GNU split; on other platforms you have to make do with foo.aaa, foo.aab, etc.
    – Gilles
    Oct 17 '10 at 11:16






  • 1




    And bear in mind that, for split, KB = 1000, K = 1024, MB = 1000*1000, M = 1024*1024 etc.
    – Zorawar
    Nov 29 '12 at 18:05










  • Shouldn't this ... cat > foo_3 be ... cat >>foo_3?
    – alk
    Jul 8 '15 at 12:10








  • 1




    If you decide to ease pain by using a utility. rar and 7zip are often used in making such splits easier to reassemble cross-platform
    – infixed
    Jun 3 '16 at 18:05








2




2




Try this command: man split cat md5sum
– Kevin M
Sep 4 '10 at 19:13




Try this command: man split cat md5sum
– Kevin M
Sep 4 '10 at 19:13




5




5




When assembling, I recommend cat foo.{000..NNN} where NNN is the last expected piece. That way you get an error message if one of the pieces is missing. But note that -d to get numeric suffixes is specific to GNU split; on other platforms you have to make do with foo.aaa, foo.aab, etc.
– Gilles
Oct 17 '10 at 11:16




When assembling, I recommend cat foo.{000..NNN} where NNN is the last expected piece. That way you get an error message if one of the pieces is missing. But note that -d to get numeric suffixes is specific to GNU split; on other platforms you have to make do with foo.aaa, foo.aab, etc.
– Gilles
Oct 17 '10 at 11:16




1




1




And bear in mind that, for split, KB = 1000, K = 1024, MB = 1000*1000, M = 1024*1024 etc.
– Zorawar
Nov 29 '12 at 18:05




And bear in mind that, for split, KB = 1000, K = 1024, MB = 1000*1000, M = 1024*1024 etc.
– Zorawar
Nov 29 '12 at 18:05












Shouldn't this ... cat > foo_3 be ... cat >>foo_3?
– alk
Jul 8 '15 at 12:10






Shouldn't this ... cat > foo_3 be ... cat >>foo_3?
– alk
Jul 8 '15 at 12:10






1




1




If you decide to ease pain by using a utility. rar and 7zip are often used in making such splits easier to reassemble cross-platform
– infixed
Jun 3 '16 at 18:05




If you decide to ease pain by using a utility. rar and 7zip are often used in making such splits easier to reassemble cross-platform
– infixed
Jun 3 '16 at 18:05












up vote
4
down vote













You can also do this with Archive Manager if you prefer a GUI. Look under 'Save->Other Options->Split into volumes of'.






share|improve this answer

















  • 5




    i tagged it 'command-line', but thanks for the answer :)
    – Stefan
    Sep 5 '10 at 19:25















up vote
4
down vote













You can also do this with Archive Manager if you prefer a GUI. Look under 'Save->Other Options->Split into volumes of'.






share|improve this answer

















  • 5




    i tagged it 'command-line', but thanks for the answer :)
    – Stefan
    Sep 5 '10 at 19:25













up vote
4
down vote










up vote
4
down vote









You can also do this with Archive Manager if you prefer a GUI. Look under 'Save->Other Options->Split into volumes of'.






share|improve this answer












You can also do this with Archive Manager if you prefer a GUI. Look under 'Save->Other Options->Split into volumes of'.







share|improve this answer












share|improve this answer



share|improve this answer










answered Sep 5 '10 at 13:44







user1498















  • 5




    i tagged it 'command-line', but thanks for the answer :)
    – Stefan
    Sep 5 '10 at 19:25














  • 5




    i tagged it 'command-line', but thanks for the answer :)
    – Stefan
    Sep 5 '10 at 19:25








5




5




i tagged it 'command-line', but thanks for the answer :)
– Stefan
Sep 5 '10 at 19:25




i tagged it 'command-line', but thanks for the answer :)
– Stefan
Sep 5 '10 at 19:25


















draft saved

draft discarded




















































Thanks for contributing an answer to Unix & Linux Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f1588%2fbreak-a-large-file-into-smaller-pieces%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

Accessing regular linux commands in Huawei's Dopra Linux

Image
4 Backstory: I have a Huawei HG8245 router in my house and I want to change the default username/password. I've tried following this guide but the config file always gets reset. My ISP isn't helping either so I'm looking for other ways to accomplish this. The router has telnet access so this seems like a way to change the credentials. The first shell I gain access to looks like this: WAP> Pressing "?" gives me a list of available commands which I've pasted here, the two interesting ones are "su" and "shell". I can't seem to get su access, even after logging in as root (challenge password prompt, and then nothing happens), but shell gives me a proper linux shell. WAP>shell BusyBox v1.18.4 (2017-12-26 17:06:34 CST) built-in shell (ash) Enter 'hel...
Read more

Can't connect RFCOMM socket: Host is down

Image
up vote 3 down vote favorite I have bluez 5.4.3 and I am attempting to communicate with a ble device through minicom. These two deceives (my computer and a bluno nano) are alreadt paired and this has been my method so far. #bluetoothctl # pair xx:xx:xx:xx:xx:xx # exit Everything is working well so far then... #sudo rfcomm connect /dev/rfcomm0 xx:xx:xx:xx:xx:xx 1 & [1] 3665 Can't connect RFCOMM socket: Host is down Does anyone know how to resolve this issue? Any help would be very much appreciated. bluetooth bluez share | improve this question asked Feb 9 '17 at 18:46 Brandon Knape 46 2 ...
Read more

Kernel panic - not syncing: Fatal Exception in Interrupt

Image
0 Every day this error occured on my pc, kernel panic error on interrupt, what will be the problem? linux-mint share asked 6 mins ago Vinod Selvin Vinod Selvin 1 1 New contributor Vinod Selvin is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct. add a comment ...
Read more