Break a large file into smaller pieces

up vote
60
down vote

favorite

How do I break a large, +4GB file into smaller files of about 500MB each.

And how do I re-assemble them again to get the original file?

edited Feb 20 '11 at 19:13

Gilles

523k12610421575

asked Sep 4 '10 at 18:51

Stefan

11.4k3182123

2

text line-wise version: stackoverflow.com/questions/2016894/…
– Ciro Santilli 新疆改造中心六四事件法轮功
Apr 26 '16 at 12:22

Related: How to split larger files into smaller parts?
– kenorb
Oct 22 '17 at 13:40

add a comment |

up vote
60
down vote

favorite

How do I break a large, +4GB file into smaller files of about 500MB each.

And how do I re-assemble them again to get the original file?

edited Feb 20 '11 at 19:13

Gilles

523k12610421575

asked Sep 4 '10 at 18:51

Stefan

11.4k3182123

2

text line-wise version: stackoverflow.com/questions/2016894/…
– Ciro Santilli 新疆改造中心六四事件法轮功
Apr 26 '16 at 12:22

Related: How to split larger files into smaller parts?
– kenorb
Oct 22 '17 at 13:40

add a comment |

up vote
60
down vote

favorite

How do I break a large, +4GB file into smaller files of about 500MB each.

And how do I re-assemble them again to get the original file?

edited Feb 20 '11 at 19:13

Gilles

523k12610421575

asked Sep 4 '10 at 18:51

Stefan

11.4k3182123

How do I break a large, +4GB file into smaller files of about 500MB each.

And how do I re-assemble them again to get the original file?

command-line split

edited Feb 20 '11 at 19:13

Gilles

523k12610421575

asked Sep 4 '10 at 18:51

Stefan

11.4k3182123

edited Feb 20 '11 at 19:13

Gilles

523k12610421575

asked Sep 4 '10 at 18:51

Stefan

11.4k3182123

edited Feb 20 '11 at 19:13

Gilles

523k12610421575

edited Feb 20 '11 at 19:13

Gilles

523k12610421575

edited Feb 20 '11 at 19:13

Gilles

523k12610421575

asked Sep 4 '10 at 18:51

Stefan

11.4k3182123

asked Sep 4 '10 at 18:51

Stefan

11.4k3182123

asked Sep 4 '10 at 18:51

Stefan

11.4k3182123

2

text line-wise version: stackoverflow.com/questions/2016894/…
– Ciro Santilli 新疆改造中心六四事件法轮功
Apr 26 '16 at 12:22

Related: How to split larger files into smaller parts?
– kenorb
Oct 22 '17 at 13:40

add a comment |

2

text line-wise version: stackoverflow.com/questions/2016894/…
– Ciro Santilli 新疆改造中心六四事件法轮功
Apr 26 '16 at 12:22

Related: How to split larger files into smaller parts?
– kenorb
Oct 22 '17 at 13:40

text line-wise version: stackoverflow.com/questions/2016894/…
– Ciro Santilli 新疆改造中心六四事件法轮功
Apr 26 '16 at 12:22

Related: How to split larger files into smaller parts?
– kenorb
Oct 22 '17 at 13:40

add a comment |

2 Answers
2

active

oldest

votes

up vote
71
down vote

accepted

You can use split and cat.

For example something like

$ split --bytes 500M --numeric-suffixes --suffix-length=3 foo foo.

(where the input filename is foo and the last argument is the output prefix). This will create files like foo.000 foo.001 ...

The same command with short options:

$ split -b 100k -d -a 3 foo foo

You can also specify "--line-bytes" if you wish it to split on line boundaries instead of just exact number of bytes.

For re-assembling the generated pieces again you can use e.g.:

$ cat foo.* > foo_2

(assuming that the shell sorts the results of shell globbing - and the number of parts does not exceed the system dependent limit of arguments)

You can compare the result via:

$ cmp foo foo_2

$ echo $?

(which should output 0)

Alternatively, you can use a combination of find/sort/xargs to re-assemble the pieces:

$ find -maxdepth 1 -type f -name 'foo.*'  | sort | xargs cat > foo_3

edited Nov 27 at 6:29

rogerdpack

3231312

answered Sep 4 '10 at 19:00

maxschlepzig

33.1k32135208

2

Try this command: man split cat md5sum
– Kevin M
Sep 4 '10 at 19:13

5

When assembling, I recommend cat foo.{000..NNN} where NNN is the last expected piece. That way you get an error message if one of the pieces is missing. But note that -d to get numeric suffixes is specific to GNU split; on other platforms you have to make do with foo.aaa, foo.aab, etc.
– Gilles
Oct 17 '10 at 11:16

1

And bear in mind that, for split, KB = 1000, K = 1024, MB = 1000*1000, M = 1024*1024 etc.
– Zorawar
Nov 29 '12 at 18:05

Shouldn't this ... cat > foo_3 be ... cat >>foo_3?
– alk
Jul 8 '15 at 12:10

1

If you decide to ease pain by using a utility. rar and 7zip are often used in making such splits easier to reassemble cross-platform
– infixed
Jun 3 '16 at 18:05

|
show 2 more comments

up vote
4
down vote

You can also do this with Archive Manager if you prefer a GUI. Look under 'Save->Other Options->Split into volumes of'.

answered Sep 5 '10 at 13:44

user1498

5

i tagged it 'command-line', but thanks for the answer :)
– Stefan
Sep 5 '10 at 19:25

add a comment |

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f1588%2fbreak-a-large-file-into-smaller-pieces%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

2 Answers
2

active

oldest

votes

2 Answers
2

active

oldest

votes

up vote
71
down vote

accepted

You can use split and cat.

For example something like

$ split --bytes 500M --numeric-suffixes --suffix-length=3 foo foo.

(where the input filename is foo and the last argument is the output prefix). This will create files like foo.000 foo.001 ...

The same command with short options:

$ split -b 100k -d -a 3 foo foo

You can also specify "--line-bytes" if you wish it to split on line boundaries instead of just exact number of bytes.

For re-assembling the generated pieces again you can use e.g.:

$ cat foo.* > foo_2

(assuming that the shell sorts the results of shell globbing - and the number of parts does not exceed the system dependent limit of arguments)

You can compare the result via:

$ cmp foo foo_2

$ echo $?

(which should output 0)

Alternatively, you can use a combination of find/sort/xargs to re-assemble the pieces:

$ find -maxdepth 1 -type f -name 'foo.*'  | sort | xargs cat > foo_3

edited Nov 27 at 6:29

rogerdpack

3231312

answered Sep 4 '10 at 19:00

maxschlepzig

33.1k32135208

2

Try this command: man split cat md5sum
– Kevin M
Sep 4 '10 at 19:13

5

When assembling, I recommend cat foo.{000..NNN} where NNN is the last expected piece. That way you get an error message if one of the pieces is missing. But note that -d to get numeric suffixes is specific to GNU split; on other platforms you have to make do with foo.aaa, foo.aab, etc.
– Gilles
Oct 17 '10 at 11:16

1

And bear in mind that, for split, KB = 1000, K = 1024, MB = 1000*1000, M = 1024*1024 etc.
– Zorawar
Nov 29 '12 at 18:05

Shouldn't this ... cat > foo_3 be ... cat >>foo_3?
– alk
Jul 8 '15 at 12:10

1

If you decide to ease pain by using a utility. rar and 7zip are often used in making such splits easier to reassemble cross-platform
– infixed
Jun 3 '16 at 18:05

|
show 2 more comments

up vote
71
down vote

accepted

You can use split and cat.

For example something like

$ split --bytes 500M --numeric-suffixes --suffix-length=3 foo foo.

(where the input filename is foo and the last argument is the output prefix). This will create files like foo.000 foo.001 ...

The same command with short options:

$ split -b 100k -d -a 3 foo foo

You can also specify "--line-bytes" if you wish it to split on line boundaries instead of just exact number of bytes.

For re-assembling the generated pieces again you can use e.g.:

$ cat foo.* > foo_2

(assuming that the shell sorts the results of shell globbing - and the number of parts does not exceed the system dependent limit of arguments)

You can compare the result via:

$ cmp foo foo_2

$ echo $?

(which should output 0)

Alternatively, you can use a combination of find/sort/xargs to re-assemble the pieces:

$ find -maxdepth 1 -type f -name 'foo.*'  | sort | xargs cat > foo_3

edited Nov 27 at 6:29

rogerdpack

3231312

answered Sep 4 '10 at 19:00

maxschlepzig

33.1k32135208

2

Try this command: man split cat md5sum
– Kevin M
Sep 4 '10 at 19:13

5

When assembling, I recommend cat foo.{000..NNN} where NNN is the last expected piece. That way you get an error message if one of the pieces is missing. But note that -d to get numeric suffixes is specific to GNU split; on other platforms you have to make do with foo.aaa, foo.aab, etc.
– Gilles
Oct 17 '10 at 11:16

1

And bear in mind that, for split, KB = 1000, K = 1024, MB = 1000*1000, M = 1024*1024 etc.
– Zorawar
Nov 29 '12 at 18:05

Shouldn't this ... cat > foo_3 be ... cat >>foo_3?
– alk
Jul 8 '15 at 12:10

1

If you decide to ease pain by using a utility. rar and 7zip are often used in making such splits easier to reassemble cross-platform
– infixed
Jun 3 '16 at 18:05

|
show 2 more comments

up vote
71
down vote

accepted

You can use split and cat.

For example something like

$ split --bytes 500M --numeric-suffixes --suffix-length=3 foo foo.

(where the input filename is foo and the last argument is the output prefix). This will create files like foo.000 foo.001 ...

The same command with short options:

$ split -b 100k -d -a 3 foo foo

You can also specify "--line-bytes" if you wish it to split on line boundaries instead of just exact number of bytes.

For re-assembling the generated pieces again you can use e.g.:

$ cat foo.* > foo_2

(assuming that the shell sorts the results of shell globbing - and the number of parts does not exceed the system dependent limit of arguments)

You can compare the result via:

$ cmp foo foo_2

$ echo $?

(which should output 0)

Alternatively, you can use a combination of find/sort/xargs to re-assemble the pieces:

$ find -maxdepth 1 -type f -name 'foo.*'  | sort | xargs cat > foo_3

edited Nov 27 at 6:29

rogerdpack

3231312

answered Sep 4 '10 at 19:00

maxschlepzig

33.1k32135208

You can use split and cat.

For example something like

$ split --bytes 500M --numeric-suffixes --suffix-length=3 foo foo.

(where the input filename is foo and the last argument is the output prefix). This will create files like foo.000 foo.001 ...

The same command with short options:

$ split -b 100k -d -a 3 foo foo

You can also specify "--line-bytes" if you wish it to split on line boundaries instead of just exact number of bytes.

For re-assembling the generated pieces again you can use e.g.:

$ cat foo.* > foo_2

(assuming that the shell sorts the results of shell globbing - and the number of parts does not exceed the system dependent limit of arguments)

You can compare the result via:

$ cmp foo foo_2

$ echo $?

(which should output 0)

Alternatively, you can use a combination of find/sort/xargs to re-assemble the pieces:

$ find -maxdepth 1 -type f -name 'foo.*'  | sort | xargs cat > foo_3

edited Nov 27 at 6:29

rogerdpack

3231312

answered Sep 4 '10 at 19:00

maxschlepzig

33.1k32135208

edited Nov 27 at 6:29

rogerdpack

3231312

edited Nov 27 at 6:29

rogerdpack

3231312

edited Nov 27 at 6:29

rogerdpack

3231312

answered Sep 4 '10 at 19:00

maxschlepzig

33.1k32135208

answered Sep 4 '10 at 19:00

maxschlepzig

33.1k32135208

answered Sep 4 '10 at 19:00

maxschlepzig

33.1k32135208

2

Try this command: man split cat md5sum
– Kevin M
Sep 4 '10 at 19:13

5

When assembling, I recommend cat foo.{000..NNN} where NNN is the last expected piece. That way you get an error message if one of the pieces is missing. But note that -d to get numeric suffixes is specific to GNU split; on other platforms you have to make do with foo.aaa, foo.aab, etc.
– Gilles
Oct 17 '10 at 11:16

1

And bear in mind that, for split, KB = 1000, K = 1024, MB = 1000*1000, M = 1024*1024 etc.
– Zorawar
Nov 29 '12 at 18:05

Shouldn't this ... cat > foo_3 be ... cat >>foo_3?
– alk
Jul 8 '15 at 12:10

1

If you decide to ease pain by using a utility. rar and 7zip are often used in making such splits easier to reassemble cross-platform
– infixed
Jun 3 '16 at 18:05

|
show 2 more comments

2

Try this command: man split cat md5sum
– Kevin M
Sep 4 '10 at 19:13

5

When assembling, I recommend cat foo.{000..NNN} where NNN is the last expected piece. That way you get an error message if one of the pieces is missing. But note that -d to get numeric suffixes is specific to GNU split; on other platforms you have to make do with foo.aaa, foo.aab, etc.
– Gilles
Oct 17 '10 at 11:16

1

And bear in mind that, for split, KB = 1000, K = 1024, MB = 1000*1000, M = 1024*1024 etc.
– Zorawar
Nov 29 '12 at 18:05

Shouldn't this ... cat > foo_3 be ... cat >>foo_3?
– alk
Jul 8 '15 at 12:10

1

If you decide to ease pain by using a utility. rar and 7zip are often used in making such splits easier to reassemble cross-platform
– infixed
Jun 3 '16 at 18:05

Try this command: man split cat md5sum
– Kevin M
Sep 4 '10 at 19:13

When assembling, I recommend cat foo.{000..NNN} where NNN is the last expected piece. That way you get an error message if one of the pieces is missing. But note that -d to get numeric suffixes is specific to GNU split; on other platforms you have to make do with foo.aaa, foo.aab, etc.
– Gilles
Oct 17 '10 at 11:16

And bear in mind that, for split, KB = 1000, K = 1024, MB = 1000*1000, M = 1024*1024 etc.
– Zorawar
Nov 29 '12 at 18:05

Shouldn't this ... cat > foo_3 be ... cat >>foo_3?
– alk
Jul 8 '15 at 12:10

If you decide to ease pain by using a utility. rar and 7zip are often used in making such splits easier to reassemble cross-platform
– infixed
Jun 3 '16 at 18:05

|
show 2 more comments

up vote
4
down vote

You can also do this with Archive Manager if you prefer a GUI. Look under 'Save->Other Options->Split into volumes of'.

answered Sep 5 '10 at 13:44

user1498

5

i tagged it 'command-line', but thanks for the answer :)
– Stefan
Sep 5 '10 at 19:25

add a comment |

up vote
4
down vote

You can also do this with Archive Manager if you prefer a GUI. Look under 'Save->Other Options->Split into volumes of'.

answered Sep 5 '10 at 13:44

user1498

5

i tagged it 'command-line', but thanks for the answer :)
– Stefan
Sep 5 '10 at 19:25

add a comment |

up vote
4
down vote

You can also do this with Archive Manager if you prefer a GUI. Look under 'Save->Other Options->Split into volumes of'.

answered Sep 5 '10 at 13:44

user1498

You can also do this with Archive Manager if you prefer a GUI. Look under 'Save->Other Options->Split into volumes of'.

answered Sep 5 '10 at 13:44

user1498

answered Sep 5 '10 at 13:44

user1498

answered Sep 5 '10 at 13:44

user1498

answered Sep 5 '10 at 13:44

user1498

5

i tagged it 'command-line', but thanks for the answer :)
– Stefan
Sep 5 '10 at 19:25

add a comment |

5

i tagged it 'command-line', but thanks for the answer :)
– Stefan
Sep 5 '10 at 19:25

i tagged it 'command-line', but thanks for the answer :)
– Stefan
Sep 5 '10 at 19:25

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Unix & Linux Stack Exchange!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

Some of your past answers have not been well-received, and you're in danger of being blocked from answering.

Please pay close attention to the following guidance:

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Sstrhsrtj