| View previous topic :: View next topic |
| Author |
Message |
Richard Vida
Joined: 16 Apr 2009
Posts: 467
Location: Slovakia, EU
|
 Posted: Sun Mar 18, 2012 2:28 pm Post subject: looking for a tool to conver line endings |
  |
|
Hi,
I am looking for a simple tool that can
1) convert Windows (CR+LF) line endings to Unix (LF)
2) trim trailing whitespace at the end of each line |
|
| Back to top |
|
 |
Horacio Montenegro
Joined: 03 Sep 2008
Posts: 33
|
| Posted: Sun Mar 18, 2012 2:44 pm Post subject: Re: looking for a tool to conver line endings |
|
|
To convert from dos to unix and vice-versa (and I think the package also includes the Mac text files, which if I am not mistaken is CR), there is dos2unix:
http://sourceforge.net/projects/dos2unix/
To trim space you could use a perl scrit - to do the format conversion as well, in fact. |
|
| Back to top |
|
|
Ignacio Garcia
Joined: 05 Jul 2010
Posts: 331
|
| Posted: Sun Mar 18, 2012 3:16 pm Post subject: Re: looking for a tool to conver line endings |
|
|
| rvida wrote: |
Hi,
I am looking for a simple tool that can
1) convert Windows (CR+LF) line endings to Unix (LF)
2) trim trailing whitespace at the end of each line |
The white space has to be before end-line, so inverting task order you can do both in a single step, run:
| Code: |
perl -pe 's/ *\r\n/\n/' inputfile
|
It will replace all (optional) white spaces followed by CR+LF by a single LF from a file to stdout.
Hope it helps.
Regards
Ignacio |
|
| Back to top |
|
|
Lucas Braesch
Joined: 31 May 2010
Posts: 1730
|
| Posted: Sun Mar 18, 2012 3:30 pm Post subject: Re: looking for a tool to conver line endings |
|
|
| rvida wrote: |
Hi,
I am looking for a simple tool that can
1) convert Windows (CR+LF) line endings to Unix (LF)
2) trim trailing whitespace at the end of each line |
Regular expressions can do it 
First remove all trailing spaces and tabs at the end of each line with sed:
| Code: |
| sed 's/[ \t]*$//' file.txt > file_trimmed.txt |
I'm not sure if sed will automatically convert CRLF into LF, but if it doesn't just pipe this into another sed
| Code: |
| sed 's/[ \t]*$//' file.txt | sed 's/\n/\n/' > file_trimmed.txt |
|
|
| Back to top |
|
|
Robert Hyatt
Joined: 27 Feb 2006
Posts: 15814
Location: Birmingham, AL
|
| Posted: Sun Mar 18, 2012 4:03 pm Post subject: Re: looking for a tool to conver line endings |
|
|
| rvida wrote: |
Hi,
I am looking for a simple tool that can
1) convert Windows (CR+LF) line endings to Unix (LF)
2) trim trailing whitespace at the end of each line |
Linux has always had dos2unix and unix2dos commands. That what you want? |
|
| Back to top |
|
|
Horacio Montenegro
Joined: 03 Sep 2008
Posts: 33
|
| Posted: Sun Mar 18, 2012 5:08 pm Post subject: Re: looking for a tool to conver line endings |
|
|
I am not fluent on perl one-liners, but now I am jealous of the previous posts, so here is a (untested) script to convert to / from any OS specific format:
| Code: |
#!/usr/bin/perl
open (fh_in, "<", $ARGV[0]) or die "could not open file $!";
open (fh_out, ">", $ARGV[1]) or die "could not open file $!";
while (<fh_in>)
{
$_ =~ s/\r//;
$_ =~ s/\n//;
$_ =~ s/\s+$//;
if ( $ARGV[2] =~ m/mac/i ) { $_ = $_."\r"; }
elsif ( $ARGV[2] =~ m/unix/i ) { $_ = $_."\n"; }
elsif ( $ARGV[2] =~ m/windows/i ) { $_ = $_."\r\n"; }
print fh_out;
}
close (fh_in);
close (fh_out);
|
just save this as whatever_you_like.pl, then make it executable:
| Code: |
| chmod +x whatever_you_like.pl |
and call it as:
| Code: |
| ./whatever_you_like.pl input_file output_file SYSTEM |
SYSTEM being any one choice of mac, unix or windows (case-insensitive). Again, untested, if does not work, just get back here and someone will point to any possible errors.
edit: this assumes you are developing on unix/mac, if not, you have to call the script as:
| Code: |
| perl whatever_you_like.pl input_file output_file SYSTEM |
|
|
| Back to top |
|
|
Ignacio Garcia
Joined: 05 Jul 2010
Posts: 331
|
| Posted: Sun Mar 18, 2012 5:42 pm Post subject: Re: looking for a tool to conver line endings |
|
|
| bob wrote: |
| rvida wrote: |
Hi,
I am looking for a simple tool that can
1) convert Windows (CR+LF) line endings to Unix (LF)
2) trim trailing whitespace at the end of each line |
Linux has always had dos2unix and unix2dos commands. That what you want? |
those programs are not installed by default and the other problem is not solved: you still have spaces before end-line.
@Horacio: Nice coding. 
My command misses tabs.. This will do the trik form command line
| Code: |
perl -pe 's/\s*\r\n/\n/' infile > outfile
|
|
|
| Back to top |
|
|
Richard Vida
Joined: 16 Apr 2009
Posts: 467
Location: Slovakia, EU
|
| Posted: Sun Mar 18, 2012 6:29 pm Post subject: Re: looking for a tool to conver line endings |
|
|
Thanks for all the answers.
I decided to use the sed based solution. It is a wonderful tool, although for people coming from Dos/Windows world the syntax is somewhat obscure...
Btw. after some googling I found a list of very useful sed one liners:
http://sed.sourceforge.net/sed1line.txt |
|
| Back to top |
|
|
Ignacio Garcia
Joined: 05 Jul 2010
Posts: 331
|
| Posted: Sun Mar 18, 2012 7:07 pm Post subject: Re: looking for a tool to conver line endings |
|
|
| rvida wrote: |
Thanks for all the answers.
I decided to use the sed based solution. It is a wonderful tool, although for people coming from Dos/Windows world the syntax is somewhat obscure...
Btw. after some googling I found a list of very useful sed one liners:
http://sed.sourceforge.net/sed1line.txt |
Sure, sed is a great.
There are little, but important, differences on how each program interprets regular expressions. Keeping track of those differences in memory is a mess so its common to stick to one of few programs.
Regular expression (regex) syntax is obscure but is all logic behind them, and crafting some regex can be great fun, as solving a chess problem! 
Ignacio. |
|
| Back to top |
|
|
Sven Schüle
Joined: 15 May 2008
Posts: 2242
Location: Berlin, Germany
|
| Posted: Sun Mar 18, 2012 11:10 pm Post subject: Re: looking for a tool to conver line endings |
|
|
| rvida wrote: |
| I decided to use the sed based solution. It is a wonderful tool, |
Hi Richard,
take care of doing both steps (CR/LF->LF conversion + trailing whitespace removal) separately, starting with the CR/LF part. Depending on the platform where you perform these conversions, "sed" as well as "perl" or other tools using regular expressions may or may not recognize a CR/LF character sequence as something that matches a "$" (end of input line) in the given pattern. Therefore a pattern logically resembling "<whitespace><whitespace>*$" may or may not match an input line that ends with <whitespace><CR><LF>. You can expect it to succeed in a typical Windows-like environment where CR/LF is the typical text file line ending, but not in a typical UNIX environment. Furthermore, also combining "<whitespace><whitespace>*<CR><LF>" in one pattern will not always succeed since line endings could be inconsistent within one file.
| rvida wrote: |
| although for people coming from Dos/Windows world the syntax is somewhat obscure... |
Hmmm ... wasn't it an invention from the DOS world to have that CR/LF line ending that created one of the biggest (in)compatibility issues in the whole IT world?
Sven |
|
| Back to top |
|
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
