TalkChess.com :: View topic - looking for a tool to conver line endings

TalkChess.com
Hosted by Your Move Chess & Games

FAQ

Memberlist

Usergroups

Profile

looking for a tool to conver line endings

TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions

View previous topic :: View next topic

Author

Message

Robert Hyatt

Joined: 27 Feb 2006
Posts: 15866
Location: Birmingham, AL

Post subject: Re: looking for a tool to conver line endings Posted: Mon Mar 19, 2012 3:47 am

IGarcia wrote:

Sven Schüle wrote:

rvida wrote:

I decided to use the sed based solution. It is a wonderful tool,

Hi Richard,

take care of doing both steps (CR/LF->LF conversion + trailing whitespace removal) separately, starting with the CR/LF part. Depending on the platform where you perform these conversions, "sed" as well as "perl" or other tools using regular expressions may or may not recognize a CR/LF character sequence as something that matches a "$" (end of input line) in the given pattern. Therefore a pattern logically resembling "<whitespace><whitespace>*$" may or may not match an input line that ends with <whitespace><CR><LF>. You can expect it to succeed in a typical Windows-like environment where CR/LF is the typical text file line ending, but not in a typical UNIX environment. Furthermore, also combining "<whitespace><whitespace>*<CR><LF>" in one pattern will not always succeed since line endings could be inconsistent within one file.

The idea of $ matching end of line is to add portability. Will match end of line at run time OS independent. So your program dealing with some data before an end of line (data)$ will always find the data even if you run your script in a different operating system.

The main problem is when you write a regular expression using $ ( end of line match) when you really want to match only one character (CR or LF). Then the code will probably fail in other OS.

In this case, the problem proposed by Vida, you are looking for specific combination of space, tabs, CR and LF. Here its ok to not use $.

Your post, wich is valid and importatnt to be aware of this details, make me think the command I posted will not work if the input has mixed data (some lines with CR+LF, other only LF). This is solved making optional the CR match. So the version 3 ( Evil or Very Mad

) is now:

Code:

perl -pe 's/\s*\r*\n/\n/' in > out

The code will not work for mac text input, because still request at least one \n. The most accurate is the perl script posted by Horacio.

Sven wrote:

Hmmm ... wasn't it an invention from the DOS world to have that CR/LF line ending that created one of the biggest (in)compatibility issues in the whole IT world? Smile

Sven

For a real printer the DOS solution is the more logical and gives more control, because you have the option to move carriage, returning to column 0 and optional feeding, This allow to overwrite by returning carriage without feeding. A nonsense for a real printer because overwrites all, but useful if you write to screen and you don't like to scroll.

Ignacio

The CR/LF IS the end-of-line sentinel (at least in Dos/Windows text files, in unix a single LF (0x0a character) marks end of line in unix and most other systems). There are no "lines" in a file at all, just a continuous stream of bytes. The system simply interprets some character (or pair of characters in dos/windows) as an end-of-line marker. One has to be careful with "$" because some will interpret $ to be simply the N/L character. Searching for spaces while the C/R character is there will cause it to not match.

It is a real headache, to say the least...

Display posts from previous:

Subject	Author	Date/Time
looking for a tool to conver line endings	Richard Vida	Sun Mar 18, 2012 2:28 pm
Re: looking for a tool to conver line endings	Horacio Montenegro	Sun Mar 18, 2012 2:44 pm
Re: looking for a tool to conver line endings	Ignacio Garcia	Sun Mar 18, 2012 3:16 pm
Re: looking for a tool to conver line endings	Lucas Braesch	Sun Mar 18, 2012 3:30 pm
Re: looking for a tool to conver line endings	Robert Hyatt	Sun Mar 18, 2012 4:03 pm
Re: looking for a tool to conver line endings	Ignacio Garcia	Sun Mar 18, 2012 5:42 pm
Re: looking for a tool to conver line endings	Robert Hyatt	Mon Mar 19, 2012 3:43 am
Re: looking for a tool to conver line endings	Horacio Montenegro	Sun Mar 18, 2012 5:08 pm
Re: looking for a tool to conver line endings	Richard Vida	Sun Mar 18, 2012 6:29 pm
Re: looking for a tool to conver line endings	Ignacio Garcia	Sun Mar 18, 2012 7:07 pm
Re: looking for a tool to conver line endings	Sven Schüle	Sun Mar 18, 2012 11:10 pm
Re: looking for a tool to conver line endings	Ignacio Garcia	Mon Mar 19, 2012 12:37 am
Re: looking for a tool to conver line endings	Horacio Montenegro	Mon Mar 19, 2012 3:45 am
Re: looking for a tool to conver line endings	Sven Schüle	Mon Mar 19, 2012 10:37 am
Re: looking for a tool to conver line endings	Robert Hyatt	Mon Mar 19, 2012 3:47 am
Re: looking for a tool to conver line endings	Sven Schüle	Mon Mar 19, 2012 10:58 am
Re: looking for a tool to conver line endings	Ignacio Garcia	Mon Mar 19, 2012 11:54 am
Re: looking for a tool to conver line endings	Lucas Braesch	Thu Mar 22, 2012 12:58 pm
Re: looking for a tool to conver line endings	Don Dailey	Thu Mar 22, 2012 2:15 pm
Re: looking for a tool to conver line endings	Gary	Thu Mar 22, 2012 3:25 pm
Re: looking for a tool to conver line endings	Don Dailey	Thu Mar 22, 2012 4:22 pm
Re: looking for a tool to conver line endings	Don Dailey	Mon Mar 19, 2012 1:52 pm

TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum