ChessUSA.com TalkChess.com
Hosted by Your Move Chess & Games
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

looking for a tool to conver line endings
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions Flat
View previous topic :: View next topic  
Author Message
Ignacio Garcia



Joined: 05 Jul 2010
Posts: 331
PostPost subject: Re: looking for a tool to conver line endings    Posted: Mon Mar 19, 2012 11:54 am Reply to topic Reply with quote
Sven Schüle wrote:
IGarcia wrote:
Sven Schüle wrote:
rvida wrote:
I decided to use the sed based solution. It is a wonderful tool,

Hi Richard,

take care of doing both steps (CR/LF->LF conversion + trailing whitespace removal) separately, starting with the CR/LF part. Depending on the platform where you perform these conversions, "sed" as well as "perl" or other tools using regular expressions may or may not recognize a CR/LF character sequence as something that matches a "$" (end of input line) in the given pattern. Therefore a pattern logically resembling "<whitespace><whitespace>*$" may or may not match an input line that ends with <whitespace><CR><LF>. You can expect it to succeed in a typical Windows-like environment where CR/LF is the typical text file line ending, but not in a typical UNIX environment. Furthermore, also combining "<whitespace><whitespace>*<CR><LF>" in one pattern will not always succeed since line endings could be inconsistent within one file.

The idea of $ matching end of line is to add portability. Will match end of line at run time OS independent. So your program dealing with some data before an end of line (data)$ will always find the data even if you run your script in a different operating system.

Please reread what I wrote:

- With "sed" on Windows, a "$" will match both "<CR><LF>" and "<LF>" line endings. (But surprisingly "sed" will write its output without the <CR> in the standard case, thus failing to preserve the "native" line ending mode of the OS it is running upon.)

- With "sed" on UNIX, a "$" will only match "<LF>" line endings, and will fail to replace anything when using a pattern like " *$" and the input lines have "<CR><LF>" endings.

That is the reason why an "sed" script using "$" is NOT portable for the task to remove trailing whitespace characters, it only works for input files with the "native" line endings. And that is further the reason why a portable solution (that runs on both UNIX and WinXX platforms without changes) does TWO steps: 1. CRLF=>LF, 2. remove trailing whitespace.

EDIT: Or uses "\r\n", which also works today (but didn't in some older "sed" versions I used).

Perl is different but not really much better for that purpose, in my opinion.

Sven


Sorry. I did not try to correct you. Only to say what is the idea about "$" and why sometimes fails, not only with sed.

Regards.
Back to top
View user's profile Send private message
Display posts from previous:   
Subject Author Date/Time
looking for a tool to conver line endings Richard Vida Sun Mar 18, 2012 2:28 pm
      Re: looking for a tool to conver line endings Horacio Montenegro Sun Mar 18, 2012 2:44 pm
      Re: looking for a tool to conver line endings Ignacio Garcia Sun Mar 18, 2012 3:16 pm
      Re: looking for a tool to conver line endings Lucas Braesch Sun Mar 18, 2012 3:30 pm
      Re: looking for a tool to conver line endings Robert Hyatt Sun Mar 18, 2012 4:03 pm
            Re: looking for a tool to conver line endings Ignacio Garcia Sun Mar 18, 2012 5:42 pm
                  Re: looking for a tool to conver line endings Robert Hyatt Mon Mar 19, 2012 3:43 am
      Re: looking for a tool to conver line endings Horacio Montenegro Sun Mar 18, 2012 5:08 pm
      Re: looking for a tool to conver line endings Richard Vida Sun Mar 18, 2012 6:29 pm
            Re: looking for a tool to conver line endings Ignacio Garcia Sun Mar 18, 2012 7:07 pm
            Re: looking for a tool to conver line endings Sven Schüle Sun Mar 18, 2012 11:10 pm
                  Re: looking for a tool to conver line endings Ignacio Garcia Mon Mar 19, 2012 12:37 am
                        Re: looking for a tool to conver line endings Horacio Montenegro Mon Mar 19, 2012 3:45 am
                              Re: looking for a tool to conver line endings Sven Schüle Mon Mar 19, 2012 10:37 am
                        Re: looking for a tool to conver line endings Robert Hyatt Mon Mar 19, 2012 3:47 am
                        Re: looking for a tool to conver line endings Sven Schüle Mon Mar 19, 2012 10:58 am
                              Re: looking for a tool to conver line endings Ignacio Garcia Mon Mar 19, 2012 11:54 am
            Re: looking for a tool to conver line endings Lucas Braesch Thu Mar 22, 2012 12:58 pm
                  Re: looking for a tool to conver line endings Don Dailey Thu Mar 22, 2012 2:15 pm
                        Re: looking for a tool to conver line endings Gary Thu Mar 22, 2012 3:25 pm
                              Re: looking for a tool to conver line endings Don Dailey Thu Mar 22, 2012 4:22 pm
      Re: looking for a tool to conver line endings Don Dailey Mon Mar 19, 2012 1:52 pm
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum




Powered by phpBB © 2001, 2005 phpBB Group
Enhanced with Moby Threads