Find duplicate words in text file

This thread is locked. You can follow the question or vote as helpful, but you cannot reply to this thread. I have the same question Report abuse.

Details required :. Cancel Submit. How satisfied are you with this reply? Thanks for your feedback, it helps us improve the site. Graham Mayor MVP. Click the 'Find' Tab. Click the Reading Highlight dropdown and select Highlight All. All the matching words will be highlighted. Dear Doug Robbins,. One of most common typos is to repeat the same word twice, as as here. I need an automatic procedure to remove all the repeated words in a text file.

This should not be a strange feature for a modern editor or spell-checker, for example I remember that MS Word introduced this feature several years ago!

Apparently, the default spell-check on my OS hun-spell can't do this, as it only finds words not in the dictionary. It sounds like something like this is what you want using any awk in any shell on every UNIX box :.

Bear in mind though - a lot of pattern matching is line oriented, so you've got to be careful if you cross line boundaries. If you can exclude that case, then you've got an easier job because you can parse one line at a time. I'm not doing that, so you'll end up reading the whole file into memory.

Sign up to join this community. The best answers are voted up and rise to the top. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams?

Learn more. Find repeated words in a text Ask Question. Asked 7 years, 1 month ago. Active 1 year, 11 months ago. Viewed 13k times. Improve this question. Burgi 6, 14 14 gold badges 39 39 silver badges 50 50 bronze badges. Is perl an acceptable alternative to bash? Otherwise, please be thoughtful, detailed and courteous, and adhere to our posting rules.

Edit Preview. H1 H2. Post Reply. Broken Link. Go Back to the Post Continue Anyways. Share Post. Permanent Link.