Remove unknown unicode character from textfiles - sed, other bash/shell methods

You should have a look using hexdump -C and find the bytes around it. Presuming UTF-8, what vi shows as <91> (decimal 145, a unicode point meaningless in text) would be two bytes, 0xc2 and 0x91.

It's implied your substitutions didn't work at all, but if what you did was just replace 0x91 with 0x27, you'll have invalidated the UTF-8 (the second byte of a two byte sequence always has the high bit set, i.e. is >= 0x80). This might complicate your analysis, although vi should then show it as ?'.

That said, I tested this and it works:

#!/usr/bin/perl
use strict;
use warnings FATAL => qw(all);

my $data = "";
my $file = $ARGV[0];

while (<>) {
    s/\xc2\x91/'/g;
    $data .= $_;
}

open my $out, '>', $file || die "Could not write $file.";
print $out $data;
close $out;  

If $ARGV[0] exists when <> is referenced, perl pops this off the argument stack and takes it as a filepath to use for input (I find short scripts easier to tweak and work with than one liners, BTW). This accumulates in memory (fine as long as the files are not massive), whereas perl -i renames the original file to avoid edit-in-place race conditions (see perldoc perlrun).

So you could use that:

  find . -name "*.txt" -exec whatever.pl {} +

If it is indeed the character U+0091 (0xc2 0x91 in UTF-8 encoding) and not the byte 0x91, then:

PERLIO=:utf8 perl -pi -e "s/\N{U+0091}/'/g" file

Would convert it to '.

With GNU sed:

sed -i "s/\xc2\x91/'/" file

Edit:

However, in your case, the file is not in UTF-8. UTF-8 characters are one byte, only for ASCII characters (for values 0 to 0x7F). The other characters are represented by two or more bytes whose value is greater than 0x7F. So a 0x91 byte, with no byte greater than 0x7F around it cannot be found in a utf-8 file.

More likely, your file is in a single-byte character set, most likely some Microsoft one like windows-1252.

In windows-1252, 0x91 is the left single quote character. The unicode equivalent is U+2018 which in UTF-8 is written 0xe2 0x80 0x98.

If you want to convert your file to UTF-8, best is probably to use a dedicated tool for that. Like:

recode windows-1252..utf8 < file

Or:

iconv -f windows-1252 -t utf-8 < file

Or if you want to do it for every filename.txt:

find . -type f -name filename.txt -exec sh -Cc '
  for file do
    mv "$file" "$file~" &&
      iconv -f windows-1252 -t utf-8 < "$file~" > "$file"
  done' sh {} +