Refused to open font files with non-ascii names - Cannot marshal: Encountered unmappable character. at SharpFont.FT.FT_New_Face(IntPtr library, String filepathname, Int32 face_index, IntPtr& aface) #89

HinTak · 2016-07-11T17:12:38Z

There is no need to restrict to ascii file names?

Cannot marshal: Encountered unmappable character.   at SharpFont.FT.FT_New_Face(IntPtr library, String filepathname, Int32 face_index, IntPtr& aface)

I see it is set in the DllImport marshalling to

CharSet = CharSet.Ansi, BestFitMapping = false, ThrowOnUnmappableChar = true

Here is the list of routines with the above set:

        internal static extern Error FT_New_Face(IntPtr library, string filepathname, int face_index, out IntPtr aface);
        internal static extern Error FT_Attach_File(IntPtr face, string filepathname);
        internal static extern Error FT_GetFile_From_Mac_Name(string fontName, out IntPtr pathSpec, out int face_index);
        internal static extern Error FT_GetFile_From_Mac_ATS_Name(string fontName, out IntPtr pathSpec, out int face_index);
        internal static extern Error FT_GetFilePath_From_Mac_ATS_Name(string fontName, IntPtr path, int maxPathSize, out int face_index);
        internal static extern Error FT_Get_BDF_Charset_ID(IntPtr face, out string acharset_encoding, out string acharset_registry);
        internal static extern Error FT_Get_BDF_Property(IntPtr face, string prop_name, out IntPtr aproperty);
        internal static extern Error FT_Get_CID_Registry_Ordering_Supplement(IntPtr face, out string registry, out string ordering, out int aproperty);
        internal static extern IntPtr FT_Get_Module(IntPtr library, string module_name);
        internal static extern Error FT_Property_Set(IntPtr library, string module_name, string property_name, IntPtr value);
        internal static extern Error FT_Property_Get(IntPtr library, string module_name, string property_name, out IntPtr value);
        internal static extern Error FT_Property_Get(IntPtr library, string module_name, string property_name, ref int value);

PropertyGet/Set never have non-ascii arguments, but file names and font names do!

HinTak · 2016-07-11T17:36:18Z

Argh, this is a more difficult issue than I thought. Mono apparently treats CharSet.Ansi as utf8, which is why I never saw it until now.

I am seeing it because I am running SharpFont under wine with the genuine MS dotnet runtime at the moment (it is just a special thing that I need to do now, instead of using native mono). The behavior with CharSet.Ansi is as documented but undesirable - freetype itself on windows can open font files with non-ascii file name or font names, just treat them as null terminated c strings. Using CharSet.Ansi is uncessarily restrictive, but CharSet.Unicode probably won't work on windows, since freetype does not really expect arguments to come in as UTF16LE's (I assume that's what is meant by marshalling as "CharSet.Unicode").

Robmaister · 2016-12-07T19:02:16Z

Yikes, this looks like a pretty complex one. I suppose we could try to manually encode UTF-8 and have the extern functions take IntPtrs instead. Does FreeType expect UTF-8 for those APIs on all platforms?

HinTak · 2016-12-08T03:07:53Z

Freetype treats the input as raw bytes, does not really apply any encoding to it - as long as it is null-terminated c byte arrays. I can't see a quick way fixing this. One possibility is you open the file in c# and pass the file content as a memory block.

Robmaister · 2016-12-08T17:47:54Z

There are already Face constructors that do this for FT_New_Memory_Face - right now two constructors:

public Face(Library library, byte[] file, int faceIndex)
public Face(Library library, IntPtr bufferPtr, int length, int faceIndex)

I can also add an extra overload that takes some type of stream (FileStream? MemoryStream?) and reads all bytes, chaining to the byte[] constructor.

Another thing we can do is adapt a C# stream to FTStream so that it's easy to call Library.OpenFace.

HinTak · 2017-01-12T01:43:52Z

I guess what I like, is for public Face(Library library, string path) and public Face(Library library, string path, int faceIndex) to open the file name on the C# side and use FT_New_Memory_Face instead of calling
FT.FT_New_Face with a non-ascii argument (and fail), and/or optionally doing the former if the filename is non-ascii.

HinTak · 2017-07-14T13:31:11Z

Back-ported streaming support
HinTak@ff46f01
and
HinTak@c574260
and modified my caller to new Face(library, stream, faceindex, false).

Running on English name was successful, I have not tried non-ascii name yet.

HinTak · 2017-07-16T16:23:46Z

I'll be testing the backport soon, so assuming that's successful, this can be closed.


          SharpFont, 4.0.1+fixes, v3.0.1-81-gc574260, TargetFrameworkVersion=v2…

….0 build net4 build of SharpFont, 4.0.1+fixes from HinTak/SharpFont:v3.0.1-75-ge1b444c cd Source/SharpFont && xbuild SharpFont.csproj git@github.com:HinTak/SharpFont.git v4.0.1+fixes v3.0.1-80-g670aa0b cd Source/SharpFont && xbuild /p:TargetFrameworkVersion=v2.0 /property:Configuration=Release SharpFont.csproj 4.1.0+fixes need to do without dll upstream addition 8efddb79a3d55bca01e5e80e8f3782e8f8c5fbab xbuild /p:TargetFrameworkVersion=v2.0 /property:Configuration=Release SharpFont.csproj ; v3.0.1-81-gc574260 Part of the fixes to microsoft#49 Robmaister/SharpFont#89 and is built for dotnet2: microsoft#42 See also: Robmaister/SharpFont#95 and also Robmaister/SharpFont#82

HinTak · 2018-09-06T16:03:36Z

The streaming support seems to lose the file handle once in a while - like one test out of 50... needs better looking at.

HinTak mentioned this issue May 8, 2017

Added streaming support to Face #112

Open

HinTak mentioned this issue Jul 14, 2017

SharpFont/FreeType backend does not like non-ascii filenames microsoft/Font-Validator#49

Open

HinTak closed this Jul 16, 2017

HinTak reopened this Sep 6, 2018

hugovk mentioned this issue Nov 28, 2018

ImageFont and non latin characters in a path to font python-pillow/Pillow#3145

Closed

Robmaister / SharpFont

Refused to open font files with non-ascii names - Cannot marshal: Encountered unmappable character. at SharpFont.FT.FT_New_Face(IntPtr library, String filepathname, Int32 face_index, IntPtr& aface) #89

Refused to open font files with non-ascii names - Cannot marshal: Encountered unmappable character. at SharpFont.FT.FT_New_Face(IntPtr library, String filepathname, Int32 face_index, IntPtr& aface) #89

HinTak commented Jul 11, 2016

HinTak commented Jul 11, 2016

Robmaister commented Dec 7, 2016

HinTak commented Dec 8, 2016 •

edited

Robmaister commented Dec 8, 2016

HinTak commented Jan 12, 2017

HinTak commented Jul 14, 2017

HinTak commented Jul 16, 2017

HinTak commented Sep 6, 2018

Robmaister / SharpFont

Join GitHub today

Refused to open font files with non-ascii names - Cannot marshal: Encountered unmappable character. at SharpFont.FT.FT_New_Face(IntPtr library, String filepathname, Int32 face_index, IntPtr&amp; aface) #89

Refused to open font files with non-ascii names - Cannot marshal: Encountered unmappable character. at SharpFont.FT.FT_New_Face(IntPtr library, String filepathname, Int32 face_index, IntPtr&amp; aface) #89

Comments

HinTak commented Jul 11, 2016

HinTak commented Jul 11, 2016

Robmaister commented Dec 7, 2016

HinTak commented Dec 8, 2016 • edited

Robmaister commented Dec 8, 2016

HinTak commented Jan 12, 2017

HinTak commented Jul 14, 2017

HinTak commented Jul 16, 2017

HinTak commented Sep 6, 2018

Essential cookies

Always active

Analytics cookies

Refused to open font files with non-ascii names - Cannot marshal: Encountered unmappable character. at SharpFont.FT.FT_New_Face(IntPtr library, String filepathname, Int32 face_index, IntPtr& aface) #89

Refused to open font files with non-ascii names - Cannot marshal: Encountered unmappable character. at SharpFont.FT.FT_New_Face(IntPtr library, String filepathname, Int32 face_index, IntPtr& aface) #89

HinTak commented Dec 8, 2016 •

edited