[Solved] 0.19pre, SVG import failure

[fc_tofu]

OS: Windows 10 (10.0)
Word size of OS: 64-bit
Word size of FreeCAD: 64-bit
Version: 0.19.20310 (Git)
Build type: Release
Branch: master
Hash: 2cac4898d7d62276fbbaaef5b4cecb9bfdd29a33
Python version: 3.8.2
Qt version: 5.12.5
Coin version: 4.0.0
OCC version: 7.4.0
Locale: Chinese/China (zh_CN)
(Same as 0.18.4/win10x64)

Summary:
Some .svg files cannot be imported into FreeCAD, prompt error msg.

Reproduce:
1. "Menu >File >Import" error msg in Report View.

Code: Select all

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "E:\WINAPP_B\FreeCAD_0.19\Mod\Draft\importSVG.py", line 1776, in insert
    parser.parse(pythonopen(filename))
  File "E:\WINAPP_B\FreeCAD_0.19\bin\lib\xml\sax\expatreader.py", line 111, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "E:\WINAPP_B\FreeCAD_0.19\bin\lib\xml\sax\xmlreader.py", line 123, in parse
    buffer = file.read(self._bufsize)
<class 'UnicodeDecodeError'>: 'gbk' codec can't decode byte 0x93 in position 165: illegal multibyte sequence

2. "TechDrawWB >Import SVG Symbol" error msg in Report View.

Code: Select all

Traceback (most recent call last):
  File "<string>", line 1, in <module>
<class 'UnicodeDecodeError'>: 'gbk' codec can't decode byte 0x93 in position 165: illegal multibyte sequence
Traceback (most recent call last):
  File "<string>", line 1, in <module>
<class 'UnicodeDecodeError'>: 'gbk' codec can't decode byte 0x93 in position 287: illegal multibyte sequence

Example file
welding symbol: "[FreeCAD]\data\Mod\TechDraw\Symbols\Welding\GOST\edge-weld.svg"
drawing exported from TechDraw:

fc019pre_techdraw_export.svg

(62.68 KiB) Downloaded 21 times

ps.
Successful importing .svg examples:
1. [FreeCAD]\data\Mod\TechDraw\Symbols\Welding\AWS\beadDown.svg
2. [FreeCAD]\data\Mod\TechDraw\Templates\A4_LandscapeTD.svg
With "TechDrawWB >Import SVG Symbol" approach, prompt error msg, but grahpics importing succeed.

Code: Select all

DrawViewSymbol:execute - SVG for Symbol is not a valid document

Wth "Menu >File >Import" approach, no error but tedious msg prompt.

Code: Select all

processing element 1: svg
existing group transform: []
name: svg3085
done processing element 1
processing element 2: sodipodi:namedview
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: namedview983
done processing element 2
processing element 3: inkscape:grid
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: grid985
done processing element 3
processing element 4: sodipodi:guide
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: guide987
done processing element 4
processing element 5: sodipodi:guide
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: guide989
done processing element 5
processing element 6: defs
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: defs3087
done processing element 6
processing element 7: linearGradient
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: linearGradient3836-0
done processing element 7
processing element 8: stop
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: stop5008
done processing element 8
processing element 9: stop
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: stop5010
done processing element 9
processing element 10: inkscape:path-effect
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: path-effect3066
done processing element 10
processing element 11: linearGradient
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: linearGradient7597
done processing element 11
processing element 12: stop
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: stop7599
done processing element 12
processing element 13: stop
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: stop7601
done processing element 13
processing element 14: linearGradient
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: linearGradient3893
done processing element 14
processing element 15: stop
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: stop3895
done processing element 15
processing element 16: stop
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: stop3897
done processing element 16
processing element 17: linearGradient
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: linearGradient7603
done processing element 17
processing element 18: linearGradient
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: linearGradient3836-0-3
done processing element 18
processing element 19: stop
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: stop3838-2
done processing element 19
processing element 20: stop
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: stop3840-5
done processing element 20
processing element 21: metadata
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: metadata3090
done processing element 21
processing element 22: rdf:RDF
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
done processing element 22
processing element 23: cc:Work
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
done processing element 23
processing element 24: dc:format
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
done processing element 24
processing element 25: dc:type
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
done processing element 25
processing element 26: dc:title
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
done processing element 26
processing element 27: text
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: text3130
processing a text
done processing element 27
processing element 28: tspan
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: tspan3132
processing a text
done processing element 28
processing element 29: path
existing group transform: [Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))]
name: path1101
data: {'style': 'fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.88976383;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1', 'd': ['M', '3.9999992', '4.0000001', 'H', '60', 'C', '60', '20', '50', '32.224259', '31.77574', '32.224259', '13.999999', '32.224259', '3.9999992', '20', '3.9999992', '4.0000001', 'Z'], 'id': ['path1101'], 'inkscape:connector-curvature': ['0'], 'sodipodi:nodetypes': ['cccc'], 'fill': 'none', 'fill-rule': 'evenodd', 'stroke': '#000000', 'stroke-width': '1.88976383', 'stroke-linecap': 'butt', 'stroke-linejoin': 'miter', 'stroke-miterlimit': '4', 'stroke-dasharray': 'none', 'stroke-opacity': '1'}
move Vector (3.9999992, -4.0000001, 0.0)
applying group transform: Matrix ((0.264583,0,0,0),(0,0.264583,0,0),(0,0,1,0),(0,0,0,1))
done processing element 29
closing group

[wandererfan]

Some .svg files cannot be imported into FreeCAD, prompt error msg.

Some(all?) of the symbol files in the GOST directories do not have the file
encoding specified. Your system is probably trying to parse the file in its
default encoding scheme (gbk?) instead of UTF-8.

Try changing the very first line of one of the files (say edge-weld.svg) to this:

Code: Select all

<?xml version="1.0" encoding="UTF-8" standalone="no"?>

And let me know what happens.

With "TechDrawWB >Import SVG Symbol" approach, prompt error msg, but grahpics importing succeed.

Code: Select all

DrawViewSymbol:execute - SVG for Symbol is not a valid document

This one I know how to fix, but it can be ignored for now. It is trying to process the
symbol before the symbol has been set (ie empty symbol). Everything gets sorted out
in the second pass.

Wth "Menu >File >Import" approach, no error but tedious msg prompt.

I don't import Svg like this much, but I think these messages are always produced.

[fc_tofu]

Some(all?) of the symbol files in the GOST directories do not have the file
encoding specified. Your system is probably trying to parse the file in its
default encoding scheme (gbk?) instead of UTF-8.

More tests show,
All .svg files under "GOST" and its subdir, have same importing error.
All .svg files exported from TechDraw, have importing error.
All .svg files under "AWS" dir, have no importing error.

Try changing the very first line of one of the files (say edge-weld.svg) to this:

Code: Select all

<?xml version="1.0" encoding="UTF-8" standalone="no"?>

I tested your method, but no help. Same error msg and failure.
My finding is,
1. all .svg files under "GOST", have abnormal character in line 4.

Code: Select all

<title>Торцевой шов. Edge Weld – tech drawing chars (FreeCAD) https://freecad-gost.ru</title>

The dash and its following space "– " ("E2 80 93 20" in hex) was misunderstood as GBK. (GBK is an obsolete Chinese character coding scheme, but has been abandoned from Win 10. Win 10 use UTF-16 internally, which has nothing to do with this error.)
The dash hex value is "E2 80 93", while a normal dash is "2D". The following space is normal "20" hex.
Workaround is 1) remove space; and/or 2)replace dash to normal one.
2. all .svg files exported from TechDraw are UTF-BOM format. ("EF BB BF" as file head.)
Converting them from UTF-BOM to UTF can help importing succeed.

So, my suggeston is to fix the SVG importer.
1. ignore illegal characters in "<title>" block, for that is a very arbitary area.
2. add support to UTF-BOM format

I don't import Svg like this much, but I think these messages are always produced.

Tedious msg output may be the normal way of SVG importer. Just a little terrific.
If possible, shorter msg would be enough.

Thank you.

[wandererfan]

With "TechDrawWB >Import SVG Symbol" approach, prompt error msg, but grahpics importing succeed.

Code: Select all

DrawViewSymbol:execute - SVG for Symbol is not a valid document

this one should be fixed by git commit ba6e870bb6

[fc_tofu]

this one should be fixed by git commit ba6e870bb6

Thank you.

[wandererfan]

1. all .svg files under "GOST", have abnormal character in line 4.

all occurrences of the abnormal character have been changed to "- " by this git commit ba5d2d1238.

2. all .svg files exported from TechDraw are UTF-BOM format. ("EF BB BF" as file head.)
Converting them from UTF-BOM to UTF can help importing succeed.

I understand that some platforms require Byte Order Mark to read the file properly. Is this not true?

So, my suggeston is to fix the SVG importer.
1. ignore illegal characters in "<title>" block, for that is a very arbitary area.
2. add support to UTF-BOM format
<<snip>>
Tedious msg output may be the normal way of SVG importer. Just a little terrific.
If possible, shorter msg would be enough.

I don't know much about the SVG importer. But I know somebody who knows it well:

ping

[fc_tofu]

I understand that some platforms require Byte Order Mark to read the file properly. Is this not true?

I just tested FC 0.18.4 /Win10 x64.

1. .svg exported from TechDraw is UTF-8 format (no BOM).
2. importing .svg of UTF-BOM format cause same error as 0.19pre.

So the UTF-BOM export ability was introduced since 0.19pre/Win.
I don't know the situation on other OS.
Can you test it on your Linux machine, whether 0.18/0.19 export .svg differently?

[wandererfan]

So the UTF-BOM export ability was introduced since 0.19pre/Win.

BOM added commit c6d1684e2a 2019-10-05

Can you test it on your Linux machine, whether 0.18/0.19 export .svg differently?

I have no difference in behaviour in v0.19 with or without BOM.

But, according to this: https://en.wikipedia.org/wiki/Byte_order_mark#UTF-8
BOM is not relevant to UTF8, so we should turn this off.

OS: Linux Mint 19 (X-Cinnamon/cinnamon)
Word size of OS: 64-bit
Word size of FreeCAD: 64-bit
Version: 0.19.20364 +5 (Git)
Build type: debug
Branch: detailDlg
Hash: 8ef0777ca1ced42db61dc74e6dcc4a0ef34ea7a4
Python version: 3.6.9
Qt version: 5.9.5
Coin version: 4.0.0a
OCC version: 7.3.0
Locale: English/Canada (en_CA)