Ideas needed: Xref

[yorik]

I'm thinking about this for a bit of time, but so far I have no clear view on the best way to proceed, so I'd like to gather more ideas. Here is what I have thought so far:

1. In architecture or BIM workflow, same as in other workflows, we need to be able to divide the model into several files. In Autodesk apps, a file embedded into another is called an Xref, I'm using the word here while we have nothing better.
2. Differently to other workflows like Assembly/Part Design, Architecture/BIM models have a lot (hundreds) of objects. It is unrealistic to imagine we'll be able to always group these objects under one structure (body). In most cases, you need to keep those objects separated. Making for ex. one body for a whole building floor would be totally counter-productive (heavy recomputes needed all the time).
3. Since Assembly/PartDesign will rely a lot on the Body, it will probably not fit for Arch workflows. We'd better start thinking on something different.
4. Now is the real problem. How to do that efficiently?
5. We could import all the objects from the other (xref) document. The MergeDocument system allows to do that easily (just need to add a python binding). This would create, in the host document, faithful copies of all the objects from the xreffed document. We could then simply gather these objects under something so they are easy to retrieve/identify.
6. Problem: This would in fact create a full, identical copy of the xreffed document, removing most of the advantage of having separate files (the host file would be as heavy as if there was no xref)
7. Other problem: If the imported objects are accessible individually, other objects can link to them. How does it work, then, when the xreffed file changes, and we need to re-import all these objects. There would be heavy relinking needed, you can imagine the dire consequences, it could easily screw a good part of the model.
8. Other solution, a bit like autodesk apps do: The imported file would behave as a compound (one object only). This would solve the problem above, but would still make the model heavy. Also, to be able to create a compound with correct face colors, we would need to import the objects individually first, like the first solution above, then create the compound, then discard the above objects. The import would be slow.
9. Other solution, we don't rely on MergeDocument. We unzip the xreffed file, read all the breps, and assemble under a compound. This would be much faster (no individual document object created), but the compound would have one color for all faces. Not very practical
10. I remember a document where Jürgen talked about importing only the OpenInventor representation of a file. This could be fantastic! You could have incredibly heavy files merged together into a master document that would be very lightweight. However, there's the big problem: The openinventor representation is not stored in FreeCAD files, how would we then get/build it? We could of course add something in the xreffed document itself (an object that stores the openinventor of the whole scene) but that's not practical either. But this solution has problems too, no way, for example, to query shape information from the imported objects.

So I think the concept must evolve. Ideas welcome!

[DeepSOIC]

Once, I had an idea of being able to create feature objects outside of document. So that they function, they create inventor nodes, they are able to recompute, etc. , but are never shown to the user. They are only accessible via python, or via links.

I'm not sure if such possibility will help very much in x-refs. What I thought it could help with, is having an instance object with altered parameters. Also, having such objects would have been very useful for Lattice ParaSeries and TopoSeries, for example . (Now, ParaSeries creates a temporary document, copy stuff necessary to recompute a feature, alters parameters in that document, recomputes it, takes the shape and closes the document; this creates funny behavior when recomputing, and even causes crashes on Linux.)

[DeepSOIC]

Another thing that can be helpful, that a document can have a field on which objects are "public". Actually it already has such property: "Tip", which in its current form (PropertyLink?) is not very useful. Then, when x-ref'd, only "public" objects are to be loaded.

EDIT: and for "public" features, FreeCAD would additionally save inventor representation.

[Pauvres_honteux]

Hi, just some thoughts:

for people like me who thinks better around a picture:
- left; concrete slab with sewer, hot n'cold water, electrics.
- middle; concrete wall with bricks/tiles (got lacy and didn't model the mortar).
- right; wooden wall with plank cladding and nails (wasn't able to get linear pattern to work in PartDesign [0.17.10236] OpenSUSE)

Many_objects.png (123.73 KiB) Viewed 2647 times

Managing large number of objects perhaps can be handled by the following principles;

Many_objects_handling_principle.png (15.31 KiB) Viewed 2647 times

My thoughts revolve around the principle of Assembly(ies) AND secondary 3D-viewing ultra optimized files (didn't Jürgen start something in that area?) AND an additional twist of file header usage (see sound/image file headers for inspiration).
In order to load as little as possible into the memory I believe as much as possible should be saved to ZIP-files header (how much can it contain?) for simple access without opening/unzipping the file.
For viewing purposes an ultra lightweight file format must be used and saved as a separate file. To save even more precious GPU clock cycles the principle of rendering farther objects coarser should be used. I think the gaming industry uses this method?

When saving the file one is currently working on, FreeCAD also automagically creates a simplified representation and saves it in a separate folder.
When opening the humongous Assembly representation of a whole block/area one really want to see something but not stalling the computer and this is where the stored information in the headers come handy. From that sparse information one should be able to calculate really simple representations of the houses (a house is usually a block, i.e. coordinates for six points are needed). When zooming in the user can choose to switch on/read more information from the ultra light weight file. When user wants even more details he/she opens up/reads the full file. All while the "game engine" does its job with not loading more than absolutely necessary into memory. (This is possible to set in catia, both static levels and dynamic levels).
One thing catia does not have is memory handling, it doesn't unload everything from memory causing it to crash in the afternoon... And that is why I stress the need of it specifically taken account of in FreeCAD.

I.E. there will be three levels of details for different purposes. All three will be usable for viewing.
- 1:st level (file header) additionally gives the number of bricks, planks, nails and so on, without loading the file into memory!
This also gives the weight/volume/length of concrete, mortar and planks, still without loading the file into memory!
The project management can get all info they want without having a super computer at hand!!
- 2:nd level (ultra light weight) shall give possibilities to measure and do section cuts, but that's about it.
- 3:rd level (full format) gives, at least, what we have today.

When sharing the file(s) one need an extra container for that. An extra ZIP-file maybe? This container shall not be created by default, only when one explicitly wants to send them (the files) to someone outside the company server.

Are any of these ideas worth considering?

[NormandC]

Differently to other workflows like Assembly/Part Design, Architecture/BIM models have a lot (hundreds) of objects.

This statement is inaccurate.

In parametric MCAD, assemblies can have thousands of objects. Think of a car or an aeroplane. EDIT: or any of ppemawm's models!

In most of these programs, different filetypes exist to differentiate objects. We already discussed this in the past, and I certainly hope it gets implemented in FreeCAD at some point. A single file containing all the parts for an assembly project quickly becomes unmanageable.

Assemblies, parts, drawings have their own filetype. In some software, objects made from specialized modules like sheet metal tools even have their own filetype.

So the assembly (master) document is actually lightweight, because it calls up all those hundreds of external parts. That's how the Assembly2 workbench works by the way.

[bernd]

I have been thinking of this too, without any good idea ...

Just would like to tell how it is in Nemtschek Allplan which is able to handle huge models too,

The have some management system which loads different drawing (model part) files (I will call it file in Allplan in German it is called Teilbild). Means in one file are just some elements, may be the walls of one floor, the slabs of one floor, the excavation, an imported dwg, some pictures to include in the drawing, the material bills, the reinforcement, the furniture and so on and on. All the data you need for a project and all splitted on hundreds of files. You could have hundreds of such files in one project. Actually we have in most projects more than one hundred of these files filled with data. The management system loads these files. You use some tree view with checkboxes for loading a file in write mode or in view mode. The files have names but internally it is distinguished by numbers. You can see and work with the numbers. The maximum loadable files and the maximum space for one file is limited. Thus you have to split your project on different files. If you make sections (for paper drawings) you gone make them on new files too. They have a reference to all files the section has elements from. As more elements and sections as more references. The max number of references is limited too. We gone spent much time to stay inside all these limitations. But the advantage is the model and the sections keeps usable.

Attached the widget for the loading system. On the left side normally are all files with model data (the solids), on the right side normally are all with derivations from your modell (the sections, material bills etc.)
http://www.allplan-architektur.ch/_Reso ... ruktur.jpg

Totally independent from all this is some plot system to put the data from the files on a drawing paper. In these plot system you can not edit the contence of the files, you can just move and place them on the paper.

bernd

EDIT: found an english screen: http://enews.scia.net/Images/building_structure.jpg

[Renato Rebelo]

For me this should be the most interesting solution because it is the most flexible, although it can be the slowest ...

[*] Other solution, a bit like autodesk apps do: The imported file would behave as a compound (one object only). This would solve the problem above, but would still make the model heavy. Also, to be able to create a compound with correct face colors, we would need to import the objects individually first, like the first solution above, then create the compound, then discard the above objects. The import would be slow.

greetings,
Renato Rebelo

[yorik]

Thanks for all the feedback guys, I begin to get ideas:

- Although some (autodesk mainly) applications allow you to embed any file into any other, without the need to do anything special in the file that you plan to embed, others (solidworks, or the assembly/body system) require you to do something: put your stuff under one structure (body), or save under a special name. This all means you have to "prepare" your file somehow to be importable. This is actually not so terrible as it sounds (to me ), it's actually very little work. I think it would be totally tolerable.

- Seeing my list above, why not have all the options? Imagine a "xref" object, which has a file path property, a timestamp (to autoreload the file if it has changed), and an enum property that can be "compound", "one shape" or "representation". Compound would make the object a compound, and list all the individual objects of the embedded file under it. One shape would show one, big, one-color shape which is internally a compound of all the shapes of the embedded file, but doesn't let you access the individual shapes. The advantage would be that the import would not require any GUI operation (we would be just reading the individual brep files from the fcstd file) and therefore would be much faster. And finally, "representation" would read only an OpenInventor node from the fcstd file.

- So for that last mode to work, one would need to "prepare" the file to be embedded, by adding all the objects one wants to embed to a certain object (a kind of lightweight "body") which will store their openinventor representation in the file. If such preparation is not made, this last mode will not be available.

- We still have to test that, but the selection system is embedded into the coin representation. So it's possible that on importing the coin node, selection still works as if these objects were really existing in the document. But there is the problem of possible double names... Somthing to check on later.

- Of course there is still the problem that in the first (compound) mode, one could create links to the imported objects, and when updating the file, these objects could be gone. But we can minimize the problem by warning the user on file update if some link will become broken...

How does that sound?

[easyw-fc]

[*] Other solution, a bit like autodesk apps do: The imported file would behave as a compound (one object only). This would solve the problem above, but would still make the model heavy. Also, to be able to create a compound with correct face colors, we would need to import the objects individually first, like the first solution above, then create the compound, then discard the above objects. The import would be slow.

A similar philosophy is applied to STEP assembly...

here there is an example of STEP file with a main file embedding children files
The main file is light because has only the links to the children files
https://www.cax-if.de/library/
https://www.cax-if.de/library/s1-tu-203.zip

loading the step file in FC 0.17 is now loaded as hierarchy structure
ATM the saving with this hierarchy is missing

step-assembly.png (201.03 KiB) Viewed 2469 times

[Renato Rebelo]

It's more than fine for me ...

greetings,
Renato Rebelo