Model hosting and one-click-upload

Have some feature requests, feedback, cool stuff to share, or want to know where FreeCAD is going? This is the place.
Forum rules
Be nice to others! Read the FreeCAD code of conduct!
jreinhardt
Posts: 329
Joined: Mon Sep 09, 2013 6:08 pm

Model hosting and one-click-upload

Postby jreinhardt » Mon Aug 11, 2014 12:45 pm

Hi everybody,

I really like the idea of the Thingtracker Network. I think it is a good idea to decentralize the publication and hosting of CAD models, in order to avoid walled gardens like Thingiverse.

I therefore started to write some Software to support this movement. Although still in an early stage, I now have a everything that is necessary to be useful and useable.

The Thingtracker network is built upon the concept of Trackers, which are JSON documents that contain metadata about models, like URLs, a description, tags, license information and so on. Trackers can also reference other Trackers.

A thingcollector is a web app whose job it is to know as many trackers as possible, and which provides full text search for all parts contained in these trackers. One can submit URLs of trackers to the collector to tell it about them. An instance of a thingcollector is running here. It currently knows about four trackers, currently containing only a few dozen models and "things".

A CADinet is a web app whose job it is to showcase models. The name is a portmanteau of Cabinet and CAD. One can publish models on a CADinet, at the moment only via a REST API. So a instance of a CADinet is a bit like a small Thingiverse. But in contrast to Thingiverse, a CADinet publishes a Tracker, which can be submitted to a thingcollector to make all models on the CADinet available in the ThingTrackerNetwork. An instance of a CADinet is running here.

Both the thingcollector and the CADinet are easily deployable in the OpenShift PaaS, i.e. it is very easy to run your own instances of these apps. It should also be rather easy to run them on your own server.

The last puzzle piece is CADinet-FreeCAD, a FreeCAD macro that uses the API provided by CADinet to easily publish models there. Parts of it are strongly inspired by the WebGL exporter of FreeCAD written by Yorik.

At the moment this is very much an experiment and has many rough edges, but I invite you to test it and give me feedback. Please don't trust it too much and don't use it on models that you don't want to be publicly available.

To get started, just follow the instructions given here from here. You can use the test instance of CADinet.

I believe this could be a very useful addition to the FreeCAD ecosystem. For example, it makes it very easy to share models, either for support purposes in the forum or for the user showcases. Or one could think about a distributed version of the FreeCAD starting parts library, where one can easily insert models from the Thingtracker Network.
BOLTS, standard parts library for FreeCAD Thread, Help with Translation
User avatar
shoogen
Posts: 2823
Joined: Thu Dec 01, 2011 5:24 pm

Re: Model hosting and one-click-upload

Postby shoogen » Sat Aug 16, 2014 6:06 pm

Excellent work Johannes,
i looked roughly into it. It looks really promising. I found some aspects that i would to bring up early on.
1. UID generation.
If i understood this correctly you use the UID from the FCStd file as the primary key. This might raise problems to distinguish between an updated version of the original file and a derivative. But to be fair. This is not a problem in your software.
This is a general problem in FreeCAD. There should be a way to generate a new UID. For example when using "Save As..."
But AFIAK the UID is not widly used inside FreeCAD, unless someone opens both versions of the same document simultaneously. But in your current model an upload by a second person would fail, because the server would reject to update the original file.
2. tagging / repository structure
The idea of one-click sharing looks appealing. But the main problem is that only data that is present in the FreeCAD document can be uploaded. One example would be, that nobody cares to the the document description.
Usually creating a meaningful CAD model takes hours of work. IMHO the benefit of one-click uploading is neglectable compared to the effort needed to create the model and to add rich and useful metadata. In the long run I would rather use a dialog, that would help me to enter those metadata and include them in the FCStd as well as in the remote repository.
The editor doesn't have to be perfect from the start. Neither does the structure of the repository. But ways to structure the models should be taken into consideration.
3. webgl vs. rendered images
in my browser the webgl view has a terible performance and provides a really bad expierence to me. It would be nice if i could disable the webgl view somehow. I like the possibility of thingiverse to attach numerous 2d images.
And when it come to a gallery view, multiple webgl views would not be feasable. Maybe we could develop a rendering service for FCStd files. But this will be a long way to go.
4. editing
I see that there is currently no way of editing existing uploads. I would like to have a mechanism to remove or hide things. When testing, sometimes thing just go wrong.
jreinhardt
Posts: 329
Joined: Mon Sep 09, 2013 6:08 pm

Re: Model hosting and one-click-upload

Postby jreinhardt » Sat Aug 16, 2014 6:51 pm

Hi Sebastian,

you are absolutely right, this is just a proof of concept, and its main purpose is to get a better understanding of the problem, than is possible through theoretical considerations. And therefore your feedback is much appreciated:

1. I agree, I found this to be a problem as well. In fact, files created by an older version of FreeCAD might not even have a uuid. this causes a new one to be generated every time it is loaded, which makes it not so useful as an identifier. At the moment I am tending towards a hash of the content of the FCStd file, similar to how git addresses content. This ensures that the same object gets the same identifier, and modified objects a different one.

2. I also agree. The main reason why it is one-click at the moment is because I was too lazy to create a dialog :) I don't have a lot of experience with web technologies, so I focused on getting the REST interface up and running.

3. Attaching images is on my todo list. Once I add such an API, there are lots of possibilities. For example, one could also offer the user to upload drawings or drafts from the file. Or one could try to generate a number of views in the client side macro. I want to avoid a dependency of CADinet onto FreeCAD, as it probably is difficult to get it to run on OpenShift. So a rendering service would need to be external. Rockn has done some work with a server-side FreeCAD, if I am not mistaken.

4. One can update existing uploads by uploading again. Again, this is because everything is in a very early stage, not because I want it to be this way.

So thanks a lot for your feedback! I guess you are one of the two people that actually tried to use it http://thingcollector-bolts.rhcloud.com/cadinet/users ?

Greetings

Johannes
BOLTS, standard parts library for FreeCAD Thread, Help with Translation
User avatar
NormandC
Posts: 18534
Joined: Sat Feb 06, 2010 9:52 pm
Location: Québec, Canada

Re: Model hosting and one-click-upload

Postby NormandC » Sat Aug 16, 2014 7:41 pm

Hello Johannes,

What you've made so far is impressive. But to me as an end user, the Thing Tracker Network is still something very theoretical, very vague that I struggle to get a handle on.

Also it does not appear to me as an efficient way to look for designs. Looking at the things listed in the thingcollector you posted, many entries are not even actual "things": it lists software (BRL-CAD, CADemia, even FreeCAD but with the obsolete sourceforge wiki url), guides or dead forum links...

I'm all for getting out of walled gardens, but at the moment the TTN apparently cannot offer the same level of functionality as the model repositories such as Thingiverse or YouMagine where you can clearly know what an entry is about because you can browse images.
jreinhardt wrote:Attaching images is on my todo list. Once I add such an API, there are lots of possibilities. For example, one could also offer the user to upload drawings or drafts from the file. Or one could try to generate a number of views in the client side macro.
What would really be important is to allow a "thing" entry to contain multiple files. Creating separate things of all the files for a project (a RepRap printer for example) would not be efficient. It would also be useful to allow uploading of different file types along with the fcstd file: the ubiquitous STL format, but also STEP and maybe IGES.
jreinhardt
Posts: 329
Joined: Mon Sep 09, 2013 6:08 pm

Re: Model hosting and one-click-upload

Postby jreinhardt » Sat Aug 16, 2014 8:06 pm

Yes, TTN is not really useful so far. And the reason why so much un-thingy stuff comes up is that the biggest site that publishes its stuff in the TTN is the reprap development tracker (http://reprap.development-tracker.info/, created by the same guy who started TTN) which had a wider scope, but is also not really alive anymore.

But the only way to make it useful is to populate it with useful stuff. There are two ways, either getting existing repositories to publish their stuff also in TTN, or to get people to upload stuff. And I believe the way to get people to do that is to make it easy. The nice thing is, that CADinet (not in its current state, but once it evolved into something more complete) could be useful on its own, e.g. to share models that are too big to attach to the forum.

The TTN specification offers ways to express relations between different things, so a 3D printer could be uploaded as different things with such relations tying them together. How this is best exposed to the user is something that remains to be figured out.
BOLTS, standard parts library for FreeCAD Thread, Help with Translation
User avatar
NormandC
Posts: 18534
Joined: Sat Feb 06, 2010 9:52 pm
Location: Québec, Canada

Re: Model hosting and one-click-upload

Postby NormandC » Sat Aug 16, 2014 8:22 pm

jreinhardt wrote:But the only way to make it useful is to populate it with useful stuff. There are two ways, either getting existing repositories to publish their stuff also in TTN, or to get people to upload stuff. And I believe the way to get people to do that is to make it easy.
On that I agree completely!
jreinhardt wrote:The nice thing is, that CADinet (not in its current state, but once it evolved into something more complete) could be useful on its own, e.g. to share models that are too big to attach to the forum.
You know I didn't even think of that. This could be useful right now. Or would you say it would be better for CADinet to be more mature before we start using it to share big FCStd files on the forum?
jreinhardt wrote:Both the thingcollector and the CADinet are easily deployable in the OpenShift PaaS, i.e. it is very easy to run your own instances of these apps. It should also be rather easy to run them on your own server.
I've been thinking of self-hosting (email, blog if I ever start one again) for a while but am not ready yet. Seems like OpenShift Online is free and offers 1GB of storage. I may be tempted to create my own CADinet.

I may have sounded critical in my first reply, please let me amend it be saying I appreciate a lot your efforts to make TTN into something less technical people like me can use. :)
jreinhardt
Posts: 329
Joined: Mon Sep 09, 2013 6:08 pm

Re: Model hosting and one-click-upload

Postby jreinhardt » Sat Aug 16, 2014 8:37 pm

normandc wrote:
jreinhardt wrote:The nice thing is, that CADinet (not in its current state, but once it evolved into something more complete) could be useful on its own, e.g. to share models that are too big to attach to the forum.
You know I didn't even think of that. This could be useful right now. Or would you say it would be better for CADinet to be more mature before we start using it to share big FCStd files on the forum?
Everything is still very much in flow, and I it sometimes happens that I have to change the database stuff, and then it is much easier to wipe everything than to build a migration path. So I would say it is a bit early to really use it. But I would very much appreciate testing and feedback.
normandc wrote:
jreinhardt wrote:Both the thingcollector and the CADinet are easily deployable in the OpenShift PaaS, i.e. it is very easy to run your own instances of these apps. It should also be rather easy to run them on your own server.
I've been thinking of self-hosting (email, blog if I ever start one again) for a while but am not ready yet. Seems like OpenShift Online is free and offers 1GB of storage. I may be tempted to create my own CADinet.
Again, don't expect a smooth process and clear update paths or persistence. And again, I am very thankful for feedback, especially from users with a different backgrounds. That helps against tunnel vision and detachment from reality.
normandc wrote: I may have sounded critical in my first reply, please let me amend it be saying I appreciate a lot your efforts to make TTN into something less technical people like me can use. :)
You were quite right to be critical, no offence taken :)
BOLTS, standard parts library for FreeCAD Thread, Help with Translation
jreinhardt
Posts: 329
Joined: Mon Sep 09, 2013 6:08 pm

Re: Model hosting and one-click-upload

Postby jreinhardt » Wed Aug 20, 2014 1:29 pm

Hi,

I found a bit of time to work on cadinet and the FreeCAD macro, and address a few of the points that were raised.
  • I don't use UUIDs from the FreeCAD document anymore as ids in TTN, but instead use one generated from a hash of its contents. However, I am not yet convinced, that this is the way to go. For example, editing uploads is essentially broken by this, as changing the model or metadata creates a new thing instead of updating the old one.
  • I created a dialog that offers an easy way to enter metadata and stuff. It is not perfect (just as shoogen requested :D), and there are some fields that are not functional yet. If requested it updates the file before uploading.
  • Cadinet got a way to disable the WebGL preview. It is based on cookies, so your mileage will vary.
Testing is appreciated.
BOLTS, standard parts library for FreeCAD Thread, Help with Translation
User avatar
NormandC
Posts: 18534
Joined: Sat Feb 06, 2010 9:52 pm
Location: Québec, Canada

Re: Model hosting and one-click-upload

Postby NormandC » Thu Nov 27, 2014 3:28 am

Hello Johannes,

I just stumbled on this article and was wondering if CADinet's 3D viewer would be similarly affected.

Designers beware: Real-time 3d viewers can easily be hacked.

I still haven't managed to set up my own CADinet instance.
jreinhardt
Posts: 329
Joined: Mon Sep 09, 2013 6:08 pm

Re: Model hosting and one-click-upload

Postby jreinhardt » Thu Nov 27, 2014 10:08 am

Hi Normand,

yes, I guess CADinets viewer would also be "vulnerable" in this sense. I actually don't see a way to make this harder except using flash or other locked down plugins. But I don't think this is a problem for CADinet, as it would only host openly licensed models anyway (everything else is too much of a legal risk for my taste), so there is no reason to prevent people from getting the model.

I am actually currently working on CADinet. Most of the code will be rewritten and reorganised. I have a few exciting (at least exciting for me) features in mind and will set up a up to date instance when its ready. And I have the use case of a parts repository for FreeCAD still in mind. You can be sure that I will report back then and ask for feedback.

Greetings

Johannes
BOLTS, standard parts library for FreeCAD Thread, Help with Translation