Offering my servers to build FreeCAD, realthunder's fork and PRs

[Cyclonit]

Hi,

I've been in contact with realthunder regarding building his fork more regularly. According to him and some posts here in the forum, there already exists some infrastructure for automated builds, but there are issues regarding server power, storage space and cost. Afaik the current infrastructure does not allow building PR branches. To keep things simple, I have at least one two spare servers I can offer to the project. A second might be available too, if necessary.

Both servers have the following specs:

• Cores: 10
• RAM: 32GB
• HDD: 1Tb
• IPv4: 1 public
• IPv6: public /64 block
• Location: Frankfurt DE

As mentioned, the server currently goes unused and I do not have an immediate use case myself. I like the idea of "sponsoring" FreeCAD this way. However there is a limitation regarding distributing the binaries. The server does not have a bandwidth limit, but the provider has a fair use policy. Thus binaries must not be distributed publicly directly from the server as this would definitely exceed the policy.

I'll give realthunder preliminary access to the server later today. If "you" as the project are interested in the offer please feel free to contact me on whatever channel you want and we can discuss the details.

[chrisb]

Please don't hijack this topic.

[jmaustpc]

Your first post, welcome to FreeCAD.

I split this into a new topic since it had nothing directly to do with the releasing 0.19 topic.

Your offer sounds generous.

I've been in contact with realthunder regarding building his fork more regularly.

I have no idea if realthunder would see value in that or not, but I would guess he would.

The server does not have a bandwidth limit, but the provider has a fair use policy. Thus binaries must not be distributed publicly directly from the server as this would definitely exceed the policy.

Perhaps you could compile the binaries and then push them to some public server like maybe GitHub or somewhere else and then people can download them from there rather than directly from your server?

According to him and some posts here in the forum, there already exists some infrastructure for automated builds, but there are issues regarding server power, storage space and cost. Afaik the current infrastructure does not allow building PR branches.

The services offered all have limits, FreeCAD is large so we had to get special permission to exceed the limits on some of these services. For example the Ubuntu PPA for the daily build of master had a compile time limit that compiling FreeCAD exceeded so we asked Canonical for an extension and they granted it.

I've been in contact with realthunder regarding building his fork more regularly.

Just another thought, FreeCAD is cross platform so there would be a lot of different versions to compile.

Are you interested in offering FreeCAD some server capacity for other FreeCAD purposes? I am not sure what but one thought is perhaps some space as an alternative back up location for our wiki data and the like. Perhaps a temporary back up while upgrading the software for example. Although I would have to ask the others if we need this or not as I am just guessing.

Anyway, welcome to FreeCAD!

Jim

[Cyclonit]

Hi,

I wasn't exactly sure where to post this and the thread seemed like a good idea. Thanks for fixing my mistake.

Realthunder is interested but he has pointed out that he is a coder and prefers coding over spending a lot of time setting up and babysitting build pipelines. He suggested I extend this offer to the entire project.

Perhaps you could compile the binaries and then push them to some public server like maybe GitHub or somewhere else and then people can download them from there rather than directly from your server?

That was my first idea too. Afaik Github offers an API for pushing releases/generic build artefacts to their servers. Their quotas are very generous too. The only limit they impose is that every individual artefact must be smaller than 2GB. As the FreeCAD binaries tend to be less than 1GB in size, that should not be an issue.

The services offered all have limits, FreeCAD is large so we had to get special permission to exceed the limits on some of these services. For example the Ubuntu PPA for the daily build of master had a compile time limit that compiling FreeCAD exceeded so we asked Canonical for an extension and they granted it.

Can you elaborate on limits that are currently impacting your progress? All I can say regarding my offer is that you can use at least one server at 100% computing capacity whenever you like. I am paying for the resources anyways

Just another thought, FreeCAD is cross platform so there would be a lot of different versions to compile.

I am not familiar with how many versions there are and how long each of them take to compile. It likely won't be possible to compile all of the different versions on every commit, but frequent updates should be possible. I am certain we could find an arrangement that works out. As I mentioned earlier, I might have a second server available too, but I need to make sure that the former users no longer need it. Lastly, you can of course use both the current services and my server(s) at the same time.

Are you interested in offering FreeCAD some server capacity for other FreeCAD purposes? I am not sure what but one thought is perhaps some space as an alternative back up location for our wiki data and the like. Perhaps a temporary back up while upgrading the software for example. Although I would have to ask the others if we need this or not as I am just guessing.

I am open for suggestions and I'll see what I can come up with. Backing up a wiki should be simple (unless you use it to host a lot of large files).

P.S.: Thank you for the welcome As one of many F360 refugees, I hope to find a new home here

[vocx]

...
I am open for suggestions and I'll see what I can come up with. Backing up a wiki should be simple (unless you use it to host a lot of large files).

ping

kkremitzki is the main guy handling the server infrastructure.

Interesting alternative option to Digital Ocean hosting

In the past I suggested that we need a place to build and host the entire C++/Python documentation generated with Doxygen. See source documentation.

The current page, https://freecadweb.org/api, has a smaller version of the documentation, which is not very useful. Having access to the entire thing, as if you had compiled the documentation locally, would be good. Each full set of documentation is around 5 GB, mainly becuase of all the generated images produced by GraphViz (the "dot" program).

And it may be good to generate the documentation semi-regularly, maybe once per month, to have access to different development versions. And of course, we would keep a copy of the documentation for the main stable versions, that is, v0.17, v0.18, v0.18.1, 0.18.2, etc.

Another suggestion that was made is to somehow link the forum to a server, so media files can be uploaded there, for example, FreeCAD files. This would allow users to upload big FreeCAD files (or PDFs, PNGs, videos, etc.) that go beyond the current 1 MB limit that the forum permits.

[Cyclonit]

In the past I suggested that we need a place to build and host the entire C++/Python documentation generated with Doxygen. See source documentation.

I am wary of the network traffic that hosting stuff on the servers would introduce. But given that developer documentation is a niche topic, we could give it a try. Afaik it should tie in to building the source code anyways to ensure that binaries and documentation are always up to date.

Another suggestion that was made is to somehow link the forum to a server, so media files can be uploaded there, for example, FreeCAD files. This would allow users to upload big FreeCAD files (or PDFs, PNGs, videos, etc.) that go beyond the current 1 MB limit that the forum permits.

I have to give you a hard no on this one. I do trust veteran contributors to FreeCAD to use the servers properly, but I do not trust (to me) anonymous users of the forum. I don't want them to end up as file dumps.

[vocx]

I am wary of the network traffic that hosting stuff on the servers would introduce...

I am not a networking guy, so I don't understand quite well the implications of this. To my ears this sounds like "we have a server but we shouldn't use it!" It sounds counter-intuitive. What exactly can we use it for, then?

I think there would not be a lot of traffic for hosting the source documentation, because this is a resource mostly useful to programmers; it's not like there is a ton of programmers constantly checking the documentation, but it has been requested in the past. Our current online documentation is mostly useless because it is incomplete; if you are a programmer, you really need to be able to click on every link to follow the classes and methods of the program. As it is right now, the best recommendation is to compile the documentation locally and browse that.

... I do trust veteran contributors to FreeCAD to use the servers properly, but I do not trust (to me) anonymous users of the forum...

I think in the previous thread, kkremitzki mentioned that maybe there is the possibility of allocating personal space, say, 200 MB for individual users, but he didn't mention more details about it. So, I understand your concern. I don't know if it's possible to allocate some per-user space (for power users), and some general space that can be used by the community at large to share big files, and which can be erased regularly. As I said, I'm not a networking guy, but maybe some intermediary solution can be found.

Last time the servers were upgraded, about a year ago, I remember it was said that the entire forum was around 20 GB in size, including all existing threads and attachments. So all in all, it isn't very big.

[Cyclonit]

I am not a networking guy, so I don't understand quite well the implications of this. To my ears this sounds like "we have a server but we shouldn't use it!" It sounds counter-intuitive. What exactly can we use it for, then?

Networking is all about scaling. Let me try to give you an example:

- If we build 20 branches of FreeCAD per day, each generating a 500MB binary and we push these binaries to GitHub, we and up with ~10GB of network traffic each day.

- If we host the 10GB of binaries on my servers and users can download them from there the numbers start growing really quick. If we have one user downloading a binary (they most likely won't download different builds all that often) its a mere 500MB. Not an issue. If 10 people download binaries, we are back up to 10GB. If 100 people download them, we are at 100GB and so on.

I don't know how often people download FreeCAD, but afaik hosting binaries on my servers is neither necessary (as we can push them to GitHub), nor is it an experiment that I want to indulge in.

I think there would not be a lot of traffic for hosting the source documentation, because this is a resource mostly useful to programmers; it's not like there is a ton of programmers constantly checking the documentation, but it has been requested in the past. Our current online documentation is mostly useless because it is incomplete; if you are a programmer, you really need to be able to click on every link to follow the classes and methods of the program. As it is right now, the best recommendation is to compile the documentation locally and browse that.

I agree with you here. Few people click through every page of the documentation just because. Browser caching of images helps here too.

I think in the previous thread, kkremitzki mentioned that maybe there is the possibility of allocating personal space, say, 200 MB for individual users, but he didn't mention more details about it. So, I understand your concern. I don't know if it's possible to allocate some per-user space (for power users), and some general space that can be used by the community at large to share big files, and which can be erased regularly. As I said, I'm not a networking guy, but maybe some intermediary solution can be found.

Technically, it is possible to allocate server space for power users. But it ends up with the same scaling issues mentioned earlier. Additionally, there is no way of controlling what kind of files users are uploading and sharing through the server. File ending filters and such are worthless as you can simply change the file extension and tell the recipient to change it back. Lastly, providing a file sharing service has legal consequences in Germany, which I rather don't get into. Binaries that were build directly from source on the server itself are much simpler in these regards.

[Cyclonit]

Smallish update: Both servers I mentioned are available for FreeCAD. In total that gives us 20 cores and 64GB of RAM to build stuff 24/7.

[sgrogan]

Smallish update: Both servers I mentioned are available for FreeCAD. In total that gives us 20 cores and 64GB of RAM to build stuff 24/7.

Hey Cyclonit, welcome from me as well.
Thanks for the generous offer.
As mentioned it would be good to here from @kkremitzki

I can think of a few uses for your offer that don't require hosting assets.
Do you foresee issues spinning up workers on the 3 main OS's, masOS, Linux, and Windows.