I am of the opinion that we should treat Travis CI as we would treat it if their infrastructure were our own.
I am an advocate of frugality (not cheap, but cost conscious). This means that I think we should have the ability to build PRs multi-compiler/platform because it gives us a considerable value despite its computational cost. At the same time we should not build PRs that we do not need (do not contribute to global warming/do not waste energy/do not block resources for others... anyone can take their pick depending on their own ideals).
So, I think you are very much right and that what you say makes a lot of sense. I think we should promote the use of [skip ci] (for those who do not know, just include it in a commit message and Travis won't build it), where appropriate.
On a totally different balance between convenience and energy efficiency, if we could tend not to PR draft code which we do not intend to have (directly) merged, that would also improve the situation with Travis. Something like not building PR that we intend as draft PRs. I do not know if there is such a functionality ( something like the [skip ci] thing, but for a PR, with the possibility of actually building it when the PR as such is intended). However, something like this should have some automated logic, we should not give up the convenience of sharing code for review with appropriate differences just because Travis is always automatically triggered. We may need to explore if there is some kind of way of working to improve this.