feat : Metrics - Remove all data collection from Stride.Engine

[Jklawreszuk]

PR Details

reopen of #2261

Description

Goodbye telemetry! 👋

Motivation and Context

Metrics feature in Stride.Engine can be considered as leftover from the time Xenko (now Stride Engine) was closed-source. While telemetry may be useful in commercial products, Stride Engine is being developed as free and open source software by independent volunteers.

Thus, information about current needs, bugs and errors is being frequently gathered by making requests on Github or directly through discussions with others on our Discord channel and there is no need to do any much more.

PR updates also Stride.Editor.CrashReporter. Previously, user could send anonymized report to support.stride3d.net, but this service has been forgotten and is useless since we are commonly use Github. Now it is possible to view and save the created report containing metrics that can be analyzed and - if the submitter wants it - attached to a new Github issue.

Types of changes

• Docs change / refactoring / dependency upgrade
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• My change requires a change to the documentation.
• I have added tests to cover my changes.
• All new and existing tests passed.
• I have built and run the editor to try this change out.

[xen2]

So, overall I agree with removing metrics client and EULA from startup (I think it was the consensus from discussions as well)
But I wanted to quickly rediscuss it again to make sure since it was a while ago.

About metrics

Overall I like that if we get rid of it, that's one less thing to maintain, remove privacy/hack/leak risks, etc.
We can always see trends with nuget downloads, website traffic, etc.
OTOH, detailed stats could have been useful.
Important ones could be:

• which platforms people develop for (to know which one to prioritize)
• versions they use (to know how long to support/migrate)
• how often they use (people trying once vs long-term user)
• etc.
  Are we sure beyond doubt we can be OK without that data? If we remove metrics, there's likely no return.

Note: if we were to keep it, we could look into how to anonymize it as much as possible (while still making sure it's not being faked).
And at least a way to opt-out.
Also, is there an existing system we could reuse nowadays?

Crash report

I still think that longer-term, we might want proper & easy crash report without user needing a github account.
It's much more targeted than Metrics and could have its own mini privacy check when reporting.

We could look into using Sentry or GlitchTip (either using their server or self-hosted; I would favor their server using OSS account to simplify our maintenance/IT).
One important feature is to be able to aggregate many similar looking crash/callstack.
A specific EULA could be part of the crash report dialog (and it can be much smaller than current one, and flowing well with user report).
Of course, adding this feature could be a fair amount of work so it could come later (that's where your github report comes in handy to fill the gap). Also it is just a proposal and will likely need further discussion/decision.

[Kryptos-FR]

Note: this PR will conflict with #3276 where the metrics have also been removed (from the launcher) and the crash report rewritten.

[Eideren]

We haven't taken any decisions based on those since the engine has become open source, I think the data we could retrieve for metrics is only actionable if we had the resources to act on it. I would guess that we barely have a third of the maintainers required to start looking into long term support, improving the on-boarding experience, or even improving platform support.

The biggest event that could increase our pool of maintainers already happened, the next one would be for Unity to cease operation, but I'm sure users would still be able to develop their games on older versions if that's the case. Our next relevant competitor, Godot, has a license that would be impervious to any issues.

Are we sure beyond doubt we can be OK without that data? If we remove metrics, there's likely no return.

We can make it opt-in in the future with a popup dialog on the first run. Wouldn't be as statistically relevant as an opt-out, sure, but if we make it clear that they are more likely to get support by opting in and that data is aggressively anonymized, more people would be inclined to opt-in.

[Jklawreszuk]

I agree with @Eideren. The key issue I see is that while we are collecting user data, we have essentially failed to process it meaningful way for the quality improvement of this project over the past 8 years. From my perspective, I see two realistic scenarios:

A. Completely rewriting the telemetry from scratch: The current architecture is simplified and, frankly, outdated. Even the web app doesn't function correctly, and even if it were fixed based on my unfinished patches from a few years ago, I’ve concluded that the current metrics do not provide significant value to the community, in my personal opinion. First, we would need to jointly decide exactly WHAT we want to achieve through data collection (e.g., monitoring editor optimization). Second, as Eideren mentions, the number of active contributors is small - let's say we can rely on 8-10 users at any given time. If we wanted to seriously manage telemetry, this goal could consume resources that are larger than what we currently have, making it a very long-term objective. If we were to proceed, I would prefer an opt-in model over the current opt-out approach. Most open-source tools follow the "express consent" model, and psychologically, it is a safer choice. While the sample size would be smaller, it would ensure higher, let say "engagement" from test users.

B. Removing telemetry and relying on GitHub issue reporting: From my subjective standpoint, the current process of bug reporting via GitHub covers 99% of cases. We have great templates that encourage users to include necessary error logs, and we can quickly contact devs via Discord to discuss bugs. I agree with @xen2 that a custom bug report feature would be a great long-term addition, but like Scenario A, it would require dedicated effort.

Addressing @xen2's points:

1. In my view, we know which platforms we want to prioritize - all "core" platforms supported by .NET, namely Desktop + Mobile. I know, this is idealistic goal, but it is a clear direction. Consoles are still very far off the radar.
2. We can find out which versions users are using by analyzing download trends on nuget.org over the last two months. The data shows that users are generally downloading the latest versions.
3. Retention data could be useful, but as I belive this requires rewriting the entire telemetry module.

Overall, regardless of the chosen path, I believe Stride requires the removal of the current metrics module. Whether it is redesigned or not will be a topic for a separate discussion.

[xen2]

Why/when could it be useful

I agree retention analysis is a much more advanced use case and is not my main intent to want some telemetry data.
Some data is easy to act on and it's already been useful in the past.
Most of those examples couldn't be guessed just looking at nuget download numbers (cf @Jklawreszuk point):

• Can we drop a platform? (i.e. UWP, or IOS/Android a long time ago when Xamarin was expensive and slow)
• Check how much mobile is worth spending time on (of course it's a bit chicken & egg: if mobile is not in a good shape, people won't use it). Still, if used by 10% of people, maybe not worth to spend 80% on it and cap it to 20%.
• Check versions usage
  It's esp. important to know how long we need to support something, such as a direct upgrading path. If a version is still much used, we better be able to load it with latest Stride version; but if too cumbersome and not a much used version, we could simplify code by dropping support for opening older versions, and force people to go through intermediate version.
  Example: Stride 4.4 can upgrade from 4.0. If you still have 3.x (Xenko) you have to upgrade it first with 4.0~4.2 versions then you can open it with 4.4. We took that decision to simplify code a lot once 3.x usage went very low.
  Note that nuget download numbers is not enough: many teams might have downloaded it long ago, but keep running it daily. Download numbers just tell you about new downloads so it will be skewed toward latest version.
  Also, I think we will eventually remove Xenko (3.x) support from Stride Launcher once it gets to almost 0 downloads.

What we don't do yet but could be useful:

• Check which plugins are used (i.e. I am pretty sure we should drop SpriteStudio, nobody uses it I think)
  Note: this specific we could guess from nuget download numbers.

OTOH removing it would simplify our infra, remove uneasiness from some users and remove any risk of data leaks.

Infra

Agree with @Jklawreszuk , if we were to keep it it will need a rewrite, using existing software/server/SaaS so that we don't have to maintain most of it ourselves. There are plenty of good system for that nowadays.
However, what will need a replacement for sure is the backend.
Client-side really depends on what happen for the backend (we might want to keep backward compat rather than having a new endpoint).
That's why I would like first to decide if we still want any telemetry (other system).
Then we can decide how much to remove.

[xen2]

OK, I have done some more research and a quick proposal.
First of all, it's OK to merge this PR (once everything is green), as new approach won't need any of the existing code.

Crash report

We still want crash reports.
Most users never open a GitHub issue, and a server-side collection lets us group crashes by callstack and see which ones are the most frequent.

I was thinking to use Sentry (SaaS, free for open source projects, in EU data region):

• Good callstack grouping for C#, release tagging, discord alerts on new crashes.
• The editor will point to a domain we control (e.g. crash.stride3d.net) that forwards to Sentry. If we ever want to switch to a self-hosted server later (GlitchTip is compatible with the same protocol), it is just a DNS/proxy change, no client update needed.
• Consent happens when submitting: the user sees the full report content and chooses to send it (or open a GitHub issue instead). No more privacy policy dialog blocking the first run.

Usage metrics

The current implementation can go away completely.

If we later want basic usage numbers (versions, platforms, OS, possibly more stuff like public plugin list), the idea would be:

• Use something like Aptabase: open source, made for desktop apps, no user identifiers at all (so no tracking, as opposed to current implementation), GDPR compliant, EU hosting, self-hostable.
• A very short privacy note on the website ("We send anonymous events: version, OS, country; here is the full list. You can turn it off in settings"). It would be one paragraph instead of the huge privacy policy we have now.
• Possibly a small dialog on first run asking if it's OK to enable it. Question is, should the checkbox should default to on or off? (open for discussion -- any opinion? or other idea?)

This would be opt-in/optional, with the exact data sent published.

This PR

As a result, nothing here blocks this PR.
As for old metric client code, if we make sure server endpoint fails fast (i.e. error 204, not a DNS removal), it won't stall so we can take down the server.

[VaclavElias]

Sounds good to me!