Support Managed Identity
Various other resources support the use of Managed Identity, like Azure KeyVault does. This prevents the hassle of juggling secrets. The Graph API could use this as well.
The current (clean) workaround seems to be to use Azure KeyVault with Managed Identity, and then get the client secret used for the Graph API from there.