Unfortunately, I use "stop tracking" to track other shows that I don't want to track individual episodes for, but that I do want to show up on the calendar. So if I change that setting then those shows won't show up on my calendar.
There are four scenarios:
Show on calendar, track episodes
Show on calendar, don't track episodes (or at least don't show them in the "Episodes" list)
Don't show on calendar, track episodes
Don't show on calendar, don't track episodes
The last one is just not doing anything with a show...never add it to watchlist. You can filter the calendar by watchlist so that is supported.
The first one is standard watchlist behavior, allowing episodes to be tracked.
The second case is the standard "stop tracking" behavior. Technically the episodes are still tracked, but the show doesn't show up in the Episode list which is sufficient.
The third case can be enabled by changing the setting that Santah mentioned. Unfortunately, this effectively disables the second case.
What I am looking for is support for all of these cases at once where I don't have to choose between the second and third cases. For some shows I would like to make use of the second case and for others, the third.