Terry,
Ancestry does have groups, please see: AncestrySupport not played with them that much but there are a number of youtube videos that explain and demonstrate their use.
You ask " Is it RootsMagic’s fault that Ancestry does not accept RM Groups in its API" well I am not knowledgeable enough to know the answer. However in another post that I was involved in regarding uploading to Ancestry (here: Uploading to Ancestry from RM 8 - Avoiding huge numbers of images - #21 by TomH) I copied a response from RM Support which stated that " *We don’t have a lot of control over the Ancestry API since Ancestry is the one that builds and maintains it.*" However much more knowledgeable people on that post thought that RM did have control over what can be passed to Ancestry.
So I am unsure what the reality is, but for sure having tested this, groups are not passed to Ancestry.
The reality for me at least is that when I have made significant changes to the tree that exists in RM I upload a complete new tree to Ancestry. This is because of the limitations in TreeShare and the change list which are acknowledged in the above post by RM support ( they state *So it’s hard to say sometime what triggers people being added to it.) So the lack of transference of groups may be an issue.