Well, technically what they are doing wrong is they are using the xbox live service with a modded console, but I don't think thats the answer that your looking for.
No one knows for sure what MS is looking for to make a basis on banning someones console. All we have is theories and things of that nature. So far, it's been thought that poorly made backups was the cause, but now, they aren't for sure, some of the really involved people think they may have some other form of detection that we don't have a clue about.