It's really sounding more like ISP throttling. If it plays for a little while then suddenly changes, for both of you. That sounds like evil ISP magic.
Yeah, it's definitely looking more like that, although a flaky router isn't entirely out of the picture (just very hard to prove, and even harder to get ISPs to take action about). It's not unheard of for them to do header or packet inspection to block/throttle certain types of traffic. In fact, every mobile provider does this to some extent, and Comcast has been caught outright blocking certain perfectly valid ports/protocols before under the supposed pretext of being concerned about piracy or network stability.
As for options, you have a few. You can try complaining, although they may lob a "no servers" clause at you and ignore it. VPNs will most certainly work if they're doing some type of packet/header inspection since they couldn't determine what the traffic is. Changing the default port may also work, since they could be just be blocking/throttling based on port numbers for known service types. Lastly, you could see if the issue crops up with simllar traffic that they wouldn't see as a "threat", piracy-related activity, or "server"...something like higher-bandwidth direct gaming connections or video conferencing would be much more agreeable to them. If it has problems with those, then that's a complaint they can't just use the "no server" card and may actually investigate (admittedly, this would be my last resort personally).