The rs485 communication is not affected directly by the noise, thats the beauty of differential signals, at least as long as you keep them clean and take care in the layout. (Did not check that layout for "EMI-Errors" yet)
A shielded cable will not help you at all, because we have cheap plastic connectors on the boards. A shield without proper connection will just "float" with the noise and basically not absorb anything, as you said, it got worse with longer cables. Thats because the shield picked up the noise, and had more "opportunity" to couple it into the supply-lines that run inside the cable.
From my point of view, a lot of the 12V voltage dip that you can see in your oscillogram comes from the fact that we run some powerful stuff from the extruder board, but only deliver the current thru a "narrow" passage over the RJ45 cable, and neither the connectors, nor the cables are actually made for that kind of power. I did not ckeck the geared DC motors current rating, but i would expect that he alone probably sucks up at least 1A at 12V, then we have the heater, sucking another 2A (if you got the 6 ohms as planned). And last but not least, if you hook up the heated build platform from makerbot, you just added another 4 ohm resistor trying to eat up 3A. So, in my case, the extruder board needs more than 6A on the 12V line!
The small cheap (high ESR) capacitors on the extruder-board combined with some probable layout mistakes and the weak connector/wire choice combined are basically just asking for something to fail here…
As a quickfix, i'll probably try to get a dedicated 12V+GND supply wire from the atx-psu directly to the extruder board. The 2x2 pin connector with the two yellow and two black wires would be perfect for that. This could help getting the voltage a little bit more stable and maybe eliminate some of the problems.
Extruder board aside, ialready found quite some design flaws in the stepper boards that make them a nice noise-generator. On one hand, these were probably just some ways to keep the cost down, on the other hand, there were some simple EMI-related layout rules/guidelines that have not been followed at all. I am actually expecting the same or similar issues on the extruder board and on the mainboard too, but i did not dare to have a better look at them before i have my own stepper drivers up and running. (Prototype boards are currently getting shipped over from china, so i should have them assembled and hopefully tested next weekend)