With everything still set up as you like it and the swing-arm pivot screws still fairly tight, it's time to reinforce the 2 x 3 upright supports.
This is fast, easy and fun and boy does it stiffen and stabilize the joints.
1) Cut two 2 x 3s to length, the inboard side resting on the end-grain of the 1 x 2 axle extender guides, and mark the outboard side using the 2 x 6 as a guide.