I'm with Trent on the importance and utility of aviation snips. At Eclectic Motorworks, we have nearly every sheet metal tool from snips to grinders to shears to brakes to bead rollers to an English Wheel and a Pullmax. I too hate noise, dust, sparks and the other nasties that come with some tools. I can cut a straighter line, faster, with snips because I've practiced, built up experience, and use them whenever possible.
To the original questions:
"would you be better off farming out things requiring a long sheet metal brake/bead roller or are there some tasks where hand tools might suffice?"
It depends, for many of these things there are less expensive workaround (like using angle iron and clamps in place of a brake), but for others outsourcing works well. Even with all of our tools, we sometimes have a piece bent up on someone else's long brake or press brake for example.
"I'm also interested in things like riveting. Any good books, perhaps something published by the EAA for aircraft construction?"
Several of Carrol Smith's books are very good as they bring common sense engineering into the fabrication. Ron Fornier's Metal Fabricator's Handbook and Sheet Metal Handbook are also very good. Both authors are gone now and the books are a little outdated, but very useful. Many similar books are out there, but most don't have the depth of these. Aircraft Spruce is a go-to place for plane builders and metal fabricators and lists a lot of books and specialty tools.
Ron Covell is a friend of mine and he's done several workshops at Eclectic. He's one of the best teachers out there. He used to sell a lot of videos, but now he puts them up on YouTube. Long before he had power tools, he used basic tools like snips and hammers and dollies. He's made many videos like the one Trent posted that are worth a watch. He's a genius at solving a complicated problem with simple tools.
I really like the Welding Tips and Tricks YouTube channel as well. Too many people try to debate MIG vs TIG while Jody just helps you get better with either.
To build your skills, I'd recommend starting with hand tools and try mild shaping with a hammer and dolly, making flanges with a hammerform, bending over straight edges and tubes, and maybe building one or more 1/12 scale model monocoques using welding and riveting (pop and hard). Use 20 gauge mild steel (not galvanized or otherwise treated) or .050" 3003 aluminum for the models as it's easy to get and work with. Better yet, try both.
Have fun and keep us posted!