The first time I drove one of those engines I had the same thought. This thing would be better with a smaller turbo!
Unfortunately there's no easy way to make big strides while sticking with the t3 family of turbos.
There are a bunch of factors that influence how quickly a turbo accelerates/'spools'. I think for this case we can basically rule out the exhaust flow of the engine itself because making it flow significantly more at pre-boost rpms would be pretty involved.
The design of the turbine housing has a huge effect, but the question then becomes, is there more than one t3 turbine housing that can easily be bolted to the stock manifold? And even if there are, will any of them make it spool faster? Assuming you have something like a .48a/r housing already, there's nothing readily available as far as t3 turbine housings that will result in earlier spool.
As for the actual rotating parts of the turbo, the vast majority of the weight of the rotating assembly is the turbine section which is the turbine wheel and the shaft that goes through the center section. The compressor wheel is usually similar in size to the turbine wheel, but since it's aluminum it is always much lighter and has a relatively small influence on spool.
What's in your turbo is considered a 'stage 1' t3 turbine. There are no smaller turbines in the t3 family, so going smaller to lose weight would require you to change to a different turbo family.
As for changing the compressor side, there are smaller t3 compressor wheels, but they will not be easy to find at all and they will require you to change the compressor cover/housing to match the wheel as well, which will probably create some minor install/fitment issues. But, since the influence of the weight of the compressor wheel (especially given the same exducer/major diameter) is so small, you would probably not gain much by going that direction.
Installing a 60 trim t3 compressor MAY result in earlier boost because even though it adds a small amount of mass that has to be accelerated, it also equates to a larger compressor output at a given shaft rpm which may end up being a net benefit. Also, while you cannot shrink the stock compressor housing to fit a smaller compressor wheel, you CAN bore it out to fit a larger one, so it creates less (basically none) install issues.
But, the 'easiest' thing to do to speed up spool is to reduce backpressure after the turbine. The turbo operates mostly on a pressure differential across the turbine. You can look at it like a fraction or a ratio like 5:2. If you add 1 to the first number you have a slightly higher ratio, but if you take one away from the second number you have DOUBLED the ratio. That sort of explains how reducing backpressure is so important to turbo spool. I don't know how much of a PITA the downpipe design is on that engine but if it's not a major headache I would put a 3" exhaust on it as the first mod and get it up to 3" as close to the turbine housing as possible.
IF that doesn't get you enough, I think you'll be looking at bolting a completely different turbocharger to your engine.