Although it was the same scoreline as the meeting in last season's competition, it was a far more even encounter.
The U's had chances and will reflect in particular on a bad miss from Justin Richards with the score at only 1-0 early in the second half.
The Blades had to work hard for their win, but their greater cutting edge won the game.
Tony McMahon's free-kick gave them a lead at the break and after Richards's miss Dave Kitson doubled the lead.
Even at 2-0 the underdogs threatened, but Nick Blackman's late strike settled the tie for good.
Richards came straight into the side to begin his second loan spell, replacing the suspended James Constable.
There were two other changes in the XI, with Simon Heslop in for Lee Cox and the more defensive-minded Tony Capaldi preferred to Sean Rigg on the left flank.
The Blades showed how seriously they took the game by naming a very strong side, with Jamie Murphy handed a debut after joining from Motherwell on Thursday.
There were 35 places between the sides in the league pyramid, but it was the lower ranked team who made the better start.
The U's had three glimpses of George Long's goal in the opening 15 minutes as they attempted to make a fast start.
Peter Leven, Heslop and Tom Craddock all tried their luck from range, but the trio's efforts were all off target.
Sheffield United were forced into an early change when Matthew Hill came off second best in a challenge with Jake Wright.
The visitors from npower League One had barely threatened, but they took the lead with the first meaningful attack, on 17 minutes.
A direct run from Murphy was halted by Wright on the edge of the box and although the U's skipper felt it was a clean tackle, referee Carl Boyeson awarded a free-kick.
The hosts brought all ten outfield players back to defend, but they were helpless as McMahon flicked a shot over the wall which caught out a flat-footed Ryan Clarke and ended up in the net.
Chris Wilder's side responded well and their best chance before the break came midway through the half.
A quick throw from Capaldi saw Craddock sprint past a flagging Harry Maguire. The striker got his shot away and Long turned it round the post for a corner.
The U's showed enough going forward to worry Sheffield United's back four.
Alfie Potter's pace in particular appeared to cause them problems when he was given space to dribble.
It did not lead to anything, but the spell before half time ensured the underdogs went in at the break still well in the tie.
They needed one superb block from Michael Duberry to keep out a shot from Michael Doyle. On the rebound Clarke saved well at point blank range.
It kept the U's in the game and nine minutes after the restart they should have drawn level.
Craddock superbly cushioned Capaldi's long ball into Richards's path, sending him clear inside the box. But the ball appeared to get caught under the striker's feet and after an age he could only shoot wide.
It looked a costly miss at the time, and so it proved.
Midway through the half Blackman rose at the back post to nod a free-kick back across the face of goal, where Kitson bundled in from close range.
Blackman came within a whisker of adding a quick third, but his curling shot was deflected just past the post.
Rigg, who had been brought on just before the second goal, gave United an added cutting edge as they looked for a lifeline.
The winger fired one chance wide and saw another effort smartly saved as the home side enjoyed their best spell of the match.
The clearest chance fell to Craddock, who got away with a shove on Marcus Williams to collect Leven's through ball. The striker steadied himself, but drove a low shot inches wide.
With three minutes remaining the Blades finally made the game safe, when Blackman buried a finish past Clarke.
Oxford Utd (4-4-2): Clarke, Batt, Duberry, Wright, O'Brien (Rigg 67), Potter, Heslop, Leven, Capaldi, Richards, Craddock.
Unused subs: Brown, Worley, Raynes, O'Dowda, Parker, Marsh.
Sheffield Utd (4-4-2): Long, McMahon, Maguire, Higginbotham, Hill (Williams 12), Blackman, McDonald, Doyle, Flynn, Murphy (Cresswell 89), Kitson (Ironside 76).
Unused subs: Tonne, Whitehouse, Kennedy, Coyne.
Attendance: 7,079 (1,244 visitors).
Referee: Carl Boyeson.