LOUISVILLE, Ky. (WDRB) -- He fought both Muhammad Ali and Smokin' Joe Frazier in their prime, and now Louisville boxing legend Jimmy Ellis has died at the age of 74. Now his family members are sharing their memories of the man behind the gloves.

Jimmy Ellis fought in an era filled with great heavyweights. The Louisville native held his own against some of the greatest heavyweights in the sport of boxing.

The boxing legend was known to do most of his talking in the ring.

"I was proud as a peacock, big brother - heavyweight champion, yes," said Jerry Ellis, Jimmy Ellis' younger brother.

The younger Ellis says after winning the title, his brother never ducked or dodged any opponent in the ring.

"He fought Joe Frazier twice, he fought Ali, he fought Oscar Bonavena, he fought Earnie Shavers, he fought a number of great fighters, yes."

Ellis and the man known as "The Greatest," Muhammad Ali, were not only close friends who shared a hometown, they also shared something else that might surprise you.

"Angelo Dundee was trainers for both, but he was Jimmy's manager as well so when they fought, Angelo's allegiance was to Jimmy because he was manager and trainer," Jerry Ellis said.

After his retirement, Jimmy Ellis eventually moved back to Louisville and worked for the Metro Parks Department where he organized softball leagues.

"He took all comers, that's just the way he was," said Jeff Ellis, Jimmy Ellis' son.

Jeff Ellis is a former standout tight-end at Ohio State and one of the former champ's six children. He says after boxing, his dad had a passion for helping young people and never mind answering questions about his time in the ring.

"I watch classic fights all the time, we bring them up on Youtube and everything, we might even have a fight night."

Jeff Ellis recalls a phone conversation he had with his dad about one of the Frazier fights.

"I was like, 'why didn't you just dance around and box?' He said, 'well, I thought I could knock him out.' I said 'well, if you would have danced around and boxed, I think you could have got him, like that,' but he said, 'but I thought I could knock him out.'"

And despite their time as opponents in the ring, Jimmy Ellis remained close friends with Ali and Frazier.

"They got along real well and did a lot of stuff together," said  Jeff.

Jimmy Ellis died early Tuesday morning after a long illness, he was 74 years old.

"Jimmy had Alzheimer's, and he has been battling that for the last 7 or 8 years," said Jerry Ellis.

Tuesday afternoon, Ali Center officials released the following statement from Muhammad Ali:

"Lonnie and I are very saddened by the loss of our friend, and fellow Louisvillian, Jimmy Ellis. Our friendship began on the local Louisville boxing show 'Tomorrow’s Champions' and continued to grow throughout the years. In the ring he was tough. In the world of heavyweights, I have always thought that Jimmy was one of the best. As a former champion, Jimmy was known for exceptional hand speed and a strong chin. He was a master in the ring. Jimmy and I were both trained by Angelo Dundee, who would often say that Jimmy “packed more punch” than he was ever given credit for. Strong chin and punching power aside, it was his gentle manner and the compassion in his heart that I found most worthy of admiration. I had a kinship with Jimmy and felt like he and I were of the same cloth. He was a great athlete and a caring man. Great competitors who happen to be great friends are rare. Jimmy Ellis was that to me and I will miss him."

Meanwhile, funeral arrangements for Ellis are still pending.

Copyright 2014 WDRB News. All rights reserved.