Mike Ramirez was in Georgia at the time of the fire. According to police, the kids' mother - 23-year-old Danielle Lynn Brockman - had left her two children alone in the house when the fire started in the 12600 block of East 13th Avenue.

When police arrived on the scene, Brockman was outside claiming that her 1-year-old Alivia and her brother 3-year-old Timothy Ramirez were inside the apartment.

"Sunday morning, I got the call," Mike Ramirez told 9NEWS. "My mom called me. And, like anyone, I didn't know how to react at first. I mean, they are my kids, so I got scared and I had to be here, so I flew out the next day and I was here with them. Before I left, though, I found out that my son had already passed away."

His son Timothy passed away from the fire on Sunday.

"He was the greatest little rambunctious man you will ever meet," Mike Ramirez said. "He was my best little bud. I mean, he was just great."

His daughter Alivia passed away on Tuesday in the hospital.

"She was my only daughter," Mike Ramirez said. "Every parent is going to say it, but she was the most beautiful baby that I have ever seen. I mean, she was great. She was always happy, smiling."

Mike Ramirez said the grieving process is going to be a long one.

"Both of them are gone," Mike Ramirez said. "It's just hard to deal with the fact that they are gone, and I am never going to have to chase after T.J. anymore; I'm never going to be able to comb Alivia's hair. It's hard."

Mike says he's trying to be around friends and family to help him through this difficult time. He says thinking about it puts him in a worse mood.

"It's a somber feeling," Mike said. "Unfortunately, it is not just one death. It's two. It's two children that I'm burying. They are my children that I have to bury [Friday,] and it's probably one of the hardest things that anyone has to go through."

Mike claims he spoke to Brockman and she said she had left the children alone that day. He was angry, but says he did not push the subject because he was in town to be with his children - not to argue. He says he trusted her with the children for the most part.

"She and I had spoken maybe a month ago that I was going to have Timothy come live with me [in Georgia] once he was out of school because she said she felt overwhelmed," Mike said. "She 'couldn't handle him anymore,' so I just told her [to] just bring him or send him to me and I [would] take care of him."

Mike says after he got Timothy, he was planning on filing paperwork to get full custody of Alivia as well.

He says Brockman was not neglectful, but that he is only in Colorado to lay his children to rest. He says Brockman would not give him the full details of what happened when he asked her in the hospital. She did tell him, Mike says, that she had left the children alone.

"She has never told me that she has left them alone before, so this is the first time that she has ever told me [that,]" Mike said. "It wasn't a surprise. I hate to sound like I'm being crass or being mean or anything, but I really don't care anymore what happens to her legally. She's going to have to deal with it, not me. I really don't want to deal with it."

He feels responsible for not taking action on custody sooner.

"If she would have been there, I can guarantee you that I would be in Georgia talking with my kids on the phone instead of being here burying them," Mike said. "So, yeah, there is a little bit of blame there."

He says police told him the fire started in the kitchen, but they are still not saying the cause of the fire. Mike says Brockman's family has been very supportive during this.

"I love them, and every time we talk, they tell me that they love me and they are helping me out with all the arrangements that we have to do," Mike said. "They are my second family. I love them to death, and they have been helping me on all this."

The thing that hurts Mike greatly is that Brockman will not be at the funeral on Friday. Mike left with a few pieces of advice to other parents.

"If you get a chance, say 'I love you' to your kids," Mike said. "Hold them, hug them, don't take anything for granted, because - in a split second - they could be gone, just like Timothy and Alivia. Parents need to take time, not even with just their children, but just anyone around them - their family."

The funeral for Timothy and Alivia will be held on Friday at St. John the Baptist Catholic Church in Longmont. They will be buried at Mountain View Cemetery.

Mike says his family is still trying to raise money to help cover the costs of the burial. He said they have set up a fund at Wells Fargo called the "Alivia and T.J. Little Angels Foundation" and are accepting donations.

(KUSA-TV © 2011 Multimedia Holdings Corporation)