Updated at 9:05 a.m. ET

Rear Adm. Ronny Jackson, President Trump's embattled nominee to lead the Department of Veterans Affairs, has withdrawn from consideration for the post amid allegations he had fostered a hostile work environment and behaved improperly while serving as the top doctor leading the White House medical unit.

In a statement Thursday morning, Jackson said, "Going into this process, I expected tough questions about how to best care for our veterans, but I did not expect to have to dignify baseless and anonymous attacks on my character and integrity."

He continued, "While I will forever be grateful for the trust and confidence President Trump has placed in me by giving me this opportunity, I am regretfully withdrawing my nomination to be Secretary for the Department of Veterans Affairs."

On Wednesday, Sen. Jon Tester, D-Mont., the ranking member of the Senate Veterans' Affairs Committee released a two-page statement detailing allegations against Jackson from 23 unnamed individuals.

The allegations included providing "a large supply" of the opioid Percocet to a member of the White House Military Office and wrecking a government car while driving drunk after a Secret Service going-away party. Jackson denied wrecking a vehicle to reporters.

In an interview with Fox & Friends on Thursday after Jackson's withdrawal, Trump said Jackson "would have done a great job" and "had a perfect record" and called the allegations against him false.

The Senate Veterans' Affairs Committee had postponed Jackson's confirmation hearing, originally slated for Wednesday, after accusations of misconduct surfaced against Jackson, who has served as the personal physician for the president under the Trump and Obama administrations.

In an interview Tuesday with NPR's All Things Considered, Tester said committee staffers "were told stories where he was repeatedly drunk while on duty where his main job was to take care of the most powerful man in the world. That's not acceptable," Tester said, noting the various allegations date back to when Jackson oversaw President Barack Obama's health care.

"Some of the exact words that were used by the folks we talked to were 'abusive toward staff,' 'very explosive personality,' 'belittles the folks underneath him,' staff that he oversaw. 'Screamed toward staff,' " Tester said. "Basically creating an environment where the staff felt that they needed to walk on eggshells when they were around him."

In the Fox interview, Trump blamed Tester for Jackson's withdrawal and said Tester "has to have a big price to pay in Montana." Tester is up for re-election in November, in a state Trump won handily in 2016.

Trump says he has someone in mind to replace Jackson, saying it's "somebody with political capability."

Veterans' Affairs Committee Chairman Johnny Isakson, R-Ga., and Tester had sent a letter to Trump on Tuesday requesting additional documentation related to Jackson's tenure as the president's doctor and his role leading the White House medical team.

The letter requested information about rumored Pentagon inspector general reports said to detail allegations into Jackson's conduct.

However, Jackson said there was no inspector general report about the allegations. "No, there was not," he said before brushing off questions about whether he would withdraw.

The White House put out a statement to make it clear that Jackson was reporting to work on Thursday in his normal post at the White House Medical Unit. But the questions about his conduct in that position may not necessarily go away just because he has withdrawn his name for the Cabinet post.

Jackson's initial nomination came as a surprise to many when the president announced his pick last month to lead the VA after the departure of Veterans Affairs Secretary David Shulkin, who had been confirmed with unanimous bipartisan support in the Senate.

While previous presidents had praised Jackson's medical care, questions arose almost immediately about whether he had the managerial experience to oversee a massive health care operation of more than 370,000 employees that serves more than 9 million veterans and has a budget of nearly $200 billion.

"Adm. Jackson is highly trained and qualified and as a service member himself, he has seen firsthand the tremendous sacrifice our veterans make and has a deep appreciation for the debt our great country owes them," Trump said in announcing his nomination.

But after the allegations against Jackson began to be reported by national media outlets this week, Trump said he wouldn't be surprised if Jackson wanted to withdraw given the rising storm, but that it was "his decision."

"I wouldn't do it. I wouldn't do it. What does he need it for? To be abused by a bunch of politicians that aren't thinking nicely about our country? I really don't think personally he should do it, but it's totally his — I would stand behind him — totally his decision," Trump remarked Tuesday during a news conference alongside French President Emmanuel Macron.

Ultimately, as he often does, Trump appeared to go with his gut in picking Jackson and it's unclear how much — if any — vetting the White House did before making the selection. Trump had become personally fond of Jackson, especially after he gave the president a glowing review after his first presidential physical earlier this year, even boasting that the 71-year-old president had such "great genes" that "if he had a healthier diet over the last 20 years, he might live to be 200."

Jackson joined the White House medical unit in 2006, helping care for President George W. Bush, and was formally named physician to the president in 2013 by Obama. A West Texas native, he attended Texas A&M University and the University of Texas Medical Branch, specializing in emergency medicine.

Copyright 2018 NPR. To see more, visit http://www.npr.org/.