The UK’s top civil servant has privately warned Boris Johnson against forcing government workers back to the office amid growing anger in Whitehall over “scare tactics”, the Guardian has learned.

Alongside Simon Case, at least four permanent secretaries – the most senior civil servants in their department – are also understood to have raised alarm over government rhetoric designed to reverse the shift to working-from-home arrangements during the Covid pandemic.

Spot checks on office working by Jacob Rees-Mogg, including “sorry you were out when I visited” notes left on empty Whitehall desks, have seen the Cabinet Office minister branded “the milk monitor” by disgruntled officials.

Case told the prime minister over the weekend that Rees-Mogg’s strategy was unwise and that the language against civil servants was going too far. A No 10 source said Case was fully on board with a return to more face-to-face working.

On Monday No 10 backed Rees-Mogg’s approach amid a rift with Nadine Dorries over the office working drive, which the culture secretary had labelled “Dickensian”.

Rees-Mogg had written to cabinet ministers urging them to coerce staff into a “rapid return to the office” and has been leaving notes for absent civil servants with the message: “I look forward to seeing you in the office very soon.”

Among the rooms where he left calling cards was the national security secretariat in the Cabinet Office, causing particular disquiet in Whitehall because staff were under intense pressure in that unit, where they work on sensitive material.

A government source said Rees-Mogg often had to move through other departments to get to his office at 100 Parliament Street, adding: “He is wholly focused on parts of the government under his control, not other ministers’ areas.”

The minister, who is responsible for government efficiencies, presented figures to cabinet last week showing that some government departments were using 25% of office capacity in early April. Dorries’ Department for Digital, Culture, Media and Sport (DCMS) was at 43%.

Rees-Mogg is understood to have done walkabouts at several government offices, including DCMS. During a visit to the Treasury, the Guardian was told he commented to one official that “you’re the first person I’ve seen working on this floor today”.

He also visited the Serious Fraud Office (SFO) and was unimpressed with the amount of empty desks so sent senior its officials a letter asking when they were going to shut the office given its apparent lack of use, sources said.

But the Guardian understands the “threat” not to renew the lease has been abandoned. It was derided as a “scare tactic” to force teams across the government estate to justify using their space. A government source said: “The SFO have had a short-term reprieve and had their long-term requests for renewal denied by two separate ministers.”

Several permanent secretaries have also privately voiced frustrations at Rees-Mogg’s “stunt”. An official working at one of Whitehall’s biggest departments said their director of communications had advised: “ignore him”.

Another Whitehall source said civil service managers had promoted flexible working, with the expectation of spending one to two days a week in the office. On the civil service jobs site on Monday, more than 1,600 jobs were advertised as having flexible working arrangements.

The disquiet at Rees-Mogg’s strategy also extends to ministers. One said: “I’m on team Nadine [Dorries] on this one. Who dreams this crap up? If the civil service aren’t working properly, reform them. Don’t wander round placing the physical equivalent of eye rolls on their desks.”

Another Whitehall official said the push was creating a sense of injustice and warned that, coupled with a pay freeze, it would affect staff retention leading to “the worst brain drain I have ever seen”, with talented younger staff seeking to leave.

They also pointed to the treatment of junior staff in No 10 and the Cabinet Office over the Partygate scandal, where civil servants have received or expect to receive fines for attending events with the prime minister or senior bosses.

Dave Penman, general secretary of the civil service union FDA, said: “I am getting lots from members saying they have had enough and they are going to go. They are not going to be treated like this.

“Ultimately civil servants will put up with what’s going on with cost of living and a significantly below inflation pay rise – but not this. It is so demoralising for civil servants because it’s not actually based on anything to do with their work and the best way to deliver public services.”

A government spokesperson said: “There is total agreement across government on there being clear benefits from face-to-face, collaborative working and we know that this is particularly important for the learning and development of new and junior members of staff. The minister for Brexit opportunities and government efficiency has written to departments to underline the importance of workplace attendance and request that they review their existing guidance on the minimum number of days staff work in the office.”

Asked if Johnson backed Rees-Mogg’s policy of leaving calling cards at vacant desks, the spokesperson said: “What the minister is seeking to achieve is to do everything possible to get the civil service to return to the pre-pandemic level … That is supported by the cabinet secretary and obviously the prime minister.”

Referring to the notes left on desks, Johnson “supports any initiative that encourages people to return to pre-pandemic working”, the spokesperson said. “We are not talking about putting an end to flexible working, which continues to have a place in the modern workplace, we are talking about returning to pre-pandemic use of taxpayer-funded departmental buildings.”