Coase's theory of the firm is based on transaction costs. And that is important. But the other point (which Coase deals with to some extent)staying within the boundaries of a firm or set of closely related firms is important when what you want does not fit into nice neat categories. Coase's example is very abstract - the buyer will want to
contractor to take one of several course of action; the contractor is indifferent as to which course of action; the buyer does not know in advance which course of action he/she will need the contractor to take. A contract on these lines is very complicated. It is one of those bloodless stories that are so common in economic textbooks. I think the way this really happens makes it as important as transaction costs.
It is much messier than the 'will need one of several alternatives" To start with real life uncertainty is not so nicely sliced as "I will need one of these, but don't know which one".Often the manager is not 100% certain of what she is trying to accomplish. Software of course is a classic case of this. If you can find a ready made package that does enough of what you want you are better off buying it and living with its limitations than trying to custom code. If you can find something that almost does enough of what you want and can do the rest with a bit of customization, that is 2nd best. But if you really need full custom coding, there are going to be all sorts of unexpected discoveries in terms of how this need affects that need, and what the hell else you need. You don't want someone just trying to meet the letter of a contract. You want someone try to fill your needs, which includes discovering them as they go along without using the discovery process as an excuse for screwing you.
And something else important is going on. There is rival theory to transaction costs, the idea of teams. But in the narrow sense you only get the benefit of team efficiency in very small groups;only in small groups can people know each other well enough to gain huge advantages that team theorists usually refer to. But there is a much simpler benefit to in-house. People tend to identify with their workplace, even when it is a very large workplace. Managers who squeeze their workers the maximum possible often do their best to kill this identification, and I think austerity has gotten so horrible that it happens less than it used, but it is still a real phenomenon. For many, their workplace is one of their tribes. Many workers, even if they hate the boss or the owners, still think of the corporation they work for as we.
And that is something you get inside a firm that is damn hard to get with outsourcing. You get people working to meet unstated goals, or produce "quality" work ( scare quotes because outside of management jargon, the meaning of quality is highly context dependent. You will even sometimes see people take the huge risk of trying to give the boss what he needs rather than what he wants. ) Yes that means workers are contributing to their own exploitation. And I don't want to oversimplify: lots of workers get mad at being screwed and screw the boss back when they can. But if you are a boss and have a vague or difficult to define goal, or worse yet a goal you need to partially discover in the course of meeting it, you have a hell of a lot better chance of succeeding using people who you employ directly who see themselves are part of your business than using outside contractors.
I won't say that never happens with contracting. I know one woman who contracted with an illustrator. That illustrator ignored several terms of the contract to give much better results. But the amount of mutual trust that took. I mean the woman who hired the illustrator could have insisted that the work be completely redone. For that matter, the woman took a risk in accepting work that differed from the contract. Well, they were personal friends and understood each other well. But how often, even among case where it is directly one person contracting with another does that happen? And how in hell can it possibly happen when one corporation is hiring another. (Of course it can happen in a very negative way. The executive in charge of hiring is personal friends of the someone who works for the contractors, and allows the contractor to screw his corporation. Business gets done that way all the time. The formal terms for this is principal-agent conflict.)
Now of course this is still oversimplifying. Firm 'boundaries" are kind of fuzzy. Corporate subsidiaries. Sub-contractors with only one customer - in practice subsidiaries. Sometimes really long term relations between a business and a sub-contractor can result in them being on the same "team". Also context is everything. In cases of class conflict, elite class solidarity can trump major conflicts between businesses.
But accepting that it is a rough approximation, I also think it is a significant phenomenon. One big advantage of having work done in-house rather than contracted is simply that the odds of the people doing the work identifying with the company the work is done for is much much greater.
That is not to downplay one of Coase's points. It is easier for managers to monitor people who work for them than people who they contracted with. With an employee, a manager does not need to prove it is not being done right; a manager can simply order an employee to change course or do something over. Often the difficulty of developing meaningful metric is used as an argument against Coase on this issue, But in truth, even without meaningful metrics a manager can sometimes tell whether good work is being done or not, just through experience. It is where meaningful metrics don't exist that the transaction advantage of direct hires over contracting is largest. Of course the lack of direct metrics results in huge problems when the manager is wrong or malicious. But when the manager is trying to accomplish the owner's goal, and knows what she is doing, direct hire is a huge advantage for the owner over contracting.
And when the manager is malicious or incompetent, direct hires are still an advantage for the owner, because the manager can't blame problems on the contractor. In fact ,avoiding responsibility is a big reason managers often like to hire contractors. When a manager knows a project is high risk, let someone else take the blame in case of failure. Unofficially, the deal when a manager hires an outside contractor is often that the contractor will be allowed screw the hiring company big time, in return for acting as scapegoat for the failure that follows.