Are most code bases in tech companies a complete mess?
Part of my work is as an expert witness when there are issues italic with code. Having been a developer on hardcore stuff but never going to write anotherpeting system means I know what Im doing but am not a threat. I will have to share the jargon of aggressive code reviews with you it not hard but as this stuff is not taught in college different people use different words. Im going to be talking here of large scale code something like Excel (which I debugged back in the day) an operating system (Ive worked on N3 of them) or the system that keeps track of where the 5th largest economies debt pile is today (yeah me as well). So that vast IT disaster you may have caused? Come in sit down s It is the way of all developers to see code written by anyone else as a mess that why we have code reviews because it is often easier to see flaws in other programmers code than our own. The majority of code is not primordial italic written by one person and put into use. It is deltaed i.e italic . many small changes by a sequence of people. I say sequence italic because that is the usual and good case. However if the code is modified by N1 people at a time chaos follows which is why we have version control. So the first source of problems at scale is delocalisation. italic Developer A checks out a file Developer B checks out a file that interacts with it somehow they mess with each other. One example that happened in a team I managed was where they managed to have two versions of the definition of a data structure one had a member the other didn. This meant 4 bytes got lost and over written randomly but rarely. It was a classic Heisenbug italic any attempt to insert code to find out where it was altered where the memory corruption went. That a bug in the small curiously delocalisation italic as a problem in large code bases is caused by trying to stop this sort of thing. You end up creating monstrous dependency trees perhaps using a CASE tool database or reams of documentation. The fear of breaking things far away italic means programmers or rather their manager can be excessively risk averse. Be clear that programming in the large is risk management italic getting the most of what you want (functionality performance) at the least risk of trashing the whole thing in a mysterious way that no one understands. Which leads us to people problems. Of all the s of software I have interacted with (I was an occasionallypetent Test Director at PC Magazine for a while Ive used a lot) italic the software Im most cynical about is Computer Aided Software Engineering (CASE) products. Management tend to like them because they produce pretty pictures and sound like they bring order when inexperienced management realise the code base has grown to be large. But ultimately there is the cadre italic of people who know where the bodies are buried. No they haven told management about these issues because management is not interested and nothing undermines your career prospects more as a developer than crying wolf when the wolf is real but probably far away. As an example saying thisponent is inherently single threaded which will hurt bad when we scale up italic is not good for your career. Inexperienced and crap managers want to hear solutions not problems and care more about delivery soon than quality delivery later. I know that not what they say doesn stop it being . In economics we call this a high discount factor. An amount of good this month is worth a lot italic more than next. This is not always a sign of crap management if youre a startup the clock is ticking loudly italic and it is much better to have revenue from a product that works but needs a re-write than to run out of money. So over time expedient italic changes i.e. kludges grow in number but they be harder to spot. I once had a nasty wound on my hand but it was covered in a white dressing so it may not be obvious that I had to seriously avoid stressing it. The effect of code reviews and coding standards is to make bad code look pretty. Comments meaningful variable names putting in the full set of error checking are all good things but if the code is inherently bad italic then what you have done is hide the wound and when you stress it blood will flow. Inherently bad italic code is something you don get taught about even I only give a lecture on it occasionally. IBCes in several forms. First is code that is statistically correct in other words the chances of thebination of factors that bring it down is acceptably small. The mostmon kind is mutlithreaded where in a simple example it takes longer to read a file over the network than to perform some calculation. Then of course the calc function is upgraded italic to get one of its inputs from a database and now you have a race. One I saw happen was where there was a bug somewhere else and logging to disk had been turned on. Local disk kept filling up so it went to a network file slowed down and misery followed. My own expedient code was inherently bad. A trading floor had a truly horrible C++ app that they really liked except it crashed a lot. italic A trader had read one of my articles on C++ and borrowed italic me to have a look at it. I offered to re-write it and guessed it would take 3 months at least. My actual boss flatly refused and wasn happy that I wasted even 15 minutes on it. I had a fix though. It was clearly a problem with freeing and deallocating memory my quick examination showed this but not where. So I wrote #define free(p) donothing() Yep I turned off freeing of memory. Program ran beautifully people actually cheered me when I went back to see how it went. Yes it did crash from memory exhaustion but that took a couple of days so the start of day process now included to stop and restart the app. Never did get the opportunity to actually fix it odds are it is still there. Never got paid for it either ( Although I was called a genius for writing that crap we both know I wasn being clever so we have another of inherently bad code that is trying to be too clever. Amon kind is absurdly deep class hierarchies in OO code where too much effort has been expended to make everything generic and flexible. Be clear I like flexible and generic code but I have finite time to understand and fix your code when it goes wrong. Odds are that I left a few wounds italic code bases like this over the years when I have not bought into the philosophy italic of the large code base enough. Yes wound italic is a word I sometimes use like code it can leak and may poison you if it goes bad. A wound italic is different to a bug. It is code that worked 1% of the time doing everything it should but stray outside that boundary and weird stuff happens. Some wounds are sloppy programming like a function that only works with a positive value of an input trusting too much that it will always get a positive and returning garbage if it is negative. Or it can be subtle like a memory allocator that is fast if it is called 5 times in a run of a program which usually only calls it 5 times but hits terrible performance problems if called more than 2 times due to fragmentation. Having a memory allocator run a bit slow is irritating as hell on a browser (are you reading this Mozilla?) but if it is controlling a machine it can actually kill people or at least damage the hardware. Resolving memory issues in real time is why the Java licence explicitly forbids you to use it for nuclear reactors seriously. What? You didn read the Java licence before agreeing to it? Shame on you. Philosophy italic is also a word I use. It a bit like patterns and in my own evolution as a software engineer it came before I ever heard of patterns. An obvious example we all know (don we?) is the way so many Linux entities are presented as files even when they are radically not. Problem with philosophies is that there are more than you may dream of and rarely are they explicitly documented in fact one way software development cultures go bad is where people whove been there a long time (where long may be as short as 18 months) use it as a way to deny newbies an understanding of the way things should be done so as to make the long timers look good in meetings. So we hear things like no don use message passing we don do that here with no explanation of really why. Don lie to me youve either done this or had it done to you good odds that it both. Another symptom of a code base in decay (like Windows) is functions with really similar names that do really similar things. There is the function MoveFile() code Nothing wrong with that. Then Microsoft added MoveFileEx() code Which is like Movefile but does more and other things. But there also the later MoveFileExA() code There many like this. Because as your code base grows you find that a function needs an extra parameter or is so screwy that you dare not change it (or internal politics stop you) and so to stop from breaking things you find it easier to simply write another function with a similar name. Typically youll cut an paste bits of the other and so you have the curious undocumented relationship between the variants that acts a slowly leaking wound in terms of maintenance. Im not going to spend too much time on internal politics here (though it has sucked up a big chunk of my life as a programmer) but officially or unofficially pieces of code are owned italic by managers or groups and not only will you be prevented from changing it but often from even using it. The brutal reality here is that if I write a piece of code that is useful to my project and you use it for yours then if it goes wrong I get hassle but no benefit for saving the firm as a whole some money. I can be dragged off my work to resolve your problem and if it turns out that it is my fault then I look bad and if your fault then you will still try to blame me and certainly won thank me for all the time it has sucked up. I typically call this an agency problem. italic Firms with large tech have often grown by mergers and acquiring whole teams. That means you have different styles of programming and code bases that were never intended to work together. The code to glue these together is about as ugly as it gets and is inevitably coded by people that can be spared italic from other work. Can be spared italic people is a key issue. If youve got a single team you allocate work items on a rough basis of who can do it and who has the time. This does not scale well with multiple teams. As you grow and projectse and go there will be people who aren so crap that you want to fire them (or they may be bootcamp graduates you can fire) so they are part of a team. But when a senior manager calls for staff for a job from his more junior team leads they are the people who as a good corporate citizen you can spare italic to give to the unattractive work. Ironically they often be the cadre italic above. If you write a lump of code to scan customer data to find those you can upsell to you can document it for the next guy. Same if a colleague in a different team created a tool that says in effect customers of this kind buy this sort of thing That mildly clever stuff to do well. But requires data to be sucked in from noisy data sources cleaned and joined up. Horrible work and both modules depend on it and require that the data is just so. italic Rarely is utility code italic so well documented because it is not business logic. italic Of all the phrases used in business software business logic italic is the one that I hate most. If the bloody code is not part of your business italic then why the hell is it in the system at all ? All code is business logic but the cool stuff that business management italic see output from gets the focus is better paid and is usually better engineered and documented. The problem with the cadre italic is that people leave and this soft information may leave with them because as any MBA graduate knows programmers are amodity and can easily be replaced by people who have done a coding boot camp. Ive only recently begun to see the effect of coding boot campers on major code bases. The current zeitgeist is that anyone italic can program. Really anyone. CompSci and other STEM degrees have the wrong demographic italic that looks less attractive in corporate publicity shots and there is a pay gap because actual programmers cost more and frankly were not very attractive. For the first couple of years this did little harm any large project has plenty of modules that are in effect isolated from anything else and where low productivity and lack of any deep software engineering skills don cause much harm. Also the turnover rate of boot campers is so high that by the time theyre not scared of the huge mass of code they know just enough to be dangerous. But there are now too many and if you hire people for any other attribute than to be able to cut code without breaking things you are buying trouble. AI Dinner in Whitehall on Creativity s
What was Silicon Valley like after the bubble burst in the early 2000s?
The collapse was sudden shocking and depressing . I think most of us saw iting but the speed and severity came as a hell of a shock. Take a seat around the fire pit youngsters and settle in for a longish answer from a Silicon Valley topic tid 959 native who labored in the trenches at two firms at the heart of the boom-and-bust cycle (parental discretion advised). By 2 it seemed like everyone I knew italic in their 2s and 3s was working for a dot. (That should have been a warning sign right there.) Not just San Francisco Bay Area topic tid 838 locals; new college grads and experienced business and technology professionals were migrating here from all over the country (and world) in the years leading up to the crash. Commentators famouslypared it to Florence during the Renaissance. The housing market was beyond italic ridiculous at the peak. I mean worse italic than the usual Bay Area ridiculous that's been the status quo since the 198s. (People were resorting to paying six months or a year's rent in cash in advance italic in order to get a decent apartment in San Francisco topic tid 835 .) Having moved back to Palo Alto CA topic tid 79 in 1998 I barely recognized my home town; every billboard up and down Highway 11 advertised some kind of Internet business with a silly name. I worked for a high-profile consumer Internetpany ( Excite@Home topic tid 12394 ) whose gleaming new headquarters itself served as a billboard of sorts looming over 11 in Redwood City CA topic tid 5353 with our logo visible to thousands of cars whizzing by or rather inching by; traffic during that period was as bad in the Bay Area as it was in Los Angeles topic tid 1383 with even I-28 turning into a pretty bad traffic jam most mornings as peoplemuted south from the City to thousands of newly created jobs at newly createdpanies up and down the peninsula. Leading up to the crash I worked in a job that was tightly tied to the state of the dot-conomy. As a lawyer in one of the corporate groups at Wilson Sonsini Goodrich & Rosati topic tid 12393 in Palo Alto the Valley's largest law firm from 1998-2 I contributed in my own little way (there were hundreds of us) to the IPO M&A and VC factory that was WSGR at that time. It was the best career decision I ever made because of the doors it opened but it also became sheer misery by late 1999 when everybody and their dog italic was trying to go public at once . When I left to go in-house to Excite (April 2 coincidentally the date of the first NASDAQ (exchange) topic tid 14239 market correction) my group alone (loosely defined to include two partners several associates and paralegals affiliated with Mario Rosati) was working on nine italic Initial Public Offerings topic tid 5599 (IPOs) simultaneously at various stages of SEC filing and review on top of all the usual Venture Capital topic tid 166 financings Mergers and Acquisitions topic tid 2251 and general corporate work. And we were a small grouppared to some in Larry Sonsini's wing of the firm! Corporate Finance topic tid 9423 is one of the most cyclical areas of transactional work around. When public markets are going strong everyone is in a rush to close public offerings while the window is open. When the window is shut nobody can do much but sit wait and about it (with the occasional fire sale of distressedpanies and assets). Of the nine IPOs we were working on that spring I think two italic of them (both life sciences not IT or Internet businesses) made it out later that year. The remainingpanies never went public. I think they either went under or were eventually acquired by largerpetitors. The IPO market has never been anywhere near that active in the decade since; I think our firm handled more tech IPOs in 1999 than there were in the entire country in 21-3. I was relieved to have joined a relatively large mature stable Internetpany in 2. Excite had been around since the dawn of themercial Internet in 1993 as a rival to Yahoo! pany) topic tid 2917 Lycos pany) topic tid 1491 and AltaVista topic tid 1538 . We had 3 employees and were doing $ 6 million revenue at the peak. I was hired to support the corporate development team in doing acquisitions strategic investments joint ventures and other fun deal stuff as well as publicpany governance and securities work. Unfortunately the slowdown was apparent soon after I arrived. Excite topic tid 84271 like all Search Engines topic tid 5584 and portals was supported entirely by Online Advertising topic tid 3728 . This was before paid search so it was mainly banner ads as well as an increasing number of rich media ads as it was already bing clear that users were getting weary of banners and Click-Through Rates topic tid 627 plummeted. Keep in mind that in that era online advertising was still relatively unproven and big brand marketers spent only a tiny fraction of their budgets online. Our business depended heavily on ads bought mostly by other consumer Internetpanies that were all spending venture and IPO money like drunken sailors in a land grab rush to gain first-mover advantage in their respective markets . When the capital markets clamped down in early 2 and VCs started getting cold feet as they saw signs of delayed exits and lower valuations the flimsiest of the dots with no revenue and crazy burn rates started going under. Their ad buys shriveled up along with them driving CPMs way down as the amount of available ad inventory kept growing while demand was shrinking. In the third quarter of 2 online advertising shrank italic overall for the first time dropping 6.5% to just under $2 billion. The average price of a banner ad plummeted from a $5 Cost per Thousand Impressions topic tid 185937 to less than $5. (See ) This exs in large part why Yahoo!'s stock price went from nearly $1 when I interviewed there in early 2 to under $1 a year later. The dominos fell pretty quickly. Progressively larger dots went under each one casting a bigger shadow on markets for labor Commercial Real Estate topic tid 468 Network Equipment topic tid 25882 Enterprise Software topic tid 1736 Advertising and Advertisements topic tid 2484 and all kinds of professional services. Equipment makers infrastructure providers and landlords soon started getting hammered as failed startups liqated everything dumping tons of barely-used equipment on the market (notoriously Herman Miller Aeron Chairs topic tid 11798 as well as the expected PCs servers and network equipment). Large corporations started downsizing as well reacting to poor macro conditions meaning they dumped millions of square feet of vacant office space on the market for sublease just as demand evaporated. I and everyone I knew started reading Philip Kaplan user 124 's on a daily basis. (Ironically FC is no longer active but you can see its own epitaph at .) Gallows humor pervaded for a while people held pink slip parties to make the most of the situation and begin networking in search of the next job but it soon became apparent to many that there were far too few jobs to be had. Stories appeared in the newspaper of net outmigration from the San Francisco Bay Area topic tid 838 for the first time in living memory; there was a U-Haul shortage (no joke) as people rented them for one-way moves back to Las Vegas topic tid 1248 or Dallas TX topic tid 1817 or wherever the economy was healthier to get a regular job. Enterprising promoters held raves in the abandoned former headquarters of failed dots (including ours). Bankers lawyers and accountants who'd struggled to hire fast enough during the boom found themselves with a large number of highly paid workers with nothing to do. From SAP ARIBA topic tid 2469 to VA Linux Systems (now Geeknet) everyone's stocks got clobbered. The NASDAQ Composite Index topic tid 21931 chart is worth a thousand words n In a lovely coincidence of timing services like E*TRADE Financial pany) topic tid 2578 *TRADE-Financialpany had brought discount stock trading to the masses and retail investors bought into risky dot stocks at highly inflated valuations only to see many of them lose 9-99% of their value within a couple of years (1% for the bankruptcies like Webvan topic tid 5452 ). Unemployment plus heavy capital losses plus high cost of living equalled financial disaster for many people. I knew many couples where both partners were laid off from dots within months of each other. (One highlight was representing a friend at an EDD hearing where she had to fight for her unemployment benefits.) On top of everything else the terrorists attacked in September 21 and the broader financial markets panicked. It had mostly been a tech meltdown up until that point but pretty much everyone's italic stock was killed in late 21-2. Back to Excite@Home we were busy selling off assets throughout 21 hoping to pay down debt and refocus on the core @Home residential broadband Internet Service Providers topic tid 1256 service. The conflicts of interest and lawsuits are fodder for another post but suffice it to say that it was too late to turn the Titanic. We filed for Bankruptcy Law topic tid 8542 within weeks after 9 Attacks topic tid 1277 making for a thoroughly depressing fall and winter. Thepany shut its doors in March 22; as part of the legal team handling the bankruptcy I was one of the last 5 or so employees left standing less than two years after joining apany of 3. One of the most poignant scenes was the Network Operation Centers topic tid 2868 at 55 Broadwayonce the pride of thepany as nerve center of a pioneering nationwide fiber-optic backbone a parallel Internet with distribution and caching architecture to push large files as close to customers as possible. (This was designed by CTO Milo Medin before Content Delivery Networks (CDNs) topic tid 5868 ( CDNs ) like Akamai Technologies pany) topic tid 3162 became significant players.) A beehive of activity that resembled NASA Mission Controlappropriate considering Medin's roots as an engineer at NASA Ames Research Center topic tid 26571 in Mountain View CA topic tid 914 the NOC was shut down along with the network after we'd transitioned the last of the cable providers off our broadband network. nThere was a countdown to the shutdown and then everything was switched off. It felt like pulling the plug on a brain-dead patient in the hospital ICU. After the last ops and engineering staff were laid off I remember wandering through the cavernous deserted NOC strewn with assorted equipment that was left to be carted off to auction looking up at the huge projection screens that forlornly flickered No Signal italic . It felt downright post-apocalyptic with huge diesel generators out back standing ready to provide backup power to a NOC and data center that no longer needed ita few short months after the summer of rolling blackouts that confirmed the necessity of such uninterruptible power sources. n But I digress. Bottom line within a year after our wedding both my wife and I were laid off from failed Internetpanies. Grateful not to be saddled with a mortgage we gave notice put our possessions in storage and left town like thousands of others uncertain at the time whether it would be temporary or permanent. Although this predated the term fun italic employment we spent a month on a cross-country road trip trying to dpress while I applied for jobs like crazy and called every recruiter I knew. There were absolutely no jobs in the Bay Area for a corporate deal lawyer like myself so I cast a wider net. Eventually I landed a good in-house job at a large techpany Gemstar-TV Ge which was a relief. The only hitch was that it was based in Pasadena so we had to relocate to Southern California . So I guess we did our part to ease the traffic and housing conditions in the Bay Area. (You're wee.) I didn't mean to turn this into an autobiography but I think my own experiences (and those of my friends and family) were representative of conditions at the time. (My folks have lived in Palo Alto and Menlo Park since 1975 and added some perspective.) The Excite@Home topic tid 12394 headquarters sat empty for five years italic before Stanford finally bought it in 27 and turned it into an extension of its outpatient medical clinic. Overall the recovery was S-L-O-W ining; it wasn't until around 23-4 that things started seeming good again. Let's hope they never return to the overheated state of 1999 What goes up muste down.