August 21, 2022

Digitizing 55,000 pages of civic meetings

It is very hard for the average resident of a U.S. city to know what’s going on with their civic government. It’s even harder for them to get any sort of historical context on why things are the way they are. Let’s take my hometown, the City of Alameda. Six months ago if you wanted to know which city meetings had discussed rent control, your options were:

Have a friend who is a constant watcher of Alameda meetings / the #alamtg hashtag and could tell you
Go through every meeting minutes on the Alameda Legistar and hope you figured it out

This is pretty common across a lot of civic government. I don’t think municipalities are willfully trying to hide this information from residents, and I don’t think it’s ineptitude. I think most cities, even the large ones, are understaffed, and without a concerted push it’s hard to make “visibility of city documents” a priority.

It hurts and helps that many cities use Legistar for both meeting management and the complete historical archive. Legistar is only designed for one of those use cases, and it’s not archiving. But the thing that Legistar does provide is an API, and one that’s reasonably sane for fetching a city’s meeting archives.

Because of that API, there’s two tools that have come online for Alameda (and now Oakland) in the past six months that have made it much easier to answer the question: “which meetings in my city have recently talked about rent control?”

The first tool is the Council Data Project. CDP is a truly incredible effort and tool, run by some fantastic people (Hi, Eva!). I’m not going to talk much about it, other than to say you should check out the Alameda and Oakland instances.

The second tool is something I’ve been working on recently: SQL-backed full text search of city meeting minutes. You can see this working for the cities of Alameda and Oakland right now. That’s 18,746 pages of city minutes for Alameda, and 37,172 pages of city minutes for Oakland, now fully searchable by anyone.

So let’s talk about how I did this, and how you can do this for your city, possibly with my help!

Here’s my process:

Figure out where official city minutes are hosted
Write a script to fetch and format those city minutes
Upload all the fetched minutes to AWS S3
Run s3-ocr across the corpus of minutes
Download the ocr’d pages into a sqlite DB
Deploy a datasette instance to fly.io with that sqlite DB.
Post to twitter so people know about it.

Datasette and s3-ocr are wildly powerful tools written by Simon Willison that will change the landscape of preservation and archiving over the next few years. I’m focused on civic data, but these tools were developed with historical records in mind, and if you haven’t dug into the ecosystem yet, please do so.

So what’s actually happening, step by step? Let’s take Alameda as an example, because Oakland followed a fairly similar pattern. All the code (and all the PDFs) for both cities are available on Github at phildini/alameda-data and phildini/oakland-data

First we need to find all the city minutes. The Legistar API has an Events endpoint that will give you all the city meetings in Legister. Except not quite, it’ll cap at one thousand, so we need to go year-by-year and fetch all the meetings for that year. For Alameda, that means going back to 2004, for Oakland it’s 2003.

for year in years:
    print(f"Fetching {year}")
    event_listing_url = f"{base_api_url}events?$filter=EventDate+ge+datetime%27{year}-01-01%27+and+EventDate+lt+datetime%27{year}-12-31%27"
    list_response = requests.get(event_listing_url)

Once we have a list of events, we pull the date out of the Event object, parse it a little, and check to see if we’ve already fetched this set of minutes.

for event in list_response.json():
    body = event["EventBodyName"]
    directory = f"./data/{body.replace(' ','')}"
    if not os.path.exists(directory):
        os.makedirs(directory)
    filename = event["EventDate"].split("T")[0] + ".pdf"
    filepath = f"{directory}/{filename}"
    if os.path.exists(filepath):
        continue

If we haven’t fetched it, we get the meeting listing url, then fetch the page with BeautifulSoup, grab the link to the minutes, and pull it down.

page_url = event["EventInSiteURL"]
page_response = requests.get(page_url)
soup = BeautifulSoup(page_response.text, "html.parser")
minutes_url = (
    base_url
    + soup.find(id="ctl00_ContentPlaceHolder1_hypMinutes").attrs["href"]
)
minutes_response = requests.get(minutes_url, allow_redirects=True)

With the PDF response ready, we write the file and move on.

with open(filepath, "wb") as minutes_pdf:
    minutes_pdf.write(minutes_response.content)

For Alameda, the script takes an hour or two. For Oakland, about four hours. Those times are both first-run times; because the script is idempotent, we won’t re-download a file we’ve already downloaded. On successive runs, the script completes in minutes or seconds.

With all the PDFs downloaded and sorted, we upload to S3. Because we’re using s3-ocr --all, each city gets its own bucket.

The OCR process takes about half an hour, but that’s again a first run time. All the parts of this process are idempotent, so running s3-ocr successive times will only operate on new files. Each new file that s3-ocr discovers on a run gets a job created for it, and when all the jobs are done we run s3-ocr index city_minutes.db. Indexing all the pages into sqlite databases took about two hours for Alameda, and about four hours for Oakland.

Once the index process is done, we upload a datasette instance with the DBs to fly.io, a stunningly good PaaS that makes the deployment as easy as:

datasette publish fly city_minutes.db --app="alameda-datasette"

While (so far) hosting these datasets on fly.io is free, the project itself has some costs. The OCR used by s3-ocr is AWS Textract. Running s3-ocr against the 18,746 pages of Alameda city minutes cost about $40, and the 37,172 cost about $80.

With that process and pricing in mind, let’s talk about Berkeley and San Francisco. The Events endpoint for the San Francisco Legistar instance is currently broken, probably because of how far back the San Francisco data goes. I have a lead on how to get around that, but it’ll still cost some processinig time.

Berkeley is a fun case. Berkeley already has full-text search of it’s city minutes, but that search doesn’t return the text, it only returns the PDFs. Using that, I was able to get a huge chunk of Berkeley city minutes downloaded. How huge a chunk? 9700+ documents, in some cases going back to 1905. I would really love to get these digitized, and be able to present this century-long searchable record to the City of Berkeley, but my estimates are that it’s going to cost $350 to run s3-ocr against the full set, and that’s more than I can bear on my own right now.

This is where you, the reader, hopefully come in. If you like what I’m doing here, I’m accepting donations via Venmo (@phildini) and Square Cash ($phildini). These donations will go straight to the AWS cost of processing more city minutes data. If you know someone in Alameda or Oakland or San Francisco or especially Berkeley who wants to see this information be accessible, please share this with them. If you know a political activist or journalist who could use this data, please share it with them. If you think of cool uses for this data yourself, please share them with me.

And if there’s data you want exposed to the sunshine like this, for your city or others, reach out to [email protected].

data politics datasette pdfs civics

April 6, 2022

Things that surprised me about golf

The Athleticism

So, it turns out golf is a sport. By which I mean, its a real sport, a competitive physical activity that taxes the body and mind. I’m writing this on a Wednesday, three days after my first golf outing as an adult, and my back and forearms are still feeling the effort it took to complete a 9-hole round.

I may not like it, but I am not what peak performance looks like, and yet I am capable in the things I thought necessary to be good at golf: I can walk for days and I can juggle for hours without feeling too taxed. My surprise stems from the fact that golf is a sport sterotypically enjoyed by old white men, who are not normally associated with athletic prowess. Hell, the 45th President of the United States is famously obsessed with golf, and he seemed like a man ready to keel over in a strong wind. Practice, or “knowing the correct way to play golf”, probably helps here, but the amount of workout I got from 9 holes still surprised me.

The Experience

My dad, who was quite fond of golf in his way, had a little plaque that said “Golf is a nice walk ruined by a little white ball”. Sometime around my transition to “teen”, he tried to take me golfing, and I think utterly despaired after my first 9 holes of ever taking me golfing again, but that quote has stuck with me throughout my life.

Golf is a nice walk! It’s actively pleasant to be out in the sunshine, normally in a little secluded patch of nature, walking with a friend, and the sport itself gives a conversation topic that feels easy. “What club are you thinking for the next hole?” “That last shot, eh?” “Look at that bird over there.”

Around the 6th hole, my golfing partner and I realized why the trope of sales people and executives being obsessed with golf exists. If you think of yourself as a person who is always busy, golf is a nice forced break in your day, out of the office and with some buddies. If you’re trying to sell someone on something, golf gives you at least an hour of time with that person, where you aren’t really competing with them because you’re both too busy competing against yourselves. You now have a shared experience as the basis for a relationship that can turn into something beneficial for both of you.

This realization surprised both of us, I think, because we are both engineers and have our own learned biases around salespeople. I certainly thought the trope existed as just another form of privilege, a sense of “I can take off part of my work day and go play around because I’m charming or powerful.” That isn’t untrue, but there is also a core of a good idea around building relationships on the golf course.

And, of course, it’s very easy to make the other person look good by deliberately being a little bad.

The Competition

By competition, I don’t mean “competing for a score against whoever you’re playing with”. That drive to be better than my golfing partner existed at the beginning, because I have a small competitive streak, but halfway through the round something changed. With each swing, as I learned from first principles the mechanics of golf, as I moved from absolute ignorance to seeing the foothils of understanding off in the distance, I started thinking about how to consciously improve my skill not because I wanted to be better than my partner, but because I wanted to be better at golf. I wanted to stop landing deep in the rough, then stop just off the green, then start landing closer to the hole.

I realized, somewhere between the 8th and 9th hole, what the real attraction of golf as a sport is. There are few sports we play where you can have anything that resembles a perfect game. Team-on-team sports are right out. There are too many variables for any player to play a platonic ideal perfect game in football or soccer or basketball or hockey. In baseball, only the pitcher can play a game perfectly, and even then they have at least 9 opponents who are there to thwart their attempt. Golf is in a class with only maybe bowling or certain races. The only thing between you and a perfect game is you, your mastery of yourself, the sport, and how those combine to overcome the elements you play in.

Golf is the only sport I’ve experienced that combines true athleticism with a competition that is solely against yourself. I find myself wanting to play the same course again, because I see how I could improve, and the improvement will itself be a reward.

The Issues

Golf is a waste of space. Especially for private golf courses, golf is land that could be better spent on just about anything else, especially in the parts of California that are under a major housing crunch. The public municipal courses are maybe better, because any member of the public can use them, but also maybe worse because they are now using public land for a sport that only a few can enjoy at a time, and using public dollars for the upkeep of that land.

Addtionally, golf is still a deliberately, institutionally exclusive club. We were both surprised at the number of golf courses, even the ostensibly public municipal courses, that don’t rent the equipment needed for using public land. The course in Alameda, where I live and my first thought for where to play, does not rent anything as far as we could tell, and we ended up going deep into the Oakland hills to find a place that would rent us clubs. A football, baseball, soccerball, or basketball can all be had for under $20; the cheapest set of clubs you can find on Amazon is over $100. Golf is exclusionary.

And yet, my biggest surprise of all, knowing the issues around land use and privilege and the long history of weaponized exclusivity, was that I enjoyed it. Every fiber of my being was ready to write off golf as some dumb thing rich white people do. To my chagrin I started really enjoying myself after a couple holes.

If every golf course was tomorrow turned into housing or a public park or some sort of open nudist recreational area, I would shed no tears. But while they exist, I surprise myself by wanting to go golfing again.

golf sports business surprise

April 12, 2021

Courage

I have the courage of my convictions, but not the courage of my perceptions. By “courage of my convictions”, I mean that my default is “strong opinions, loosely held” — I will defend my position, but try to be very open to being proven wrong. Occasionally, I will put a real stake in the ground on something I believe in, because it is fundamental to who I am or because I have done the research or have the experience to have the confidence in the position I’m taking.

Courage of my perceptions is the trickier one, because I’m using “perceptions” in two ways. In the first way, perceptions means “that which I am perceiving”, and for a lot of topics that’s inherently suspect. I am a white man from California. When I observe events in the world involving people who don’t look like me or have my background, my perceptions are suspect because the society I live in has trained me to view those events in a certain lens. Deprogramming bias from my perceptions of human interactions has been a large part of the background work of my adulthood, and that work will continue until I die.

But I’m also using “percepions” to mean “how I am perceived”, and this is a facet of my personality that brings me anxiety. When I present something about myself, or when I take a position on a topic in front of people I care about, my anxious brain kicks into overdrive, trying to anticipate and analyze the responses to see how my courage of conviction is landing.

If you’ve ever talked to me, and had the impression that I have used a hundred words when ten would do, then you have seen the effect of this. My defense mechanism is to produce a deluge of supporting information to any position I take, so that I will seem at least reasonable in my convictions and therefore be accepted by the people I’m being vulnerable around.

What do? I’m someone who believes that our minds, if they can be changed, can only be changed through a change in behavior, so I’m attempting to change my behavior. For those of you who know me, this will probably be a net positive: I’m going to try talk less, and smile more.

brain personality anxiety

October 20, 2020

2020 Alameda Voter Guide

Note: The State of California calls it’s ballot initiatives “Propositions”. Alameda County lists them as “Measures”. I’m speaking to Alameda County here, so they’re listed as Measures.

Summary

President: Biden/Harris
13th Congressional: Barbara Lee
9th State Senate: Nancy Skinner
18th Assembly: Rob Bonta
Judicial: Elena Condes
Peralta Trustee, Area 1: No Endorsement
AUSD School Board: Heather Little, Jennifer Williams, Megan Sweet
City Council: Malia Vella, Jim Oddie
Auditor: No Endorsement
Treasurer: No Endorsement
AC Transit Director At-Large: Victoria Fierce
Measure 14: Yes
Measure 15: Yes
Measure 16: Yes
Measure 17: Yes
Measure 18: Yes
Measure 19: Yes
Measure 20: NO
Measure 21: Yes
Measure 22: NO
Measure 23: edit No Endorsement
Measure 24: edit NO
Measure 25: Yes
Measure V: Yes
Measure W: Yes
Measure Z: Yes
Measure AA: Yes

The Weeds

President: Biden/Harris. Hopefully the Presidential choice is a no-brainer if you’re reading this. If you’re undecided, I’m confused. If you don’t like this democratic duo, then I hope you’ll join me in voting them in and committing to spending the next four years hell-raising to keep them moving left.

Congress: Barbara Lee speaks for me. That’s it that’s the tweet.

State Senate: Nancy Skinner Skinner has pushed for youth programs, criminal justice reform, and climate action. She is a solid advocate for this region and the state.

State Assembly: Rob Bonta Bonta continually pushes for civil rights, equality under the law, and programs that lift up the least enfranchised. His record would be impressive anywhere, and is doubly so for coming up through Alameda politics.

Judicial: Elena Condes I want as a judge someone who has seen what happens to people in the system, and has had to fight prejudice and injustice and systemic bias their whole life. Her opponent, Fickes, also seems like a standup advocate, but Condes has just that little bit extra that gets my vote.

Peralta Trustee No endorsement here because every time I’ve heard Bill Withrow speak I’ve been convinced that his eye is on Peralta’s bottom line, and not on students or faculty. I have trouble endorsing Heyman because there’s nothing about him in the voter guide.

AUSD School Board I am the son and grandson and great-grandson of public educators. I will always side with teachers, and the teachers’ picks. That means Heather Little, Jennifer Williams, and Megan Sweet

Alameda City Council Here’s the hardest one on this list. Reading my last post on what a City Council is would be good prep for this section.

Let’s start with the easy part of it. Malia Vella is a shoe-in. She is possibly the most dedicated public servant I have met, and has received nothing but garbage from the worst parts of this city. After the kerfluffle with the fire chief, she was served suits multiple times in an attempt to recall her, suits she had to defend out of her own pocket while trying to have her first child. It says something rotten about Alameda that she has received so much abuse while others get wrist-slaps in the press. Beyond that, her voting record and activist work speaks for itself in her tireless defense of the most vulnerable.

Jim Oddie is harder, for me. I like Jim. I want to get beers with Jim. I went to school with Jim’s daughter. But Oddie’s record is… not spotty, but confusing. He has a tendency to run towards the center when pushed, especially in oral statements at council. I wish he would commit more, and his handling of certain gaffes, like the Fire Chief kerfluffle, leave a lot to be desired.

But his opponents are awful, so he’s a comparatively easy choice. Gig Codiga is a Republican who, based on his campaign website, might be keeping a red hat in his closet. Amos White is a nice guy, but I can’t for the life of me tell what he stands for or what he plans to get done. Trish Spencer continues to fight tirelessly for the rights of the aggrieved money classes, the landlords and realtors of Alameda, and has a track record of making sure teachers in Alameda know their place at the bottom of the heap.

Vella, Oddie, in that order.

Auditor/Treasurer No endorsement. On the one hand its fun and small-town and cute and folksy that these two brothers have been doing this job for who knows how long. On the other hand, I like turnover in elected officials. Keeps everyone sharp.

AC Transit Director At-Large: Victoria Fierce Chris Peeples has been a thorough administrator and dedicated advocate for transit for decades, and there is a soft spot in my heart for him since we went to the same university and the same college in that university.

I wish Chris a long and well-earned retirement when Victoria Fierce takes his seat. I’m proud to be a friend of Fierce, and think she will do incredible things as a herald for a new age of transit centricity and advocacy. Fierce brought her activism from Ohio to Oakland, and the landscape looks different as a result. As one of the founding members of East Bay For Everyone, Fierce has helped raise the level of discourse around housing in the East Bay, while always remembering that transit has to be part of housing justice. I’m excited to see what she’ll do on the dais.

State Measures

Measure 14: Yes My family has a history of Alzheimers, you bet I want more brain disease research.

Measure 15: Yes See the note above about educators, now read it louder.

Measure 16: Yes The current system isn’t working, let’s try another one.

Measure 17: Yes When people have served their time, we should hand them a job, an aparment, and a ballot. These people are citizens, it was criminal that we took the right to vote away from them in the first place.

Measure 18: Yes I’m in favor of lowering the voting age to 16, so 17-year-olds voting in primaries seems like a step in the right direction.

Measure 19: Yes Our property tax law is broken and makes everything else we do harder. I wish this went further, but it’s a start.

Measure 20: NO I’m pretty sure we as a society want more restorative justice, not more prisoners.

Measure 21: Yes More rent control sounds great!

Measure 22: NO Hell no. Uber and Lyft treat their drivers like employees, and should have to actually provide them those protections.

Measure 23: No Endorsement ~~Yes? No? I’m on a knife-edge for this one, but the way it’s worded it might be fine? Feel free to reject my endorsement here.~~ Edited because some friends pointed out some things I missed, and now I think it shouldn’t even be on the ballot.

Measure 24: NO ~~This isn’t a perfect privacy law. There is no perfect privacy law. This is at least getting people to think about it.~~ Edited after some actual privacy experts changed me thinking.

Measure 25: Yes Cash bail is awful. The suggested replacement is only maybe less awful, but at least doesn’t fund the predatory bail bond industry.

County Measures

Measure V: Yes Yes? I guess? The unincorporated areas need services, so… they should collect taxes to pay for those services?

Measure W: Yes Half a percent sales tax to provide services to the homeless seems good? I wish it were a parcel tax, but you get what you get.

City Measures

Measure Z: Yes HELL YES. Alameda has to build housing. Stylish multi-family housing close to transit sounds way better than sprawling into the Navy Base forever.

Measure AA: Yes It’s a bunch of kitchen-sink charter changes for Alameda. Sure.

politics elections

October 13, 2020

The City, Inc. board of directors

This post owes its inspiration, as so many posts do, to a tweet from patio11. I have tremendous respect for Patrick, and enjoy every conversation I have with him, but in one particular case I think he’s wrong, and he’s wrong in a subtle way that took me a long time to understand, because it goes against our idea of what an elected official in the US does, especially at the local level.

I’m going to put this in corporate tech-ish business terms, because that’s my bias and most of my readership: Your elected officials aren’t individual contributors. They aren’t even management. They are the Board of Directors.

This might seem obvious, but I really want to play it out. In most municipalities, especially those with a weak Mayor / strong City Manager government, your City Council or Board of Supervisors aren’t even members of the executive team for your city. They have, effectively, the same powers as large corporate boards do: They can vote on policy initiatives, they can authorize spending money, and they can hire or fire the CEO, who in your city is probably the City Manager.

Where I live, in Alameda, we have strong CEO / weak Chair of the Board governance, by which I mean a strong City Manager / weak Mayor system. The reporting chain for the City Staff, who are the people actually spending the money and enacting the programs and doing the work that affects us as citizens directly, goes up to the City Manager. That City Manager wields enormous power in how this money is spent and who the City is hiring to do the spending. The City Manager is hired, reviewed, and fired by the City Council. The City Council cannot themselves hire or fire any City Staff, and they themselves cannot spend one dollar of the City’s money directly.

Why does all this matter? Because it means that the qualities we look for in our elected officials should be management qualities and values qualities, more than effectiveness or expertise as an Individual Contributor. The job of a City Council member is a management job. When a candidate says they deserve our vote because they’re going to “get things done” at City Hall (or the Statehouse, or the Capital) we should remember how they will actually get things done: by suggesting policies, and building coalitions with their fellow elected officials to pass policies, and by hiring or firing the Chief Executive.

As an aside, the one way that government differs from most corporate structures is there are often multiple Chief Executives who can be hired or fired. For example, the City Manager, the Chief of Police, and the Fire Chief are all hired, fired, and reviewed by the City Council. In this case, those three departments operate like three different companies with the same board and funding source, so the analogy still holds, I think.

For me, this means that it often isn’t in my best interests to vote for the person who completely aligns with my values, even if such a person who isn’t me could exist. It also means it’s not in my best interests to vote for the most incredibly effective, deeply expert technocrat. My winning strategy, and I suggest the winning strategy for most of us, is to vote for people who:

Mostly align with our values
Have a demonstrated capacity to learn new information and make good decisons based on that
Have a good head around managing people and collaborating with peers

Candidates who have a Master’s in Public Policy or Public Adminstration are great. But in the same way I’d hire an engineer from a bootcamp with seven years of experience over a freshly-graduated Master’s, I’ll take the activist who’s been coalition building and running an NGO for a decade over the first-time-running technocrat every day.

When you vote for a representative, at the State, Local, or Federal level, you’re voting for the Board of Directors of the organization that controls where you live, in a way you can very rarely vote at work. Vote for the Board you wish you had.

politics management elections

July 14, 2020

Where are we going? Tech and the Bay Area in 2025

In 2017, I attended one of the first Arena Summits in Raleigh, NC. Arena is an organization that formed after the 2016 election to try to solve a problem: The Democrats in 2016 had no “bench”. Bench is a sports term, here, and if you spend any time in politics you’ll know that the American political community adores sports metaphors. When we say that in 2016 the Democrats had no bench, what we mean is that Democrats were finding it hard to field candidates for national office in rural areas because young Democrats weren’t running for state and local office in rural areas. “Young” here has that particular American politics definition that means “under 40”, but the point stands. It was hard, in 2016, to field qualified Democratic congressional candidates in some regions, because there weren’t many qualified Democratic State Assembly Candidates. There weren’t many qualified State Assembly candidates because there weren’t many Democratic County supervisor or city council candidates, and so on all the way down.

This is one of the side effects of the massive coastal-and-urban concentration of well-paying jobs for college-educated folks. One of the ideas that Arena presented to counter this was “Go Home”. Throughout the summit, there was an exhortation to figure out some way to leave the major urban centers and return from whence you came, especially if you could be convinced to run for local office there.

I think there’s a lot of merit to this idea, but it had some cognitive dissonance for me, because I am from one of the major liberal urban strongholds: I grew up and still live in the Bay Area. I could, and did, get involved in local politics. But the idea of making a huge difference by moving to where I was born and advocating for the liberal values of “all men are created equal” wasn’t going to have the same impact in a region that started jumping on the Sanctuary City movement as early as Jan 21, 2017.

There was additional cognitive load from the fact that, for most of my adult life, the Bay Area has been the destination if you work in tech. We are the Northern partner to the Hollywood Dream; where multiple generations have traveled to Southern California to make it big on the Silver Screen, we are now at the end of the second or third generation of immigrants to Northern California who want to make it big in Silicon Valley. Being from here, but working in tech, gives me something of an out-of-body experience: I can see what it was like here before the Silicon Valley explosion, and I can see what’s changed as a result. Some people I’ve met seem to think that the Bay Area and Silicon Valley are synonymous. To me, Silicon Valley is just another part of the Bay Area.

This is an interesting time to exist in the world, and a doubly interesting time to be from the Bay Area, and a triply interesting time to be from the Bay Area and working in tech. For decades, Silicon Valley has drawn some of the best and brightest here, having us work on tools for lightning fast global communication while requiring us to all work out of the same office in San Francisco. Video conferencing has existed for at least a decade, realtime chat has existed for at least two, and email is around its 40th birthday. Even with all that, before March of this year, fully distributed Silicon Valley companies were seen as quirky rebels. The very idea of hiring a remote employee at many startups required VP-level approval. The world changed quickly: thanks to COVID-19, all Silicon Valley companies are now fully distributed.

We can’t know until it happens what effect a forcibly distributed Silicon Valley will have on the Bay Area. We’re starting to see early results, but we won’t really know what’s changing until it’s changed. The rest of this article is my attempt to make a prediction on what I think is going to happen, based on the trends I’m seeing.

So let’s talk about those trends. I find it hard to believe that any Silicon Valley company will be returning full-force to an office in 2020, and some may never do so. Even if we put aside the mega-corps that have publicly stated they won’t require employees back at the office ever, I’ve heard rumors of a half-dozen companies who are planning to not renew their SF office lease in late 2020 or early 2021. Those companies aren’t claiming to be fully distributed forever, they’re instead claiming to be remote “for the foreseeable future”. In internet years, that’s forever, and so I expect to see more and more Silicon Valley companies with an SF P.O. Box, and little else.

At the same time, tech workers are waking up to the fact that they don’t have to be in the Bay Area and work in tech. Beyond the articles breathlessly suggesting mass migrations, I can think of 5 engineers without checking Twitter who are moving out of the Bay Area but keeping their current job. It’s a sign of where we are that that idea seems normalized to me, only three months in. For most of these folks, had they asked to relocate in February their boss would’ve interpreted that as a resignation. This flexibility is having an impact on hiring for Silicon Valley as well. I’m a hiring manager, and I talk to a lot of other hiring managers. I can say for myself I have no real intention to hire from the Bay Area candidate pool in 2020, mostly because of cost.

The Bay Area is, currently, an expensive place to live, with a shitty regional public transit system, and a dominant economic industry that couples with disjointed civic governments to promote rapid gentrification and displacement. I don’t blame a single soul for taking advantage of the small silver lining COVID-19 is providing, and taking their tech salary to somewhere they like better.

As a result of all of this, I’m already seeing some immediate changes that give us a glimpse of where the Bay Area is headed. Home price is a rough indicator of the desirability of a region, and since so many people have, over the past two decades, flocked to the Bay Area, home prices have steadily gone up. Even in 2008, home prices in the circle of Bay Cities didn’t fall so much as flatline. Right now, I’m seeing the beginnings of a fall. Homes that I’ve been watching in Alameda, where I’m from, have lost as much as $100k in predicted resale value in the past 30 days, and for the first time I can remember I’m seeing sellers lower listing prices to attract more bids. Two years ago, that was unthinkable.

So let’s play these trends out, and show you where I think the Bay Area is going. Where will we be, 5 years from now, in 2025? These are a series of bets, and if anyone wants to stake some monopoly money (read: cryptocurrency) or beers on me being wrong, reach out.

In 2025, we still have Silicon Valley in the Bay Area, but it looks pretty different. Most tech companies still have a Bay Area address, because there’s still cultural cachet attached, but for many companies that amounts to little more than a P.O. Box. Most executives of those tech companies still have pied-a-terre’s in the area around San Francisco, as do a large number of sales folks, because the types of work that benefitted from in-person meetings in early 2020 still benefit from in-person meetings. Tech Venture Capital is still largely clustered in the Bay Area, but the companies they’re funding have the most distributed workforce they’ve ever seen, and VCs are full-throatedly encouraging distributed hiring for the cost savings.

As for the tech workers themselves, the majority of those left in the Bay Area are the ones who still want to be here, because being in the Bay Area is no longer be any sort of requirement for working in tech and making a good salary. Speaking of, those salary prices have dropped dramatically in the Bay Area, but have risen in the rest of the country. Tech VC money is spread through more of the country than ever, because tech workers can now live wherever they want.

In 2025, the 2010s career path of “move to SF” → “get an internship / take a bootcamp” → “land a job in SF” has been replaced with “take an online bootcamp” → “get a remote job in tech” → “live where you want”. This started in the middle half of 2020, and has been slowly gaining ground ever since.

2025 is also a special year for the Bay Area: it’s starting to feel like a thriving region again after 4 years of COVID-and-recession driven pain. The slow exodus of tech workers spreading money around, combined with COVID closures, meant that the vibrant restaurant scene that was building in the Bay Area ground to a halt in 2020, and neighborhoods that were up-and-coming became food deserts almost overnight. House prices dropped significantly as a result, and the value of commercial real estate plummeted. Especially in 2021 and 2022, the Bay cities were unable to handle the rapid drop in the tax base, and struggled to restructure their budgets to match the region’s new needs. There were a few attempts in late 2021 and early 2022 to privatize ever more services, and Uber briefly experimented with getting into the paramedic business, before realizing that there wasn’t enough left of the ridiculously affluent population to support their business needs, and that most of their drivers had also moved to where it was cheaper.

The inner cities of Oakland and San Francisco started feeling emptier than they had in the past decade, but this was no real surprise to Bay Area lifers — it felt like 1995 all over again. BART, no longer crushed under capacity, was able to use its bond funds to make major upgrades throughout the region.

By late 2022, cities were starting to figure out how to handle this new world, and they found great and greater partnership in their citizenry. The people who were in the Bay Area were now there because they wanted to be, and those communities built on the activism of the late 2010s to rally around a greater sense of regional community.

Where we are now, in 2025, we’re seeing a reverse migration of some of the folks who were driven out by the booming 2010s, and are now drawn back by the cheaper housing. The commercial real estate quickly abandoned in 2020 and 2021 by tech companies is being leased for a song to new restaurant concepts, new community spaces, new art galleries, and new, smaller, tech companies with a local focus. While the Bay Area, and most of the world, struggled with how to adapt to a recession and a pandemic, tech entrepreneurship continued on its late 2010s path of profitable, small, boutique companies that were focused not on total market capture but on building the perfect product for their community. In 2025, many of these companies are flourishing in the Bay Area, but just as many are springing up across the country, from Buffalo to Boise.

As we reach the midpoint of 2025, we by no means have anything close to a utopia, in the Bay Area or anywhere, but the de-concentration of this country’s best and brightest is already being felt. Communities in the center of the country that were close to death are being revitalized, and learning from the successful late 2010s campaigns of cities like Tulsa on how to attract remote tech workers.

The Bay Area is my home, so I’m still here, and I’m one of the people working with boutique software companies and partnering with civic governments. Our community feels tighter than ever, because we know the people around us will be here for years. Instead of being pushed out by gentrification or pressured to cash out of their startup and move somewhere cheaper, we’re putting down roots. The people who want to live in the Bay Area live in the Bay Area, and the people who wanted to go home went home, and the people who wanted a new home found one.

This was a doozy, and thanks for sticking with this post. I’m sure I’m wrong about some of this, and I’d love to hear where you disagree with me. Ping me on twitter @phildini or email [email protected].

bay area tech future hiring

Page 1 of 6 Older →