Down the rabbit hole we go: Do professionals trust AI?

The reality is that developers are already using AI. 44% of them use AI tools in their development process now, and 26% plan to soon. Even more of those learning to code (55%) are currently using AI tools, and these early adopters will become established AI users once they settle into a professional developer position. 

While a growing number of developers are using AI tools, they aren’t using a wide variety. We asked developers which tools they were using, and out of the 21 options we listed this year, a vast majority are just using the two popular products: ChatGPT (83%) and GitHub Copilot (56%). Professional developers may need time to adjust existing workflows, and will most likely be motivated by their junior colleagues who are using AI tools while learning to code.

It’s early days in the hype cycle for these newer AI technologies. We expect that a little more time may need to pass before we see developers using more AI tools more broadly. Another factor that may be contributing to the slow adoption of AI tools into the development process is that professional developers don’t highly trust the accuracy of AI tools. Of those using or planning to use AI tools, only 3% highly trust the accuracy they provide, and twice as many (6%) highly distrust them. While the majority (39%) are somewhat trusting of AI tools, this response tells us that AI tools still need to prove their usefulness. 

When it comes to important matters like school or work, these new tools are not an easy button, and the axiom “trust but verify” is most likely the best approach to integrating new tools in the development process. On the AI Stack Exchange site, one of the most-visited questions so far this year is “Why is ChatGPT bad at math?” The question deals with the inability of AI tools, specifically those trained on large language models, to understand complex questions. But people get these sorts of problems wrong regularly, as well. 

Where you are currently working also has an impact on whether or not you are using AI tools. Professional developers from India, Brazil, and Poland are most likely to use or plan on using AI tools as part of their development workflow—all responding at 70% or higher. UK, French, US, and German developers are more likely to say they don’t plan on using AI tools (responding at 36% or higher). A lot of new tech talent is being sourced from India and Brazil, a trend inferred from the large number of young developers in high salary-growth positions as reported in past developer surveys, so more prevalent use of AI tools in these regions makes sense. 

Why would the EU countries and the US be more likely to report resistance to AI tools? Recent news has highlighted the possibility of imminent EU regulations for AI via pre-existing GDPR laws, and France has already begun investigating complaints against ChatGPT. In the US, no regulations currently exist, but companies are beginning to publicize their policies for banning ChatGPT at work, while Stack Overflow and Reddit have also publicly stated that they will start charging for data access to protect the attribution and contributions of their communities. Accuracy concerns and rules, whether legal or in the workplace, are keeping developers from rapidly adopting some AI tools.

Curiouser and curiouser: Why use AI tools at all?

Regardless of current use and intentions to use later, developers are positive about the possibilities.  Most feel favorable (77%) towards using AI tools as part of their development workflow—but we do see some differences between groups. Blockchain developers, data scientists, and front-end or fullstack developers are more likely to be using AI tools now. Embedded application developers, desktop/enterprise developers, and hardware engineers are less likely to be using these tools and less inclined to use them in the future.

AI has a complexity cliff. Much like Helmsman’s complexity cliff, a concept found in project management, after a certain point, the ability for AI to handle all the nuances and interdependencies of a solution drops off. That’s when humans, their adaptability to apply judgment and have original thought, saves the day.   

Out of all the developer roles who have a favorable view of AI tools, senior executives top the list (63%) alongside data scientists and blockchain developers.  Developers working in data science and blockchain using AI makes sense: they are well-versed in this type of technology. It makes even more sense when we see the response to our question about what use cases developers have for AI tools they’re using today. The overwhelming majority use AI tools to write code (83%) or debug code (49%). We’d expect hardware engineers not to use AI for now, given their low confidence in these tools’ accuracy and the hands-on aspects of their job.  When we asked developers about the benefits of AI tools, 33% believe increasing productivity is the most important benefit.  Senior executives that do not have direct experience with these use cases may be in positions to see their colleagues’ explorations with productivity around writing and/or debugging code and want to try it out within their team or share amongst other people managers.  

We asked developers how they thought AI tools would evolve their workflow over the next year, and 77% feel that the next year will look somewhat or very different for writing code with AI, and 75% feel the same about debugging code with AI. Updating the tools and evolving the learning curve will likely benefit those currently using AI tools or planning to soon. 

The age of AI-integrated development is still in its infancy, but developers around the world have let us know that they are actively engaged, curious, and not looking away. Make sure to check out all the wonderful details at Stack Overflow Labs!

The post Hype or not? AI’s benefits for developers explored in the 2023 Developer Survey appeared first on Stack Overflow Blog.

We also took a look at the real sentiments behind this year’s surge in AI popularity. Is it making a real impact in the way developers work or is it all hype? Head over to the blog post for a recap of the top insights, but check out Stack Overflow Labs for the deep dive details.

The times, they are a-changin’ 

For those that are very early or very late in their coding careers, or just prefer other methods of online learning besides Stack Overflow, it’s safe to assume we may be under-indexing those folks.  Our top five key territories are the same as previous years with a slight shift in which countries are offering the top number of responses: the U.S. is consistently the number one country for responses, but this year we saw India drop from the second highest response rate to third place behind Germany.  Over two years, we have seen the responses from India drop 50% while responses from Germany have grown by 30%.  Indian developers taking the survey this year are skewing a little younger (89% under 34) compared to the age distribution of all respondents (62% under 34) so we may be missing responses from a segment of more experienced developers in India.  Overall, the age distribution has moved slightly older this year with 37% of respondents aged 35 or older compared to 31% last year.  

Among the top ten programming languages this year, we see three popular languages increase their standing: Python, Bash/Shell (all shells), and C.  

We know from tracking tags on Stack Overflow that Python has been growing in popularity for the last three years, and this year it’s moved up one spot to take number three. Along with the rise in Python’s popularity, we consistently see Python libraries rank highly in our popular “Other technologies and frameworks” section—NumPy and Pandas have been in the top ten for the last three years.  

C is an interesting language to see rise in popularity: it has not been in the top ten for programming languages in the Developer Survey before despite being around since the 1970’s.  The TIOBE index has C as the second most popular programming language as of June 2023 and has engagement numbers close to their number one language, Python.  Given that C is one of the most popular low-level languages, it’s the language of choice in embedded programming within IoT devices,  and it “powers almost every technological experience we have” according to Codecademy, it makes sense that developers are using C more than ever this year. 

The more popular a programming language is, the less experience on average we see reported from developers this year (around ten years for our top three languages, and closer to 14 years for those languages reporting less than 1% usage).  Less experienced developers reporting higher usage of some programming languages possibly strengthens the argument that a lower barrier of entry into the job force increases their popularity among coders.

Overall, the baseline for salary growth in 2023 was 10% over last year.  When controlling for comparable years of experience, we see a wide range of growth and decline amongst the programming languages (as much as 42% growth and 69% contraction year-over-year).  If we take a language’s popularity as a proxy for prevalence of developers in the labor market, then we could assume that salaries for popular languages will fall and less popular ones will rise. Our top three most popular languages—JavaScript, HTML/CSS and Python—all saw negative growth in median salary (ranging from 5-10% contraction) while the three least popular languages we can match to comparable experience—APL, Crystal and SAS—we see even less salary growth (ranging from 8 to 80% contraction).  But if we look at the same languages but adjust for less than average experience (for example, 5 years less experience than average per programming language), we see marginal growth for our top three and double-digit growth for APL and Crystal (SAS was filtered from the results for not having enough responses).  What we surmise is that developers’ negotiating power has been affected this year and those with a few less years of experience and working in more niche languages have more salary upside this year than developers with more experience in more popular programming languages. 

Get inspired by technologies admired and desired 

This year, we added a new section to the survey results for technology trends for those who have used or want to use programming languages, tools, environments, libraries, etc., that we have dubbed “Admired and Desired”.  Developers discover new technologies via Stack Overflow or their network, blogs, online forums, etc., and based on preliminary research, would like to try them out.  Once a developer has had a chance to try said technology out, how many of them would still want to use it?  

To better gauge hype versus reality, we created a visualization that shows the distance between the proportion of respondents who want to use a technology (“desired”) and the proportion of users that have used the same technology in the past year and want to continue using it (“admired”).  For those with wide distances between them (and the admired metric remaining on the larger end of the scale), we can infer that momentum generated by the hype grows with hands-on use. With shorter distances, we can infer that the hype is doing much of the heavy lifting as far as general popularity is concerned.  For example, we can see JavaScript, ranked as most popular programming language since 2011, has a relatively short distance between admired and desired (<10 percentage points), while Rust, a top choice for developers who want to use a new technology for the past eight years, shows a wide distance (>60 percentage points); Rust is a language that generates more desire to use it once you get to know it than JavaScript.  Seeing this growth in admiration for certain technologies gives us insight into what has staying power and what needs help in order to generate coveted evangelists to convert new users into ones that will stick around. 

Do admired technologies equate to financial security, as well?  The top admired programming languages (>70% who have used them in the past year want to continue using them)—Rust, Elixir, and Zig—all have 20% or more higher salaries than the overall median developers in 2023 (roughly $75K annually). There is certainly not a direct correlation between salaries and the admiration factor since we see Python, TypeScript, and HTML/CSS all have admiration scores above 60% but only make marginally better annual salaries than the overall median.  Perhaps part of the devotion to some of these popular and admired programming languages is the confidence developers attain by having that lower barrier to entry mentioned earlier; programmers can become sufficient and hired with less experience and get the benefit of so many online resources and Stack Overflow answers to help continue their learning process.

We have a video where we take you through our favorite highlights from the 2023 Stack Overflow Developer Survey. We started out trying to make this a 10-minute video, but we had so much to share…check it out here.

The post 2023 Developer Survey results are in: the latest trends in technology and work from the Stack Overflow community appeared first on Stack Overflow Blog.

But for any piece of tech to work as intended, you have to make sure everyone on the team agrees on using this technology to solve that problem.

If you’re thinking about rolling out a new tool to your team, you should also be thinking about how to get colleagues and management on board, how to embed that tool in your everyday workflows, and how to assess whether it’s working as it should. Tech that solves human problems needs humans to participate in those solutions.

However passionate you might be, declaring that Tool X will improve every aspect of every business process is unrealistic, and unlikely to earn you the support of your colleagues and managers—especially if you’re asking them to learn something new or tweak their workflows to accommodate a tool whose value they’re not convinced of. 

Particularly when budgets are tight, many companies scrutinize tool usage to make sure they’re spending money wisely, not wasting it on shelfware—software you buy but never use. If your team isn’t aligned around the need for a new tool, adoption will probably be underwhelming, and your solution will end up gathering metaphorical dust.

In this article, we’ll lay out some strategies for keeping your new tool from becoming shelfware.

Focus on use cases

You can make a more compelling case for adoption by identifying the most impactful use case for your solution and leading with that. Aligning yourself from the beginning with a specific process like cloud migration or product releases can also help you articulate and demonstrate the value of a new tool. 

In a recent post on our blog, Chelsea Troy, a staff data engineer at Mozilla, suggests starting by talking about the problem you’re trying to solve (“We get so many notifications that we miss important alerts and aren’t able to focus on our work”) rather than centering your proposed solution (“Look at this neat tool I found for managing alerts”).

Another tried-and-true method of showcasing a tool’s value and sparking interest among the wider org is to roll it out first to one or two teams in a pilot, like Dropbox did with Stack Overflow for Teams.

Identify your advocates

A successful tool adoption starts with the early adopters, who play a critical role in demonstrating a tool’s value and communicating that value throughout the org. These early adopters are your true leaders, says Stack Overflow engineering manager Peter O’Connor. Identify champions at the individual-contributor level: tech leads or stack engineers who have the knowledge and credibility to drive adoption by demonstrating and socializing the tool’s value.

Once your early adopters are in place, using your tool to solve their problems and telling all their friends about it, it’s time to think about identifying someone in leadership who can serve as an executive sponsor and high-level advocate for your adoption. This person can help ensure that your proposed implementation aligns with your company’s strategic goals, assist in building support and overcoming resistance from other executives, and otherwise provide support and direction to drive adoption. An executive sponsor should understand:

• What problems the new tool can solve, and how that positive change will impact the organization
• Who’s involved in introducing the new tool and how those teams/individuals can support adoption
• How implementation will affect employees, consultants, and customers

To encourage ongoing usage after your initial rollout, it can be helpful to identify power users or advocates who can spread the news about the tool’s value, help others get up and running, and incorporate the tool into your existing workflows.

Build consensus over time

You want people to want to use your tool. For that to happen, you must understand the context of the technical system you’re operating in. “Context is king” when it comes to getting technical teams to make big changes, says Troy: “Who has context on the system is who has power on the team. And that drives more technical decisions than we’d like to admit.”

Understanding that context requires talking to your teammates about what they need—and, more importantly, listening to them. Commit to understanding what your colleagues need and show them that their input matters to you.

Your goal, in Troy’s view, is to create “an environment in which people are able to ask questions and express concerns.” Make your coworkers feel heard, and they’ll be more likely to listen when it’s your turn to speak. This environment, Troy explains, “makes the difference between the big changes that sail through with support and the ones that get stuck in the mud because the team dragged their feet or outright resisted them.”

This view is echoed by O’Connor, the engineering manager at Stack Overflow. Rather than issuing commands and rushing the adoption timeline, he says, teams that roll out new tools successfully ask for input, take time to build consensus, and foster a sense of psychological safety in their teams.

Make adoption frictionless

You can minimize friction around adoption if you invest the time and energy up front to talk with and listen to your teammates, understanding the nature of their pain points and laying out how your solution will improve things. If you come charging in with a fully formulated plan and no interest in a collaborative discussion, teams will be slow to adopt a new tool or simply reject it.

Keep in mind that a tool that fits seamlessly into the ecosystem of tools and processes your team is already using has a much better chance of being adopted enthusiastically than a tool that interrupts an established workflow. That’s one reason why developers tend to adopt our paid platform quickly and easily when our customers introduce it: they’re already used to turning to Stack Overflow’s public platform when they get stuck.

In a previous role, my colleague Ryan Donovan saw firsthand how reluctant people are to use a new tool if it introduces clunkiness to their workflow. He was tasked with helping the engineering org adopt a Q&A product, but people primarily went to Slack for questions. 

“Nobody was using the tool because it had a lot of proscriptive processes around it,” said Donovan. “First, all the questions were routed through pre-selected categories that we selected instead of letting the people with the questions select tags. Second, while the new tool had a Slack integration, leadership wanted to route questions to new channels specifically created to host questions. Both of these demanded workflow changes. No surprise that everyone still asked their questions in the team channels.” 

Say no to shelfware

You may have noticed that many of the strategies we recommend here come into play before the tool is actually rolled out. That’s because the best way to avoid shelfware is to do your homework ahead of time: 

• Learn where your tool can have the biggest impact
• Enlist advocates to demonstrate and socialize your tool
• Build consensus over time
• Minimize friction around adoption

To learn more about making tool adoption successful, register for our upcoming webinar: Five ways to drive tool adoption and avoid the dreaded “shelfware.”

The post How to keep your new tool from gathering dust appeared first on Stack Overflow Blog.

While the field is still developing fast, and factual errors, known as hallucinations, remain a problem for many LLM powered chatbots, a growing body of research indicates that a more guided, auto-regressive approach can lead to better outcomes.

This gets really interesting when applied to the world of software development and CI/CD. Most developers are already familiar with processes that help automate the creation of code, detection of bugs, testing of solutions, and documentation of ideas. Several have written in the past on the idea of self-healing code. Head over to Stack Overflow’s CI/CD Collective and you’ll find numerous examples of technologists putting this ideas into practice.

When code fails, it often gives an error message. If your software is any good, that error message will say exactly what was wrong and point you in the direction of a fix. Previous self-healing code programs are clever automations that reduce errors, allow for graceful fallbacks, and manage alerts. Maybe you want to add a little disk space or delete some files when you get a warning that utilization is at 90% percent. Or hey, have you tried turning it off and then back on again?

Developers love automating solutions to their problems, and with the rise of generative AI, this concept is likely to be applied to both the creation, maintenance, and the improvement of code at an entirely new level.

More code requires more quality control

The ability of LLMs to quickly produce large chunks of code may mean that developers—and even non-developers—will be adding more to the company codebase than in the past. This poses its own set of challenges. 

“One of the things that I’m hearing a lot from software engineers is they’re saying, ‘Well, I mean, anybody can generate some code now with some of these tools, but we’re concerned about maybe the quality of what’s being generated,’” says Forrest Brazeal, head of developer media at Google Cloud. The pace and volume at which these systems can output code can feel overwhelming. “I mean, think about reviewing a 7,000 line pull request that somebody on your team wrote. It’s very, very difficult to do that and have meaningful feedback. It’s not getting any easier when AI generates this huge amount of code. So we’re rapidly entering a world where we’re going to have to come up with software engineering best practices to make sure that we’re using GenAI effectively.”

“People have talked about technical debt for a long time, and now we have a brand new credit card here that is going to allow us to accumulate technical debt in ways we were never able to do before,” said Armando Solar-Lezama, a professor at the Massachusetts Institute of Technology’s Computer Science & Artificial Intelligence Laboratory, in an interview with the Wall Street Journal. “I think there is a risk of accumulating lots of very shoddy code written by a machine,” he said, adding that companies will have to rethink methodologies around how they can work in tandem with the new tools’ capabilities to avoid that.

We recently had a conversation with some folks from Google who helped to build and test the new AI models powering code suggestions in tools like Bard. Paige Bailey is the PM in charge of generative models at Google, working across the newly combined unit that brought together DeepMind and Google Brain. “Think of code produced by an AI as something made by an “L3 SWE helper that’s at your bidding,” says Bailey, “and that you should really rigorously look over.” 

Still, Bailey believes that some of the work of checking the code over for accuracy, security, and speed will eventually fall to AI as well. “Over time, I do have the expectation that large language models will start kind of recursively applying themselves to the code outputs. So there’s already been research done from Google Brain showing that you can kind of recursively apply LLMs such that if there’s generated code, you say, “Hey, make sure that there aren’t any bugs. Make sure that it’s performant, make sure that it’s fast, and then give me that code,” and then that’s what’s finally displayed to the user. So hopefully this will improve over time.”

What are people building and experimenting with today?

Google is already using this technology to help speed up the process of resolving code review comments. The authors of a recent paper on this approach write that, “As of today, code-change authors at Google address a substantial amount of reviewer comments by applying an ML-suggested edit. We expect that to reduce time spent on code reviews by hundreds of thousands of hours annually at Google scale. Unsolicited, very positive feedback highlights that the impact of ML-suggested code edits increases Googlers’ productivity and allows them to focus on more creative and complex tasks.”

“In many cases when you go through a code review process, your reviewer may say, please fix this, or please refactor this for readability,” says Marcos Grappeggia, the PM on Google’s Duet coding assistant. He thinks of an AI agent that can respond to this as a sort of advanced linter for vetting comments. “That’s something we saw as being promising in terms of reducing the time for this fix getting done.” The suggested fix doesn’t replace a person, “but it helps, it gives kind of say a starting point for you to think from.”

Recently, we’ve seen some intriguing experiments that apply this review capability to code you’re trying to deploy. Say a code push triggers an alert on a build failure in your CI pipeline. A plugin triggers a GitHub action that automatically send the code to a sandbox where an AI can review the code and the error, then commit a fix. That new code is run through the pipeline again, and if it passes the test, is moved to deploy. 

“We made several improvements in the mechanism for the retry loop so you don’t end up in a weird scenario, but that’s the essential mechanics of it,” explains Calvin Hoenes, who created the plugin. To make the agent more accurate, he added documentation about his code into a vector database he spun up with Pinecone. This allows it to learn things the base model might not have access to and to be regularly updated as needed. 

Right now his work happens in the CI/CD pipeline, but he dreams of a world where these kind of agents can help fix errors that arise from code that’s already live in the world. “What’s very fascinating is when you actually have in production code running and producing an error, could it heal itself on the fly?” asks Hoenes. “So you have your Kubernetes cluster. If one part detects a failure, it runs into a healing motion.” 

One pod is removed for repairs, another takes its place, and when the original pod is ready, it’s put back into action. For now, says Hoenes, we need humans in the loop. Will there come a time when computer programs are expected to autonomously heal themselves as they are crafted and grown? “I mean, if you have  great test coverage, right, if you have a hundred percent test coverage, you have a very clean, clean codebase, I can see that happening. For the medium, foreseeable future, we probably better off with the humans in the loop.”

Pay it forward: linters, maintainers, and the never ending battle with technical debt

Finding things during CI/CD or addressing bugs as they arise is great, but let’s take things a step further. You work at a company with a large, ever-growing code base. It’s fair to assume you’ve got some level of technical debt. What if you had an AI agent that reviewed old code and suggested changes it thinks will make your code run more efficiently. It might alert you to fresh updates in a library that will benefit your architecture. Or it might have read about some new tricks for improving certain functions in a recent blog or documentation release. The AI’s advice arrives each morning as pull requests for a human to review. 

Itamar Friedman, CEO of CodiumAI, currently approaches the problem while code is being written. His company has an AI bot that works as a pair programmer alongside developers, prompting them with tests that fail, pointing out edge cases, and generally poking holes in their code as they write, aiming to ensure that the finished product is as bug free as possible. He says a lot of tools for measuring code quality focus on aspects like performance, readability, and avoiding repetition. 

Codium works on tools that allow for testing of the underlying logic, what Friedman sees as a narrower definition of functional code quality. With that approach, he believes automated improvement of code is now possible, and will soon be fairly ubiquitous. “If you’re able to verify code logic, then probably you can also help, for example, with automation of pull requests and verifying that these are done according to best practices.”

Itamar, who has contributed to AutoGPT and has given talks with its creator, sees a future in which humans guide AI, and vice versa.  “A machine would go over your  entire repository and tell you all of the best (and so-so) practices that it identified. Then a few tech leads can go over this and say, oh my gosh, this is how we wanted to do it, or didn’t want to do it. This is our best practice for testing, this is our best practice for calling APIs, this is how we like to do the queuing, this is how we like to do caching, etc. It’ll be configurable. Like the rules will actually be a mix of AI suggestion and human definition that will then be used by an AI bot to assist developers. That’s the amazing thing.”

How is Stack Overflow experimenting with GenAI? 

As our CEO recently announced, Stack Overflow now has an internal team dedicated to exploring how AI, both the latest wave of generative AI and the field more broadly,  can improve our platforms and products. We’re aiming to build in public so we can bring feedback into our process. In the spirit, we shared an experiment that helped users to craft a good title for their question. The goal here is to make life easier for both the question asker and the reviewers, encouraging everyone to participate in the exchange of knowledge that happens on our public site.

It’s easy to imagine a more iterative process that would tap in the power of multi-step prompting and chain of thought reasoning, techniques that research has shown can vastly improve the quality and accuracy of an LLM’s output.

An AI system might review a question, suggest tweaks to the title for legibility, and offer ideas for how to better format code in the body of the question, plus a few extra tags at the end to improve categorization. Another system, the reviewer, would take a look at the updated question and assign it a score. If it passes a certain threshold, it can be returned to the user for review. If it doesn’t, the system takes another pass, improving on its earlier suggestions and then resubmitting its output for approval.

We are lucky to be able to work with colleagues at Prosus, many of whom have decades of experience in the field of machine learning. I chatted recently with Zulkuf Genc, Head of Data Science at Prosus AI. He has focused on Natural Language Processing (NLP) in the past, co-developing an LLM-based model to analyze financial sentiment, FinBert, that remains one of the most popular models at HuggingFace in its category. 

“I had tried using autonomous agents in the past for my academic research, but they never worked very well, and had to be guided by more rules based heuristics, so not truly autonomous,” he told me in an interview this month. The latest LLMs have changed all that. We are at the point now, he explained, where you can ask agents to perform autonomously and get good results, especially if the task is specified well. “In the case of Stack Overflow, there is an excellent guide to what quality output should look like, because there are clear definitions of what makes a good question or answer.”

What about you? 

Developers are right to wonder, and worry, about the impact this kind of automation will have on the industry. For now, however, these tools augment and enhance existing skills, but fall far short of replacing actual humans. It appears some of bots have already learned to automate themselves into a loop and out of a job. Tireless agents that are always working to keep your code clean. I guess we’re lucky that so far they seem to be as easily distracted by time consuming detours as the average human developer? 

Technology marches on, but procrastination remains unbeaten. 

We are compiling the results from our Developer Survey and have tons of fascinating data to share on how developers view these tools and the degree to which they are already adopting them into their workflows.

If you’ve been playing around with ideas like this, from self-healing code to Roboblogs, leave us a comment and we’ll try and work your experience into our next post. And if you want to learn more about what Stack Overflow is doing with AI, check out some of the experiments we’ve shared on Meta.

The post Self-healing code is the future of software development appeared first on Stack Overflow Blog.

A few months ago, we wrote about the overlap between people with ADHD (attention-deficit/hyperactivity disorder) and people who code for a living. We noted the plethora of online advice by and for programmers with ADHD and the rise in ADHD diagnoses for both kids and adults. And we wondered whether there’s anything to the fairly widespread idea that coding is a particularly good career fit for a person with ADHD. (For our purposes, we’ll use “developer” and “programmer” more or less interchangeably to refer to people whose jobs involve a lot of coding.) 

“Coding can give ADHD brains exactly the kind of stimulation they crave,” writes one full-stack developer. “Not only is coding a creative endeavor that involves constantly learning new things, but also once one problem is solved, there’s always a brand new one to try.”

Of course, when you’re talking about two things as complex as 1) the human brain and 2) computer programming, generalizations like “people with ADHD make great programmers” can only take you so far. Takes like that risk collapsing the experiences of people with ADHD, skimming over individual variations and nuances in favor of an appealing soundbite. 

For this follow up post, we spoke with two Stack Overflow software engineers with ADHD about their experiences being diagnosed as adults, taking medication, and communicating about their ADHD at work. Here’s what developers with ADHD want you to understand.

It’s less superpower, more invisible disability

It can be frustrating for people with ADHD to hear a symptom like hyperfocus referred to as a “superpower,” when in reality hyperfocus has downsides—plus it exists in conjunction with other symptoms that can be much less empowering, like executive dysfunction. 

As someone with an OCD (obsessive-compulsive disorder) diagnosis, I’ve heard similarly frustrating takes: I wish I had OCD so my house could be as clean as yours! (You do not wish this.) Oh, I’m super-OCD too; I color-code my bookshelves. (Not the same!)

In fact, ADHD is an invisible disability, which is exactly what it sounds like: “a physical, mental or neurological condition that is not visible from the outside, yet can limit or challenge a person’s movements, senses, or activities” (Invisible Disabilities Association). Plenty of forms of neurodivergence, including autism spectrum disorder, depression, anxiety, OCD, PTSD (post-traumatic stress disorder), and learning differences can be considered invisible disabilities. 

Sure, developers with conditions like ADHD might occasionally find that an output of their condition gives them an edge. In part one of this series, we recognized that the hyperfocus associated with ADHD—“a state of laser-like concentration in which distractions and even a sense of passing time seem to fade away,” as one developer put it—can help programmers access the sought-after flow state. 

Other developers with ADHD say their thinking style lends itself to creative problem-solving. “One positive aspect of being a dev with ADHD,” explained one engineer we talked to, “is that my brain zooming around different ideas can help with inventiveness and creativity, and seeing things in a different light can really help with solving more difficult problems.” Intersectional thinking FTW.

Still, many developers want to be clear that it’s not all upside. “You get into flow, and you’re being really, really productive,” one software engineer explained, “but on the opposite end of that, time goes by really quickly, and you realize, ‘Oh, crap, I had three other things I promised somebody today, but I just lost a few hours.’” The engineer also pointed out that developers who progress past the senior level or switch tracks into management are no longer responsible for their individual productivity alone; their role is to multiply the productivity of their team. “That’s where executive dysfunction holds me back a lot,” they said.

It’s not the coding; it’s the accommodations

What if it’s not that people with ADHD make good developers; it’s that developers are more likely to have access to the accommodations that make ADHD manageable?

Companies that employ developers, particularly tech companies with flexible and hybrid schedules and robust healthcare coverage, are in a better position to accommodate people with ADHD (and other invisible disabilities) than employers in other industries. 

For instance, when it comes to managing their ADHD at work, one of our engineers stressed the freedom of a job that can be largely asynchronous and remote: “It helps a lot that I have a job that supports flexible hours and isn’t babysitting me all day.” In tech and developer circles, the unfair stigma associated with ADHD and other forms of neurodivergence is beginning to dissipate, as we discussed on the Stack Overflow podcast last year. “I’ve been open about my diagnosis mostly,” said one interviewee. “I might not always refer to it by name, but I definitely bring up things that are relevant when I need to,” such as a need to establish a firm deadline to stay focused.

So a better way of putting the relationship between coding and ADHD might be that (some) coding jobs are likely to give (some) people with ADHD what they need to thrive professionally.

It’s also reasonable to assume that people with reliable, affordable healthcare are more likely to seek out and receive an ADHD diagnosis and the accommodations, including access to medication and therapy, that come with that diagnosis. That’s another reason why it might seem like there’s an overlap between people who code and people with ADHD—US-based developers tend to have good healthcare coverage through their employers.

A diagnosis might be life changing

Both software engineers we interviewed were diagnosed as adults, in both cases in their late 20s after they’d already embarked on their careers. “I was talking to a friend who mentioned she went to a doctor and was surprised to find that she had ADHD,” one of the engineers said. “She talked about the symptoms and I thought, ‘Hmm, that sounds familiar.’”

For both engineers, being diagnosed as adults cast their lifelong experiences in a new light. “I always just thought that I was lazy and had a tendency towards procrastination,” said one engineer. “But once I embraced [my diagnosis] and realized that a lot of stuff I thought was an ‘everyone problem’ was not actually a problem for neurotypical folks, I felt a lot better about myself and about the strategies I needed to cope.” 

Being diagnosed as an adult, said the other interviewee, “is really interesting, because you all of a sudden understand where a lot of your weird traits come from. You realize, ‘OK, that’s why this is hard for me; that’s why I struggle with this or that.”

One interviewee called their ADHD medication “a life-changer.” The other called it “a complete game-changer for me in terms of focus and ability to get things done.” An official diagnosis is generally a necessary prerequisite for ADHD medication, so for many folks, getting a diagnosis is the first big step toward effectively managing their ADHD.

A diagnosis can also give people with ADHD the confidence to ask for accommodations at work or school—and even the awareness to know what kinds of accommodations are available and would benefit them. “A diagnosis definitely helped me at work,” said one engineer. “I haven’t ever asked for formal accommodations, but knowing more about how I personally work—for example, if I don’t have a deadline, it’s basically impossible for me to get it done—has helped me a lot in advocating for myself and my own working style.”

(Neuro)diversity is strength

As we said in part one, dispelling the stigma around neurodiversity requires an open dialogue about ADHD and other forms of neurodiversity or invisible disability. At Stack Overflow, we think everyone benefits when work and the hiring process are inclusive of neurodiverse people. An estimated 15-20% of the population is considered neurodiverse; that’s a lot of talent employers can miss out on if they’re not willing or able to offer certain accommodations. And you never know—the next person on your team to receive an ADHD diagnosis might be you.

The post What developers with ADHD want you to know appeared first on Stack Overflow Blog.

Every day Stack Overflow helps all types of developers learn new skills by answering their questions or helping them discover new solutions and approaches. We also know that learning is hard. Oftentimes, users don’t know what questions to ask – they don’t know what they don’t know. That’s why earlier this year, we were excited to announce more ways to learn and grow your skills with our online course recommendations. This new native advertising product recommended online courses that were selected based on the content a user viewed or is currently viewing and provided them with a structured learning path that was relevant to what they wanted to learn.

The following post dives deep into how we designed and built the recommendation system and how it has laid the foundation for other AI/ML initiatives at Stack Overflow. 

The idea

Recommending external content is nothing new to Stack Overflow. As part of our current Ads business, we offer a content/question-matching product called Direct-to-Developer. What was new about our online course recommendation system is that while developing it, we were simultaneously launching a new, modern data platform for Stack Overflow. This gave us fresh infrastructure, unrestricted tech stack options, and the opportunity to use state-of-the-art machine learning techniques.

Building upon a clean slate allowed us to hyper-focus on making the learning process easier and more efficient by bringing relevant and trusted courses to developers and technologists. We also wanted to run highly configurable experiments throughout the pilot—bonus points if we could reuse components for other initiatives. 

This lead us to build a cloud-first content-based recommendation system that recommends content that is similar to the posts you have or currently are interacting with. This system would need to account for three types of recommendations; post-to-content, user-to-content, and a fallback of the most popular. Each addressed a specific use case to ensure a relevant course could always be served while respecting user preferences. 

Our approach

Post-to-content 

At the core of a content-based recommendation system is how similarity is defined. At Stack Overflow, we have a lot of text and the catalog of courses to recommend also includes text. So we needed a way to measure the similarity between two pieces of text; a Stack Overflow post and an online course. If you are familiar with NLP (natural language processing), you know there are dozens of ways to approach this problem. Recently, there has been one approach that does extremely well: semantic similarity using embeddings from a pre-trained transformer model.

In our approach, we used a pre-trained BERT model from the `SentenceTransformers` library to generate an embedding for all 23 million Stack Overflow questions and a separate embedding for every course in the catalog. These embeddings allowed us to calculate cosine distance to determine how similar two pieces of text are. We only had a couple thousand courses, so we were able to load all the course embeddings into a nearest-neighbor model and perform a brute-force search to match courses to all questions. The output of this model is our post-to-content recommendation. 

In our prototyping phase, this approach performed extremely well! Not only was it able to find the most relevant courses, but by using the BERT model it already had foundational knowledge like “Node is related to JavaScript” and “Swift is related to Apple”. It allowed for offline evaluation and helped us identify which popular technologies on Stack Overflow were missing from the course catalog. 

User-to-content

Our post-to-content recommendations ensured that every Stack Overflow question had a list of relevant courses for an ad to be served. This provides an alternative just-in-time learning opportunity for all visitors. But for the users that opted-in to targeted ads, we wanted to provide a personalized recommendation that leveraged our first-party data.

To make a recommendation to a user we needed a way to represent the sum of the content that they have viewed. To do this we took all the posts they viewed within a certain lookback window and averaged them into one embedding. For example, if a user viewed ten posts, we took those ten post embeddings described above and averaged them element-wise into one embedding. This is commonly referred to as mean pooling. With this one embedding, we can use the same nearest-neighbor model and perform find the most similar course to what the user viewed to produce a user-to-content recommendation.

This approach also performed extremely well when we evaluated our own interactions with questions. Yes, Stack Overflow employees also use Stack Overflow! This personalized recommendation consistently outperformed post-to-content in clickthrough rate once we launched.

Top content

The last type of recommendation was to randomly select from a list of top courses, mostly JavaScript or Python, and was used purely as a fallback scenario. Either if there was no recommendation available or if something upstream failed. Regardless top was only served a fraction of the time as priority was set to serve user-to-content and then post-to-content.

How we built it 

With any project, especially a machine learning one, the hardest part is often moving it into production. Throw in simultaneously building a brand-new cloud data platform and a non-existent machine learning platform, we had our work cut out for us. Early on we decided not to rebuild the wheel and to empower “full-stack data science” work. This allowed our data scientist (myself) to have almost complete ownership of the entire project. This blessing and curse made it clear that leveraging one platform for as much as possible would be wise, and our platform of choice was Azure Databricks. This allowed us to keep all data processing, feature engineering, model versioning, serving, and orchestration all in one place.

Recommender systems are inherently complex because of all the separate components. So we designed our system to be as modular as possible. This allows us to control dependencies and experiment with different model parameters and techniques. The crux of our system is an offline-batch recommender system, meaning all recommendations would be pre-computed offline and stored somewhere ready to be served when needed. While there are many intricacies, think of our system as being broken down into three core components: generating embeddings, making recommendations, and serving recommendations.

Generating embeddings

As mentioned earlier, the core of a content recommendation system is how similarity is defined. For us, it was computing sentence embeddings and performing a nearest neighbor search. Doing this for 23 million posts and millions of users each run is no small task. Because Databricks is built on PySpark, naturally we leaned heavily into the PySpark ecosystem, which allowed us to rapidly prototype on “small” amounts of data and scale with no code changes. 

For the one-time job of generating post embeddings, we wrapped the BERT model in a PySpark Pandas UDF that ran on a GPU cluster. Then on a regular cadence, we computed new and edited question embeddings. Each time the post text, embedding, and other meta-data were written to a feature store. This was the same approach we took to generate the course embeddings. 

The user embeddings had to be refreshed each time we wanted to make new recommendations to account for the most recent user activity. We set a minimum number of questions a user had to view within the lookup window, if they met the threshold, then they were eligible for a recommendation to be made. For eligible users, we pulled the post embeddings from the post feature store table, pooled them, and wrote them to a separate user feature store table. For pooling, we also leveraged a Pandas UDF as there were other pooling methods we wanted to try, like linear or exponential weighting. 

Making recommendations

With only a couple thousand courses, we loaded all the embeddings into a modified nearest neighbor model, which allowed us to log it directly to the MLflow Model Registry and track lineage to the content feature store. We only had to retrain this model if we added or removed courses to the catalog. Logging the model to the registry also meant there was a clear path to going real-time if we so chose, as we could deploy the model as a serverless API.

Whenever making the recommendations, the logged model was then pulled from the registry and predictions were made directly on a Spark DataFrame. The data was then written to an intermediary Delta table. 

For those familiar with recommender systems, you may notice that this is a retrieval-only system, and you are correct. The initial release of the system did not include a ranker model as we needed to collect labels to train a model to re-rank results based on predicted clickthrough rate or other business-driven metrics, not just similarity. 

Serving recommendations

Once all recommendations were computed and a series of quality checks were passed, the final results were then written to an Azure SQL Server database. This database served as a low enough latency database for our internal ad-serving platform to select from when it needs to serve a course recommendation.

Once our internal ad server was told to serve an online course ad, it first looked to see if the user was opted-in to targeted ads. If so, it would then check to see if a user-to-content recommendation was available. If not, it would use post-to-content, and if there was a failure or the ad was being served on a non-question page, then a top content recommendation was served. This logic can be seen in the below flowchart.

Deployment and experiment 

Each of the above three core components had various parameters that could be changed and configured. Each potentially affecting the final recommendation and the execution time significantly. What embedding model, lookback window, or pooling method was used? What cluster configuration, execution schedule, and batch size were used? To address this, we used Databricks’ CI/CD tool dbx to parametrize the workflows and configurations, then had GitHub actions execute the builds. This allowed us to easily move from our development workspace to our production workspace and version changes, while always knowing how a recommendation was computed. 

Looking forward

We achieved a lot during the course (no pun intended) of building our online course recommendation system. We went from having no data platform to having a feature-reach machine learning platform that has been handling millions of recommendations every week. It was a learning experience for everyone involved but, most of all, an exciting milestone for Stack Overflow’s AI/ML initiatives. We have already re-purposed some of the work to improve our Related Questions section, which saw a 155% increase in the clickthrough rate (look for a future blog post on this). We have a lot of other ideas and experiments we’re working on for other applications of AI/ML to improve our platforms.

The post More on our AI future: building course recommendations and a new data platform appeared first on Stack Overflow Blog.

We recently hosted a webinar about searching for jobs with several experts from Indeed. A big part of this discussion (and of the questions asked afterwards) was about resumes. That’s no surprise—resumes are the first contact you have with any prospective employer. Just sending the same resume to every job opening may eventually land you a job, but it’s not going to be the fastest and most productive method. 

When a company is looking for someone to fill an open role, they want to shop around and find the best fit. It’s similar to a person looking to make a big purchase. They want to see a quick overview of the specs when evaluating the field, then hop on a sales call with the best options. This article will talk about how you can make that resume you apply with more likely to lead to a sales call—that is, an interview. 

Think like a marketer

I know plenty of you cringed when you saw the word “marketing.” A lot of engineers are allergic to marketing, especially when it comes to dev tools. But marketing is all about communicating an idea in a way that’s easy to understand and compelling to a person. It’s about putting yourself in another person’s shoes and talking about your products and services in a way that matches their interests and pain points. Obviously, the end result is sales, but hopefully you are selling something that the buyer actually needs and appreciates. 

When we’re planning a campaign to increase sales, we start with trying to understand who our potential buyers are. To this end, we sketch out general personas: what they like, what they do, what their pain points are. It doesn’t have to be completely detailed, but the exercise helps understand who we think will buy our products or services. 

From there, we can create a one-pager that covers the basic features and benefits that would appeal to a specific persona. Sometimes detailed feature specs can interest someone in a product or service. But you’re asking them to do a little work to connect the dots between what you’re offering and what they need. In marketing, we try to connect as many of those dots as possible in our one pager. The end goal is to get them interested so we can discuss their particular needs and explore how our solutions solve them. This process is known as the buyer’s journey. 

When applying for jobs, you can map your application process to the buyer’s journey, placing the hiring organization as the buyer. For jobs with a posting somewhere, creating the persona is pretty easy—they tell you exactly what they need—but if you’re applying to lots of jobs, you may want to group some applications together. The one pager is your resume, full of the features and benefits of you as an employee. If you can get their attention and spark their interest, you’ll proceed to the next step.

Let’s take a look at what we can do to make our resume convert our leads into sales calls—I mean, interviews. 

First, start with the fundamentals

Your resume is made from the raw material of your everyday work performance, so it’s important to collect that raw material. Save performance reviews, evaluations, positive comments and emails, quantifiable data, and anything else that speaks to your performance in a role. You can use these materials to produce a base resume that has everything about your jobs. Because we tend to have a better memory of more recent events, update your resume every six months or so. You might not be looking for a job now, but it’ll be worth it once you are. 

There are some basic things that work across resumes. The first filter for applications comes from an applicant tracking system (ATS), software that weeds out applicant resumes that don’t have the keywords they are looking for. 99% of Fortune 500 companies use an ATS, so ensuring that your resume can easily be read by an ATS is key. That means clean formatting, no charts, no pictures, no vital information in headers or footers as all of these may not be parsed by the ATS system.

At this stage, you’ll want to build a base resume that contains all of the material that will be used to create targeted resumes. Add all of your education, experience, and skills to this document, regardless of how small. You may find they are useful for specific applications or application types. While you won’t likely send this base resume out in applications, it can form the basis of your LinkedIn profile, which serves as the homepage of your professional brand, plus additional context that won’t be on your resume. Most recruiters and hiring managers look at LinkedIn profiles as part of their hiring process. If you don’t have a LinkedIn profile yet, set one up so they can find you.

When putting together this base resume, note a couple of things. First, technical roles rely heavily on your technical skills, so lead with those. Include proficiency levels with these. It can be tempting to only list skills in which you are proficient, but low proficiency skills can highlight your ambition to learn new languages or technology outside of your current role. If you’ve been working in roles that use C#, Java, and Python but list a low proficiency in Rust, a prospective employer may see that as a way to onboard a passionate advocate of a language they are migrating to. 

Second, use the experiences sections to focus on the impact of your work and the soft skills that you used to achieve them. What did you do, how did you do it, but most importantly what was the outcome. Try and use hard numbers—migrated X user records, handled X logins per day, generated X dollars in revenue—if the hiring manager can understand your impact, they can more easily apply that impact to their own business. 

Additionally, if you have side projects or certifications, find a place to include those. The programming hobbies that you have can showcase your skills as well as jobs. Even better, you can include links to code repositories so that potential employers can actually see the work you’ve done. 

Create resumes that tell your stories

Ok, you’ve put together good base material to build some targeted resumes from. Remember, marketing tells a story about the product, which in this case is your services as an employee in a specific role. A resume here needs to quickly tell the hiring manager why you’re a good fit for this role. You may think that your general purpose resume is doing that, but unless it tells a story understandable to the reader, you might be missing the “quickly” part of that directive. 

While creating a completely different resume for every job is a pretty good recipe for job hunting burnout, grouping those jobs into broader types and creating resumes for those types will help get your resume in front of the right people. For example, if you are applying to a full-stack developer position at a small tech start-up and a cloud infrastructure engineer position at a global manufacturing company, your story will be a bit different. It might even be different if you’re applying to two different cloud engineer positions, one at a large company and one at a startup, as they may require different skill sets. 

Exactly what to write on each resume is up to you—you are the “content writer” here, so you make the editorial decisions based on what you think the job posting is asking for. However, if they’re asking for a specific skill, make sure that it’s listed somewhere on the resume. I’ve known recruiters who ask candidates that they’re representing to include specific skill keywords on their resume just so they can get past that first round of ATS processing. 

As for the rest of it, as the cohost of our podcast said, “It depends.” How you structure your resume and describe your experiences, skills, and projects will depend on the material on your master resume and the requirements of the role. For each resume, you can add a tl;dr—a professional summary of your career targeted towards the role you’re applying to. This serves as the “hook” to your resume, a little teaser to get them to read to the end. 

You may think that professional writing means fancy writing, but that’s not true. You want to get your story across as clearly and effectively as possible, which means using plain language and simple sentence construction. You’re not writing a resume to impress them with your verbiage or buffalo them with word counts. You want them to understand what you’re writing, first and foremost. 

While cover letters are not as important for technical roles, they can add a little extra context to a resume. You can use a cover letter to describe the impact you’ve made in previous roles and the impact that you can make for the potential employer. While a good resume can do the persuasive work of a cover letter, some applications require them. Use your judgment as to when to write one, but don’t burn yourself out on them. 

Conclusion

In marketing, we want to make sure we have materials that inform potential customers of how our software and services can help them. To make sure we’re telling the right story to the right people, we create multiple stories for each type of potential customer. 

When you’re looking for a job, your potential customer is the hiring manager. Your resume is often your first opportunity to impress a hiring manager, so the more thought you can put into it, the better. By creating multiple resumes tailored for the various positions, industries, and company sizes that you’re applying for, you’ll have a better chance to get a call back and an interview. Drop your resume writing questions or tips that have worked for you in the comments. 

For more tips on getting your next dream job, check out Indeed’s Career Guide, an online resource designed to help connect people with the information they need to get a job and develop a successful career.

—————–

The Stack Overflow blog aims to serve developers by providing articles that help our readers learn new skills, level up at their jobs, and explore changing technologies. Indeed is a strategic partner of Stack Overflow and this article was created in collaboration with them to assist developers looking to grow in their careers and explore new opportunities.

The post How to use marketing techniques to build a better resume appeared first on Stack Overflow Blog.

Productivity is definitely top of mind, but whether productivity can be measured is an open question. Many organizations, however, have been measuring how fast their teams ship fixes and features to your customer, which is usually called development velocity. Google devised the DORA metrics to put concrete numbers to this abstract concept. 

There are a lot of factors that affect development velocity: CI/CD pipelines, code quality, developer experience, and more. In this article, I want to discuss that last one, as real velocity means solving the problems that haven’t yet been solved. Hard problems need focused, sustained attention from developers. This attention—the time and freedom to focus—is your team’s most valuable resource.

We often describe working with focused attention as a flow state. As described in the book Flow: The Psychology of Optimal Experience by Mihaly Csikszentmihalyi, a flow state allows one to become fully engaged and focused on the task at hand. It leads to better results and greater happiness. But it can only happen when you have the attention to focus fully on whatever it is that lies before you. 

Why can’t we focus at work?

The contemporary workspace, whether in-person or remote, is full of demands on your attention. We have chat programs, email inboxes, and project management apps all throwing notifications our way. In offices, you have other people tapping you on the shoulder and creating general noise (and woe betide those in open offices). Working remotely avoids some of these, but places the entire communication burden on chat and email applications with their little red notifications. These apps promise asynchronous communications, but that doesn’t always happen in practice.

On top of that, we regularly juggle multiple tasks simultaneously. Our attention is pulled in multiple directions, regardless of whether you live an inbox zero life or not. Flow states require sustained attention over time, and our days get chopped into pieces by interruptions and meetings. Researchers at UC Irvine found that on average, office workers switched tasks or were interrupted about every three minutes. Recovering from those interruptions could take workers up to 20 minutes to get back to where they were. In fact, these interruptions could cost individuals up to six hours every day. 

When we have multiple demands on our attention, we try multitasking—splitting our spotlight or shifting it rapidly to focus on the many tasks that come our way. The truth is, we’re bad at multitasking. There’s a mental cost to switching tasks, and that cost translates to up to 40% more time to complete the tasks. Small errors of inattention slip in—typos, missed cues, and quickly forgotten details. Even trying to do only two things at once can mean you do both badly. 

All these interruptions can lead to greater stress and anxiety. Depending on the task, productivity may not suffer, but interruptions may cause us to work faster, which leads to greater time pressure, frustration, and stress. It takes more effort to complete the same amount of work with interruptions in the mix. In the longer-term, enduring regular interruptions—up to 85 per day—can cause decreased job satisfaction and burnout. 

Stress and anxiety form a feedback loop. Both can cause attentional problems, like difficulty concentrating. Without the ability to pay attention to what you’re working on, you may forget steps or not remember solutions that you’ve found. Anxiety has been linked to memory lapses, and if you are constantly forgetting information you’ve received from those chat messages that interrupt people, you may end up interrupting your coworkers over and over. 

It turns out that those coworkers you interrupt are usually the most senior employees at a company. They’ve been there the longest, they have the most historical knowledge about processes and tools. And they suffer the greatest number of interruptions from their coworkers. 

There are no notifications in heaven

Let’s back up. It’s pretty easy to complain about what’s terrible, but harder to imagine what the better world looks like. 

In Flow, Csikszentmihalyi talks about optimal experience as one in which the individual has control over their moment-to-moment experience, bringing their consciousness into a state of order. What makes us miserable about our current state of constant interruptions is that it destroys our control. Our attention is not under our command; instead, it ends up being at the mercy of notification on top of notification, multiple applications chiming for attention. Wrangling them under your control seems to be the root of the solution. 

To do good work during your day, you need some time where you aren’t doing work. Downtime is essential to reducing stress, avoiding burnout, and maintaining a healthy brain. But as we’re all working from home more, work creeps into our home life. Our phones are filled with the same notifications as our work computers except that they can reach us at any hour. You can—and probably should—mute and ignore these notifications after hours; some chat programs set working hours during which notifications will be sent. 

Our always-on work culture doesn’t always let us ignore notifications, even if it isn’t your turn to monitor incident alerts. Many governments have started passing right-to-disconnect legislation to allow employees to ignore any work-related communications that come after working hours. At the very least, we can have control over what we do in our free time. 

But what about within the workday? There are ways to recover some measure of control over the attention of your day. You can use “timeboxing,” where you place a meeting on your calendar that is just for you, a time where everyone understands that you will not respond to notifications. You can be disciplined about statuses and notification settings, indicating when you are available and when you are not. But these techniques only try to manage notifications; what if you could reduce them?

In the end, we may want to rethink why we have notifications in the first place, same as some companies are doing with meetings. Every email, every chat notification is an invitation to collaborate and share information. This is great! Collaboration is a force multiplier that increases employee effectiveness. But notifications are synchronous; they demand attention right now. Better collaboration lets all stakeholders come to the table when they are ready, either scheduled so they can plan around it or asynchronous so they can address it when they have the attention to give it. 

A quieter, more focused workspace

We needed to stay productive by working in the ways that add value instead of reacting to pings, dings, and shoulder taps. That’s mostly writing code, supporting infrastructure, collaborating on solutions, and sharing our hard-won knowledge. One of the reasons that chat apps (and email, to a lesser extent) drain so much of our attention in modern workplaces is that they have become the central application in our workflow, especially for those last two items. Chat apps are immediate; they want your attention right now.

Changing workflows is hard; I’ve seen how developers work in the absence of processes and  it’s all in the chat channels. Implementing a new process is hard; the chat workflow works, albeit in a way that isn’t great for some of our other work. We’ve seen the impact that the Stack Overflow community has had on programmers around the world, and have tried to replicate that for organizations with Stack Overflow for Teams. Of course, we have integrations with many of the major chat programs; think of it as using the Strangler Fig pattern to change workflows. 

There are plenty of other productivity applications that are looking to move the relevant workflows into spaces that are less attention hungry. They have integrations to your favorite chat app too. Of course, the risk here is that your app becomes absorbed in the chat app workflow instead of moving you away from it. 

Ultimately, the problems of attention drain have cultural solutions. You can get the right tools to support that culture, but there’s two questions that an organization needs to answer: (a) How do you collaborate? and (b) How do you share knowledge? So long as the answers to these involve some analogue or digital version of tapping a coworker on the shoulder to get their attention, we’ll remain distracted and constantly trying to get back to the context we had before our interruptions.

The post Modern work requires attention. Constant alerts steal it appeared first on Stack Overflow Blog.

Software engineers use a design pattern to build this separation of concerns into their software architecture. The model-view-controller (MVC) was created in the late 1970s by Trygve Reenskaug when working on Smalltalk. Though it is an older, pre-web design pattern, it is ubiquitous in modern applications. The pattern can be found on the web in both backend and frontend applications, on mobile, desktop, and more. 

The pattern’s name references models, views, and controllers. These are the core building blocks in MVC apps. Each of these components has a specific role and behavior. Let’s take a closer look at the use and implementation of this pattern.

The basics of the pattern

Models provide access and storage for the application data. Typically a model represents a database table. However, models can represent data in any form, they may represent data held in memory, cache, or even a third-party service. Models implement methods to access or modify the data, most commonly these are create, read, update, and delete (aka CRUD) operations.

Views transform the application data in memory to user interface elements. They may also just serialize the data or transform the data to other user interface representations like the native user interface in mobile apps.

Controllers are the glue of the pattern. They are the entry point of the pattern and make models and views work together. All operations are first handled by controllers. Controllers offer high-level actions with required data inputs. Upon an action call, they handle action on the data, pass the data to the models, and then pass the models or the data to views for rendering. All user actions in the application are passed through the controller. The controller decides what should happen next.

MVC on the web

In web applications, controllers handle HTTP requests and responses. They are responsible for pulling the right data from the requests, performing the business logic with error handling, and returning the correct HTTP status and data in the HTTP response. Controllers are typically stateless and own the stateful instances of the models and views. In web applications, controllers create model and view instances when a request comes in or when an action is called, but they don’t own or call other controllers.

Models are responsible for holding the state of the application. In other words this means models store and do low-level work with the data, it can be a class or an object with methods and members holding data or it can be a wrapper of a third-party API. Models frequently have an asynchronous API because they work with low-level asynchronous operations like working with the database, reading from a file, or communicating over a network. Models can work with other models, call them, and use their methods. One model may create composite and more complex operations based on the functionality of other low-level models. As an example, a model may pull some data using another model that wraps a third-party service and store that data into a database table using a model for that table. Models should not work on their own. They don’t call controllers or views or hold any references to them.

Views render the data in a web browser or mobile application. Web application views are stateless and they can be as simple as JSON serializers or as complex as HTML templating engines. Views take data in the form of model instances or custom objects and translate the data into the requested form. Sometimes web applications can support multiple forms of data representation driven by the HTTP Accept header. In MVC, this could just mean employing a different serializer for JSON or XML or a templating engine for the data. Views may call other views to render different types of objects or sub-objects. Views may receive model instances and work with model APIs. However, patterns where a view would directly manipulate data in the model is not recommended. It is always and only the responsibility of the controller to call functions on models that could manipulate the state of the application. Avoid the possibility of this pattern by creating a proxy object that holds the data ready to be rendered in the controller, initializing these objects with data from the models and passing this proxy object down to a view. This way, the controller stays in control of the view’s access patterns and data exposed to the view.

Interfaces

Each component of the MVC design pattern has an interface. This interface provides a way to interact with the objects. In an MVC application on the web, this interaction starts with the controllers. A web server typically interacts with a controller after it receives an HTTP request. The requests are first processed by lower levels of framework or web server logic. This processing handles parsing and deserializing the request and calling a method on the controller.

The web MVC controllers typically implement methods that represent operations provided by the HTTP protocol: GET, POST, PUT, and DELETE. But these HTTP methods don’t need to map to a single method on the controller. The mapping is controlled by routing logic that provides additional options to invoke the right controller method based on the paths or parameters provided in the request URL. In an app using a database, some controllers might implement methods like list or index to wrap model calls and display lists of records with their ids and then a get method to get a specific record based on its id.

For retrieving records, the call comes from controllers to models. In web application frameworks like Ruby on Rails or Django, you may find models providing or implementing several find methods. These methods accept a record id and or other parameters that are passed to queries to lookup specific records. To create records, the models implement create factory methods or allow instancing the model. The models are instanced via class constructor and their properties are set through the constructor parameters or through setter methods defined on the model. The models sometimes wrap specialized record creation into class or static factory methods. These factory methods can provide a nice interface allowing you to quickly understand how the model is instanced in the application code or in tests.

Implementation pitfalls

The most common pitfall when implementing new MVC apps is the lack of respect towards the separation of concerns presented and enforced by the pattern. Developers sometimes decide to cut corners and skip the separation of concerns in order to be quickly done with their changes.

Too smart views

The views in MVC web applications are responsible for translating internal data structures into presentable text format like XML or HTML. Some logic is necessary to do so. You can iterate over arrays of records using loops. You can switch between different types or sections of content for display using branching. Views by nature contain a large amount of markup code. When the markup code is mixed with looping and branching logic, it is much harder to make changes to it and visualize the results. Too much branching or conditions in the view also make the markup hard to read. Spaghetti only works as food, not code. There are two tools that help with this problem: helper functions and view file splitting.

Helper functions are a great way to isolate logic that may be called multiple times, shared, or re-used between views and let you follow DRY principles. Helpers can be as simple as providing an element class name based on some input or as complex as generating larger pieces of the markup.

File splitting is another powerful approach that can improve code readability and maintainability. Every time a view grows too large or too complex, you can break it down to smaller pieces where each piece does only a single thing. Just the right amount of view file splitting can make a big difference in organization of the project.

Overapplying helper functions or view file splitting can also lead to problems, so it is best to drive changes when there’s a reason—and that reason should be a too complex view.

Heavy controllers

Controllers are responsible for implementing actions coming from the frontend, the UI, or the API of the application. For many developers, it becomes tempting to implement data processing logic in the controllers. Sometimes you may see database queries built directly in the controllers and the results stitched together with other data in some complex logic and then the results returned to the views or serializers. Other instances of this problem look like third-party API calls made directly from controllers and the results adjusted or mixed with other data from the local models and returned back to the client. Both of these are cases of heavy controllers.

The controllers should be light on logic. They should process input parameters and pass the adjusted input down to the models. The controllers take the inputs, choose and call the right models and their methods, retrieve the results, and return the results in the right format back to the client. Heavy lifting with data shouldn’t be done directly in the controller methods to avoid code complexity problems and issues with testing.

Light models

Models are built to manage, hold and represent the application data. Every time the application needs to work with the data, it should reach into some model. Sometimes application developers decide to perform data manipulation in controllers, which makes the controller code more complex and creates code maintenance problems. Since models work with data, any incorrect logic on the models may corrupt the data. This is why the models should have the highest test coverage out of the entire app. Well-implemented models are very easy to unit test. 

Application logic that reads and writes data should be concentrated on the related models. For example, if the data queries are spread all over the controller code then it is harder to get good model test coverage. When the developers debug issues like slowness in the database, they may not see all queries performed against a table right away so it may take them longer to optimize the queries and implement better indexing.

Another example can be an interaction with third-party services. Applications frequently require communication with cloud and data providers and this is done through asynchronous calls and APIs. These methods return data that gets processed, altered and saved in the application. It is beneficial to treat these interactions with third-party APIs as model logic and place it on the respective model. This puts the third-party logic to a specific place in the app where it can be quickly covered with integration tests.

Conclusion

The MVC pattern was developed to separate concerns and to keep the application code highly organized. Frameworks like Ruby on Rails that are built on top of this pattern and reap the benefits of strict naming and placement. Closely following this pattern makes projects based on it easier to understand, maintain, and test. Correct usage of its rules reduces developers’ mental load and speeds up any bug fixing or new feature development. Long live the model-view-controller!

The post Keep ‘em separated: Get better maintainability in web projects using the model-view-controller pattern appeared first on Stack Overflow Blog.

The Developer Survey asked respondents to supply country information so we can understand trends in salary roles while controlling for factors like experience and location. India and Brazil saw salary growth of 22.7% and 16.9% respectively across all tech roles, compared to 8% in the U.S. and 3.4% in Germany. India and Brazil’s large pool of educated and skilled workers may have been undervalued prior to the pandemic, and the increased demand for tech was good timing for these two countries in different ways. An increased willingness to hire remote for full-time roles in critical positions may also have driven this change.

The world had high demand for technology in 2022, and India was successfully positioned with the supply of talent to meet it. India’s revenue growth in the tech sector increased 10% year-over-year during the time our 2022 survey was collecting data, compared to 8.6% for the same year in the U.S. 

While Brazil doesn’t have the population that India does, it is certainly supplying an abundance of new talent: data from Brasscom posits that even with  ~250K new IT graduates per year and growing, the demand is still exceeding the supply of new talent available to work in a U.S. time zone. This new tech talent isn’t poised to reap the same salary benefits as experienced developers, but increasing demand and rising inflation  will help drive up their salaries. A lot of the growth is also local.  Venture capital investment into Brazilian companies in 2021 tripled from its pre-pandemic levels. Brazilian tech schools are growing to meet this demand. 

The three-year growth results also show the velocity in developer role value (in USD) and trends where specific technology specializations may start to become localized. Data scientists and machine learning specialists in India have seen about a 32% increase per year in median pay, the highest rate of growth among our top responding developer roles. 

Will data science/machine learning developers in India remain the top role for salary growth in 2023? Will we see continued improvement in prospects for back-end or embedded app developers in Brazil? Brazil and India’s competitive growth in developer salaries will certainly be a trend to look for in our 2023 survey findings. If you haven’t taken part yet, share your experience and add to this year’s Developer Survey.

The post Stories from our survey: Salary in the time of pandemic appeared first on Stack Overflow Blog.

So imagine you’ve gotten past all of those—secured time, obtained sanction from the business, investigated technical feasibility. Maybe you even have a working spike of your solution in a pull request. It feels like the hard part is over. With pride, you stand up in a full-team meeting and announce your brilliant solution…

…to crickets. Your team seems disinterested, or maybe avoidant, or maybe they even resist the idea.

After the meeting, you try to nail down individuals to get their opinions on your solution. Folks respond noncommittally. “Maybe later.” They want to get out of meeting with you. 

What happened? 

Two things happened. First, your solution probably threatens team context—or at least seems like it does. Second, you presented it in the format least amenable to letting the team express that.

Context is king

We reward developers for building new features; for delivering tickets and merging lines of code. But the lines of code themselves are not the currency of technical power. That currency is context—it’s knowledge about the system and how to change it. Who has context on the system is who has power on the team. And that drives more technical decisions than we’d like to admit.

Whole code bases get deprecated on a regular basis because the team considers them “legacy” and “unmaintainable.” Not because they don’t have features or they don’t work: they have features! They work fine! But the current tech team no longer understands them. I’ve seen a team deprecate a service because it was in Rust and rewrite it in a language they considered easier to find devs for: Python. I’ve seen a team rewrite a functioning mail service because the variables in the old one were named “b”, “u”, and “m”—by an architect who had retired. I’ve seen several teams rewrite a monolith as microservices, run into the authentication challenges of microservices, and promptly rewrite the whole thing again as a monolith—but newer this time. 

Just as often, some developer springs up a new service that they’re excited to use to add a new feature or replace an old one. The new service is in some completely different stack than the existing code base because the developer saw a YouTube video and got excited. The stack is Django and somebody went with Vue. The stack is RESTful and somebody went with GraphQL. The database is postgres and someone spun up Neo4j. The rest of the team groans. Why? The new service has features! It works fine! But it increases the amount of context that every team member needs—by an entire framework—to be able to move freely within the system for feature development and maintenance.

Those are extreme examples, but it reflects what’s happening any time you change a code base. Because when individual contributors understand how a system currently works, changes make some part of that understanding obsolete. And the obsolescence of that understanding means an initial investment in rebuilding the understanding to restore one’s ability to maintain the system. To restore one’s power on the team.

The bigger the change, the larger the amount of individual and team context you’re wiping out. Like chemotherapy—designed to attack fast-growing cells like cancer cells but also, sadly, hair follicles and wound-healing cells—large refactors make sweeping changes that wipe out swathes of context. Hopefully most of that lost context is a benefit:  frustration, regret, or concern about some aspect of the code base (I talk a lot about what sorts of changes those might be in this self paced course on technical debt). But in the process of improving the system, the change wipes out team context on the existing system and poses collateral damage to perfectly desirable feature development efforts. 

Context begets power to such a degree that an extreme enough change can upset the team power structure. I once worked on a team where an iOS developer unilaterally changed all navigation in the app from the standard default approach, segues, to a custom, functionally-oriented central navigation module. Not only did this completely change the backbone of the app; it changed it to a custom, un-Googleable approach. The whole team went immediately from being able to Google their questions about screen navigation to depending entirely on Jim. Without Jim, no one could create a new screen in the app. If Jim was out for the day, whole tickets became blocked. People left it that way because no one wanted to upset the now-most-powerful member of the team.

Then Jim got a new job. His last day was a Friday. The next Monday, the team got to work un-refactoring the app back to segues. It took five days. It was worth every expensive, profanity-filled minute to restore the team’s ability to work in its code base. 

Presentation matters

In the movies, failing companies experience their big turnaround after a lone genius stands up in a meeting, slams their fist on the table, and announces “Here’s what we’re gonna do!”

In addition to being dramatic and fulfilling, that mechanism for introducing changes feels efficient: tell everybody about it at once! The problem is, it doesn’t work in real life. When you announce a giant change that threatens teammates’ power in a joint setting where people aren’t expecting it, you set your team up to resist your idea. This is a big surprise to them, and announcing it in a big meeting makes it clear that you expect this change to go through—whether or not they agree with it or even understand it. 

Even if you say you are open to questions, the team at this juncture is completely unprepared to ask questions because you’ve sprung this change on them. They’re stuck operating from whatever context they walked into the meeting with. Since context is king and you’ve given them no opportunity to gain context on your change before you announced it, you’re not demonstrating to them that their precious context is safe with you. This isn’t a collaborative way to suggest a change.

A very real part of the work of making large changes is to socialize them. Tech heads don’t like it, but it’s true. And it makes the difference between the big changes that sail through with support and the ones that get stuck in the mud because the team dragged their feet or outright resisted them.

Socializing your change

What does the socialization process look like? First of all, it involves going to teammates one on one. That doesn’t feel efficient, but it creates an environment in which people are able to ask questions and express concerns. When you chat one on one with someone before announcing a big change, you indicate that their input matters to you. You care how this change is going to affect them. Their precious context is safe with you.

Second, start by talking to this person about the problem you’re trying to solve with your change rather than the change itself. What is the pain in the code base? Does your colleague also experience this pain? What does it look like to them? Establish that the problem you’re trying to solve is bad enough to merit a solution like yours. In the same way it doesn’t make sense to prescribe chemotherapy to someone who doesn’t have cancer (or who has a very localized cancer that can be surgically removed), it doesn’t make sense to propose a huge refactor to fix a problem that no one actually experiences or one person experiences rarely. 

Third, if the person you’re talking to does experience this pain, ask what obstacles they see for fixing it, and what solutions they might propose. Your colleagues have a different perspective on the system than you do; they might think of obstacles and solutions that had not occurred to you. Don’t refute these ideas in this meeting, even if you don’t like them or don’t think they’re worth considering. Your job, in these meetings, is to make colleagues feel heard, so that a) you can propose a solution that is most beneficial for the team and b) your audience will feel compelled to hear you out when it’s your turn.

Once you’ve talked to each of your teammates individually about the problem, take the time to revisit your solution and incorporate their input. If they brought up things you had not considered, think about how to address them. If they brought up solutions that look different from yours, consider the tradeoffs of each. 

Once you have done this, it’s time for another round of talking to teammates. Explain that you’ve been thinking about the problem, and ask what they think of the solution you’ve come up with at the end of your revisiting period. Here, because you requested their input and you’re talking to them one on one, they’ll be more likely to share their objections than they would be in a big meeting. These objections are gold to you: they are the reasons why your team might resist your solution, and you’ve given yourself the chance to address them up-front. To the extent that you can address them, you reduce resistance to the change. For those you cannot address, you have at least made your team feel heard. I offer more ideas about what to say in this meeting right here.

Once you’ve completed a second round of talking to your team, you have reached a more appropriate time to introduce your idea in a large meeting. This introduction, however, should include specific attention to the context problem: how are you going to restore the context that you’re about to destroy by making this change? What preventive and therapeutic treatments will you use to preserve all the context you can and heal all the context you can’t? Are you going to meet with the team to collectively design the solution? Will each pull request require some number of reviews from your team? Will your documentation strategy address any process changes you’re introducing, and will you specifically notify the individuals that the process changes affect?

Your context mitigation strategies will help you address even the hidden objections your colleagues may not have mentioned to you—their worries about their own context and their own power. Your change is not only more likely to get through; you’ll have also strengthened your trust relationships on your team.

These sorts of strategies can feel superfluous to the individual contributor who is accustomed to thinking of the code as the work. To this person, all of this talking and revisiting and reflecting and convincing feels, as Michael Feathers puts it in Working Effectively with Legacy Code, “suspiciously like not working.” But we’re knowledge workers in an environment where context is king. Building a shared understanding, more often than not, is the work. And the further we progress as engineers, and the larger the change we get to implement and oversee, the more critical it becomes for us to integrate this work into our daily habits. At scale, it’s exactly how we’ll get anything done.

The post How do we get a tech team to make a big technical change? appeared first on Stack Overflow Blog.

It’s unlikely that the perfect role is going to come to you; however. You’re going to need to put in the time and effort — which is understandably tough for people with busy schedules. 

We put together this collection of tips to help you stay focused and execute. 

1. Stop making excuses for why you can’t do it 

What if you have no previous experience in climate tech? What if you haven’t taken a single environmental science class? Don’t rule yourself out. Your skills could be the perfect fit for the right role.

Instead of worrying about why you can’t make a transition into climate tech, shift your attitude to finding an employer who will support your transition. 

“Since so many of us in climate have come from tech (myself included), there has been tremendous support from all corners of the climate community to welcome and assist tech workers interested in making the leap,” explains Jonathan Strauss, CEO and co-founder at Climate Draft, a coalition of climate tech companies and VCs, in an article for Business Insider.

2. Don’t expect recruiters to find you

Unlike larger companies with in-house and external recruiting forces (i.e. budget) to deploy campaigns on LinkedIn and other employer branding platforms, climate tech startups may be a bit more behind the scenes.

 Just because you’re not receiving an influx of messages on LinkedIn doesn’t mean the jobs aren’t out there. But you’ll need to look beyond your inbox. 

At the time of writing this article, there were 5,205 jobs posted on the Climate Draft job board. Remember that your objective is to find just one job, in an ocean of many, that’s right for you.

3. Explore climate-tech focused job boards

Climate Draft isn’t the only resource to help job seekers find careers in climate tech. 

As the industry continues to heat up (bad pun), so will the economic movement that ensures continuity of life on our beautiful planet. Unsurprisingly, indie platforms are emerging to help people apply for jobs—and why not? Recently, on the Stack Overflow Podcast, we interviewed one entrepreneur who bootstrapped his niche job board business to $60,000 per month.

Looking for a great job board? Here are some options to help you get started:

• Climate People
• Climatebase
• Climate Jobs List
• Wellfound (formerly AngelList Talent)

Keep your eyes open for more job boards and communities to spring up. As interest in climate tech grows, so will ancillary organizations. If you don’t have time to spend searching on Google, you can try ChatGPT or follow people talking about climate tech jobs on LinkedIn (like this awesome VC, Taj Ahmad Eldridge, who regularly shares insights about sustainability investment trends).

4. Follow the financing

You can streamline your job search by going directly to VC firms. Many will feature their portfolio companies and open roles. 

When browsing through VC firms, it’s helpful to know what stage of company you’re targeting. You may also want to dig deep into your VC’s values and reputation to see if they align with the purpose that you’re seeking out.

Climate Tech VC is a helpful resource where you can browse investors and access a range of educational resources about the market. Crunchbase is another platform to learn about investors and their portfolio companies.

As you research venture financing and deal flow, keep your eyes open for VCs who demonstrate a clear commitment to protecting the environment. If you find investors or firms that interest you, you can consider signing up for their email list 

5. Join a cohort-based community

Especially if you’re working at a job that makes you feel unhappy—or you’re recovering from the emotional setback of getting laid off—it’s a great change of pace to step outside of your comfort bubble. Peer-based learning experiences can help you gather new perspectives.

One helpful resource is Terra.do, which is a global professional network. There are a range of online courses for building new skills, in addition to forming a new professional network. Terra regularly hosts virtual career fairs where job seekers can meet employers.

Independent communities seem to be popping up everywhere, which makes sense given the passion people feel for protecting the environment. Check out The Impactful, which recently launched this Climate Career Toolkit.

6. Create your own job by building direct relationships with employers.

Climate tech is, to some extent, a nascent ecosystem—a new economic movement that people are building from the ground up. The foundational objective is, at its core, to make the planet safely inhabitable for more people, for more time.

As with any new movement, people are figuring their stuff out. Founders, especially, may not be aware of the roles for which it makes sense to hire. If you’re a particularly talented tech-human, there’s nothing stopping you from reaching out to the founding team, saying hello, introducing yourself, and sharing the skills that you bring to the table.

You may find that newer company cultures, especially at the early startup stages, are open to hearing from prospective talent — especially since they don’t have recruiting budgets. You’re actually saving them time by proactively reaching out.

Stay focused on the big picture

The Intergovernmental Panel on Climate Change (IPCC), made up of the world’s leading climate scientists, recently published the sixth and final part of its report. The message is simple, direct, and clear.

“This report is a clarion call to massively fast-track climate efforts by every country and every sector and on every timeframe,” said António Guterres, United Nations Secretary General. Our world needs climate action on all fronts: everything, everywhere, all at once.”

The IPCC is also optimistic that a warming limit of 1.5 degrees is entirely possible by means of focused, collective action. 

How’s that for motivation?

If you end up pursuing a new role at a climate tech startup, you’ll likely need knowledge-sharing infrastructure at your company to make fast engineering decisions.  Learn how Stack Overflow for Teams can help.

The post How to land a job in climate tech appeared first on Stack Overflow Blog.

The nature of these challenges evolve as time goes on, but some things stay consistent in how you view the challenges and how folks can work to make sure that you get back to good reliably. And to be clear, we aren’t talking about run of the mill production bugs, we are talking about issues that are large and sweeping but at the same time delicate and brittle. 

Having done more than my share of organizing and solving some of the big challenges organizations face when these events happen, I have a high-level playbook that I try to make sure my team follows when things fall apart. A lot of these expectations started to take shape during my first big outage as a developer. This helped me understand what people should do as developers, SREs, managers, and everything in between. And the cause of this first big outage: a brand new checkout process on an ecommerce site. All of these takeaways are applicable for folks at all levels, and hopefully can give some insights into what folks in other roles go through.

Step 1. Don’t panic and identify your problem.

My very first “outage” came when I was a developer working on an application that had a new checkout process. It was nothing fancy, but like all applications at one point or another, this key piece of functionality stopped working with our newest launch. As if people not being able to check out and complete sales wasn’t bad enough, shopping carts lost items and product descriptions were showing up blank. Pieces that weren’t within scope or crossed our mind to test stopped working. We immediately grabbed folks into a room to get to work and figure it out. 

Our first instinct right away was “Quick, roll it back!” This was an understandable feeling to have, we introduced problems, and naturally you want to take the problems away. But with quick actions come quick mistakes, and a seasoned senior developer stopped everyone from scrambling to ask the pertinent question: “Well, why isn’t it working?” In my mind I was screaming “Who cares! Our embarrassing mistake is out there for the whole world to see!” But the calm nature and analytical demeanor of this senior developer settled us down and assured us that what we were doing right now in that room was the right thing to do: ask questions and investigate. 

Step 2. Diagnosis and understand the source(s) of your problem

This sounds like an obvious thing, but with concern and panic overtaking the team, not enough folks asked us why things were breaking. The senior engineer left the problem out in the wild for a full 30 minutes after we found it to make sure we knew why it wasn’t working. We checked and double checked exception logs, we did a few different checks with separate workflows, and even checked if there was anything odd at a systems level. After all, we had good development environments setup to replicate production, and things were breaking so double and triple checking ourselves became important. Retracing these steps with new context from the errors we were seeing helped us go through all of these steps in a new light. After we had enough to know what we did wrong, and gathered enough confidence for next time we release, we then started our rollback. It is a delicate balance, but I learned to always take all the opportunities you can before a rollback before you lose your best source of information to find the root of your problem: the actual problem in the wild.

The same senior dev who was tempering our poorer instincts was the one who took “point” or tech lead during this time, while relying on our director to be the incident leader. You will hear many names for these roles, but they are someone who is technical and can help coordinate those efforts (usually a more senior developer) and someone who is responsible for communicating around it and giving air cover for those who may want to take time away from the fixes (usually a director or engineering manager). This is to protect the most valuable resource during a crisis: The time and focus of those who can actually implement the plan to fix.

The more technical person will be there to help set milestones and delegate or divvy up the work that needs to be done. The incident leader, as they are often ironically named, are there to facilitate and not to dictate. I remember hearing from my mentor at the time that the best incident leaders asked two questions: “Where are we at?” and “What do you need?” The first so they could keep people off our back, and the second so the last thing our engineers had to worry about was resources, including time. 

Step 3. Remediation: let’s start working on the problem.

We know we have a problem, we know the source of the problem, now let’s make a plan and fix it. We all love to jump to this step, go right into fixing it. And sometimes we have that luxury for simple issues where the problem is so apparent that confirming and understanding the source of the problem, or problems, are very quick steps, but most times if the problem has made it this far and is this impactful, we need to be more deliberate. Much like how we were potentially shooting ourselves in the foot by instinctually rolling things back too quickly, the same instinct to just fix it can come up, 

This point person is going to help prioritize the work to do, find out where the biggest mitigation steps are, and make sure that other stakeholders have clear expectations of the impact. As a developer working on an issue, you also have a responsibility to hold this person accountable, make sure they give you the resources you need to help figure out the issue. This can be time, access, or other people who have answers you don’t. And this is an important theme throughout this phase: Give the engineers what they need to fix things. Arguably this should be a theme for all of engineering leadership, but nothing more pronounced than when things have gone down and vital workflows have gone silent.  

When we were working on the checkout bug, the biggest piece missing was not information or other developers to help, but focus. This may sound odd, but I am willing to bet it is a familiar feeling to any who have been in the boat with leaders who are panicky or never understood the fallacies of the mythical man month. The leaders were eager for progress updates, and what better way to get those updates than to get everyone in a meeting together four times a day to tell us how things progressed. That means every two hours we lost 30 minutes, had to context switch, and update tracking sheets.When I told my tech lead about this, he immediately had the meeting moved down to once a day for developers, and optional at that. The speed gains from this alone were huge; being able to focus and remove distractions was the bigger factor for remediating the problem. 

Step 4. Verification and learnings

If all goes well, tests are confirmed, and all the valuable information you got from steps 1 and 2 have led to confidence in your new test plans, you can move the fix out to production. Once it is out live, lean on your teammates in all departments to confirm and explore. Interestingly, I have found time and again that if patience and freedom are given to the engineers at the beginning of these incidents, there is a correlated confidence and calmness to the subsequent release and fix. 

However, once the fix is out live and everyone feels strongly about the current state, your work is only half done. Now you need to make sure that expensive, hard earned lessons from this problem grow your whole organization. People often take the measure of a good retrospective from big events like this as problems never happening again, but that is plainly unreasonable to any reasonable person. Often I have found the best learning is how we can DEAL with problems better, not pretend like we can make them go away.

In the end for our checkout issue, it all came down to a missed release step by our deployment team. An honest mistake that can happen to anyone. This doesn’t mean we ignore the issue: we thought of adding redundancy or perhaps trying to automate certain bits more, but that wasn’t the best bit that we learned. Our tech lead was far more focused not on preventing errors, but sharpening our ability to deal with them. Though they wanted to prevent future errors, they saw a lot more room to improve in how we respond to the error. What did we learn about engineer focus time? Where were we able to investigate quickly? Slowly? And even good questions outside of engineering such as who was best at handling comms and what was the info they needed?

There have been shades of this outage throughout almost two decades of my career, and I have no doubt the days of having to deal with things like it are coming to a comfortable middle. But the themes of how to approach it, process it, and most importantly enable my team to tack it tend to be the same. 

The post Don’t panic! A playbook for managing any production incident appeared first on Stack Overflow Blog.

The excitement around these advancements has been intense. We should give the skeptics their due, though: human beings are easily swept up in science fiction. We’ve believed that flying cars, teleportation, and robot butlers were on the horizon for decades now, and we’re hardly discouraged by the laws of physics. It’s no surprise that people are framing generative AI as the beginning of a glorious sci-fi future, making human labor obsolete and taking us into the Star Trek era.

In some ways, the fervor around AI is reminiscent of blockchain hype, which has steadily cooled since its 2021 peak. In almost all cases, blockchain technology serves no purpose but to make software slower, more difficult to fix, and a bigger target for scammers. AI isn’t nearly as frivolous—it has several novel use cases—but many are rightly wary of the resemblance. And there are concerns to be had; AI bears the deceptive appearance of a free lunch and, predictably, has non-obvious downsides that some founders and VCs will insist on learning the hard way.

Putting aside science fiction and speculation about the next generation of LLMs, a realistic understanding of generative AI can guide us to its ideal use case: not a decision-maker or unsupervised agent tucked away from the end user, but an interface between humans and machines—a mechanism for delivering our intentions to traditional, algorithmic APIs.

AI is a pattern printer

AI in its current state is very, very good at one thing: modeling and imitating a stochastic system. Stochastic refers to something that’s random in a way we can describe but not predict. Human language is moderately stochastic. When we speak or write, we’re not truly choosing words at random—there is a method to it, and sometimes we can finish each other’s sentences. But on the whole, it’s not possible to accurately predict what someone will say next.

So even though an LLM uses similar technology to the “suggestion strip” above your smartphone keyboard and is often described as a predictive engine, that’s not the most useful terminology. It captures more of its essence to say it’s an imitation engine. Having scanned billions of pages of text written by humans, it knows what things a human being is likely to say in response to something. Even if it’s never seen an exact combination of words before, it knows some words are more or less likely to appear near each other, and certain words in a sentence are easily substituted with others. It’s a massive statistical model of linguistic habits.

This understanding of generative AI explains why it struggles to solve basic math problems, tries to add horseradish to brownies, and is easily baited into arguments about the current date. There’s no thought or understanding under the hood, just our own babbling mirrored back to us—the famous Chinese Room Argument is correct here.

If LLMs were better at citing their sources, we could trace each of their little faux pas back to a thousand hauntingly similar lines in online math courses, recipe blogs, or shouting matches on Reddit. They’re pattern printers. And yet, somehow, their model of language is good enough to give us what we want most of the time. As a replacement for human beings, they fall short. But as a replacement for, say, a command-line interface? They’re a massive improvement. Humans don’t naturally communicate by typing commands from a predetermined list. The thing nearest our desires and intentions is speech, and AI has learned the structure of speech.

Art, too, is moderately stochastic. Some art is truly random, but most of it follows a recognizable grammar of lines and colors. If something can be reduced to patterns, however elaborate they may be, AI can probably mimic it. That’s what AI does. That’s the whole story.

This means AI, though not quite the cure-all it’s been marketed as, is far from useless. It would be inconceivably difficult to imitate a system as complex as language or art using standard algorithmic programming. The resulting application would likely be slower, too, and even less coherent in unfamiliar situations. In the races AI can win, there is no second place.

Learning to identify these races is becoming an essential technical skill, and it’s harder than it looks. An off-the-shelf AI model can do a wide range of tasks more quickly than a human can. But if it’s used to solve the wrong problems, its solutions will quickly prove fragile and even dangerous.

AI can’t follow rules

The entirety of human achievement in computer science has been thanks to two technological marvels: predictability and scale. Computers do not surprise us. (Sometimes we surprise ourselves when we program them, but that’s our own fault.) They do the same thing over and over again, billions of times a second, without ever changing their minds. And anything predictable and repeatable, even something as small as current running through a transistor, can be stacked up and built into a complex system.

The one major constraint of computers is that they’re operated by people. We’re not predictable and we certainly don’t scale. It takes substantial effort to transform our intentions into something safe, constrained, and predictable.

AI has an entirely different set of problems that are often trickier to solve. It scales, but it isn’t predictable. The ability to imitate an unpredictable system is its whole value proposition, remember?

If you need a traditional computer program to follow a rule, such as a privacy or security regulation, you can write code with strict guarantees and then prove (sometimes even with formal logic) that the rule won’t be violated. Though human programmers are imperfect, they can conceive of perfect adherence to a rule and use various tools to implement it with a high success rate.

AI offers no such option. Constraints can only be applied to it in one of two ways: with another layer of AI (which amounts to little more than a suggestion) or by running the output through algorithmic code, which by nature is insufficient to the variety of outputs the AI can produce. Either way, the AI’s stochastic model guarantees a non-zero probability of breaking the rule. AI is a mirror; the only thing it can’t do is something it’s never seen.

Most of our time as programmers is spent on human problems. We work under the assumption that computers don’t make mistakes. This expectation isn’t theoretically sound—a cosmic ray can technically cause a malfunction with no human source—but on the scale of a single team working on a single app, it’s effectively always correct. We fix bugs by finding the mistakes we made along the way. Programming is an exercise in self-correction.

So what do we do when an AI has a bug?

That’s a tough question. “Bug” probably isn’t the right term for a misbehavior, like trying to break up a customer’s marriage or blackmail them. A bug is an inaccuracy written in code. An AI misbehavior, more often than not, is a perfectly accurate reflection of its training set. As much as we want to blame the AI or the company that made it, the fault is in the data—in the case of LLMs, data produced by billions of human beings and shared publicly on the internet. It does the things we do. It’s easily taken in by misinformation because so are we; loves to get into arguments because so do we; and makes outrageous threats because so do we. It’s been tuned and re-tuned to emulate our best behavior, but its data set is enormous and there are more than a few skeletons in the closet. And every attempt to force it into a small, socially-acceptable box seems to strive against its innate usefulness. We’re unable to decide if we want it to behave like a human or not.

In any case, fixing “bugs” in AI is uncertain business. You can tweak the parameters of the statistical model, add or remove training data, or label certain outputs as “good” or “bad” and run them back through the model. But you can never say “here’s the problem and here’s the fix” with any certainty. There’s no proof in the pudding. All you can do is test the model and hope it behaves the same way in front of customers.

The unconstrainability of AI is a fundamental principle for judging the boundary between good and bad use cases. When we consider applying an AI model of any kind to a problem, we should ask: are there any non-negotiable rules or regulations that must be followed? Is it unacceptable for the model to occasionally do the opposite of what we expect? Is the model operating at a layer where it would be hard for a human to check its output? If the answer to any of these is “yes,” AI is a high risk.

The sweet spot for AI is a context where its choices are limited, transparent, and safe. We should be giving it an API, not an output box. At first glance, this isn’t as exciting as the “robot virtual assistant” or “zero-cost customer service agent” applications many have imagined. But it’s powerful in another way—one that could revolutionize the most fundamental interactions between humans and computers.

A time and place for AI

Even if it always behaved itself, AI wouldn’t be a good fit for everything. Most of the things we want computers to do can be represented as a collection of rules. For example, I don’t want any probability modeling or stochastic noise between my keyboard and my word processor. I want to be certain that typing a “K” will always produce a “K” on the screen. And if it doesn’t, I want to know someone can write a software update that will fix it deterministically, not just probably.

Actually, it’s hard to imagine a case where we want our software to behave unpredictably. We’re at ease having computers in our pockets and under the hoods of our cars because we believe (sometimes falsely) that they only do what we tell them to. We have very narrow expectations of what will happen when we tap and scroll. Even when interacting with an AI model, we like to be fooled into thinking its output is predictable; AI is at its best when it has the appearance of an algorithm. Good speech-to-text models have this trait, along with language translation programs and on-screen swipe keyboards. In each of these cases we want to be understood, not surprised. AI, therefore, makes the most sense as a translation layer between humans, who are incurably chaotic, and traditional software, which is deterministic.

Brand and legal consequences have followed and will continue to follow for companies who are too hasty in shipping AI products to customers. Bad actors, internet-enabled PR catastrophes, and stringent regulations are unavoidable parts of the corporate landscape, and AI is poorly equipped to handle any of these. It’s a wild card many companies will learn they can’t afford to work with.

We shouldn’t be surprised by this. All technologies have tradeoffs.

The typical response to criticisms of AI is “but what about a few years from now?” There’s a widespread assumption that AI’s current flaws, like software bugs, are mere programming slip-ups that can be solved by a software update. But its biggest limitations are intrinsic. AI’s strength is also its weakness. Its constraints are few and its capabilities are many—for better and for worse.

The startups that come out on top of the AI hype wave will be those that understand generative AI’s place in the world: not just catnip for venture capitalists and early adopters, not a cheap full-service replacement for human writers and artists, and certainly not a shortcut to mission-critical code, but something even more interesting: an adaptive interface between chaotic real-world problems and secure, well-architected technical solutions. AI may not truly understand us, but it can deliver our intentions to an API with reasonable accuracy and describe the results in a way we understand.

It’s a new kind of UI.

There are pros and cons to this UI, as with any other. Some applications will always be better off with buttons and forms, for which daily users can develop muscle memory and interact at high speeds. But for early-stage startups, occasional-use apps, and highly complex business tools, AI can enable us to ship sooner, iterate faster, and handle more varied customer needs.

We can’t ever fully trust AI—a lesson we’ll learn again and again in the years ahead—but we can certainly put it to good use. More and more often, we’ll find it playing middleman between the rigidity of a computer system and the anarchy of an organic one. If that means we can welcome computers further into our lives without giving up the things that make us human, so much the better.

The post AI isn’t the app, it’s the UI appeared first on Stack Overflow Blog.

If you find yourself on the receiving end of a layoff—or feeling existential dread more generally—the timing might be right for a major life change. 

Remember that for centuries, in times good and bad, philosophers have provided humanity comforting wisdom that even though it’s impossible to control the external world, you can control your own thoughts, intentions, and actions. As a technologist with agency, it’s well-within your control to seek out a job that is personally fulfilling and makes a positive impact in the world.

If you’re stuck in a job that lacks purpose, maybe now’s a good time to start thinking about switching up your career to one with more potential for good. How about climate tech?

So what could a career in climate tech look like?

The answer to this question is pretty straightforward. It’s a lot like the job that you already have, even if you do not have a background in sustainability, environmental science, or an adjacent field. Your skills are more transferable than you may realize.

Consider the experience of Kevin Cianfarini, a 27-year-old senior Android developer at Octopus Energy, a renewable-energy supplier. 

• He didn’t have to take a pay cut
• He received multiple offers with higher salaries compared to his prior role at a fintech company
• It was a sense of meaning and purpose that inspired him to join Octopus—something that he did not experience at the fintech company where he previously worked
• At Octopus, he earns $185,000 a year plus a $40,000 stock option grant that vests over four years

As another example, Cassandra Xia left her software engineering job at Google where she worked on projects related to click-through rate predictions, according to an article in protocol. A year after leaving her position, she joined Evergrow, a climate fintech startup, as the head of engineering. What energizes her about the position is that climate “is a problem that’s not figured out, and there isn’t really an interactive solution,” in contrast to Google where “the problems have already been picked over.”

Cianfarini and Xia aren’t alone. According to Justin Hardin, co-founder and CTO at the job board Climatebase, more than 500,000 people have used the platform to seek out and apply for a climate tech job.

You won’t know what interesting opportunities are out there unless you look.

A climate tech economy is starting to take shape

It turns out that the foundations of a climate tech economy are starting to form, thanks to investors who are realizing the importance of committing capital to the problem—despite the overarching trend of a venture funding downturn. 

“Climate tech funding in 2022 represented more than a quarter of every venture dollar invested in 2022, even though the market is not yet efficient at meeting climate objectives,” explains PwC.

“This represents aggregate funds raised for climate tech since the start of 2018 of US$260 billion, over which more than US$50 billion has come in 2022,” explains the report.

Specifically, PwC observes increased investment in carbon capture, removal, utilization, and storage. 

Catherine Boudreau, a writer at Business Insider, interviewed several VCs who explained that “a new era is underway because of the passage last year of the Inflation Reduction Act, which includes some $370 billion in federal subsidies over a decade aimed at expanding renewable energy and boosting manufacturing in the US.”

“Climate is like the internet in that it’s going to disrupt every corner of the global economy,” explained Andrew Beebe in Bourdreau’s article, the managing director of Obvious Ventures, which has more than $1 billion in assets under management. “It might be like a dirty little secret in Silicon Valley that some of the best targets for venture capital are where governments will regulate the fastest.”

Where there’s growth, there is a likelihood of new opportunities emerging.

Focus on signals—ignore noise

You might also be hearing contradictory perspectives about a downturn in VC climate tech funding. 

Remember that stories like these are created by humans (and sometimes AI). One explanation for these varying perspectives is that reporters may be analyzing data over different time horizons. There may also be differences across data sources that analysts are using. We may never know the reason, as financial paper trails are ultimately in the decision-making hands of VCs choosing to fund companies. All we know is what we can see and what we are told. TLDR: Who knows what’s up, really?

No matter what macroeconomic data you’re examining, whether positive or negative, it’s a good idea to look beneath the surface —to discover signals in the midst of noise, overall. Even in areas where funding may be stagnating or slowing down, you can still find data points that are relevant to your job search.

“Funding for climate tech was relatively flat from 2021 to 2022, compared with investment slowdowns in other sectors, but a more granular look at the data suggests that resources have begun to shift away from previously dominant segments like transportation toward emerging technologies, according to Climate Tech VC (CTVC),” explains one article in Tech Brew. 

Despite these macro trends, remember that your goal is to simply finda job. So make your own judgment calls. 

Final words of empowerment

The economy feels larger than life sometimes. That’s because it is. After all, there are eight billion people in the world—and a large number are transacting with each other in some way on a regular basis. Yes it’s true, according to the United Nations, that “that climate change is a grave and mounting threat to our wellbeing and a healthy planet.”

Despite how massive (and out of control) problems seem sometimes, remember that your mind has the power to shape reality. Brain formation travels at 268 miles per hour, according to researchers, with a storage capacity that’s unlimited. Imagine the power of putting that intellect to work to solve the climate crisis. 

After all, the IPCC says that despite the potential for doomsday scenarios, the 1.5°C warming limit is still attainable. 

As Yoda would say, “Do or do not do. There is no try.”

Maybe with enough brain power from the tech sector, humanity can achieve this goal. 

Next week, we’ll publish a part two in this series, sharing practical, tactical tips to help you find a role that’s right for you. 

Climate tech is a new industry, which means that you’ll be responsible for building technical architecture from the ground up. Learn how Stack Overflow for Teams can help support problem-solving at your organization.

The post Looking for job perks? How about saving the world? appeared first on Stack Overflow Blog.

[Note to readers: before leaving a comment about someone with no skills wasting your time, refer back to the title and remember that you can’t insult a person who has already completed a perfect self-own.]

Anyway, over the last month, I decided to try diving back in, inspired by a steady stream of posts I saw on social media that showed people writing natural language instructions that today’s AI assistants can transform into working code. Maybe, I thought, my basic understanding of how a web app gets built will allow me to engineer the right prompts, and the AI can handle writing and debugging the actual code. 

The app I had in mind is fairly simple: people can list let others know when they are going to be at the local dog park. After some discussion, I found that this is a basic CRUD app:

• Create: a web form where users can input the name of their dog and the time they plan to arrive at the park. 

• Read: Other users can see who is planning to visit and what time they’ll be there. 

• Update: I’m not even trying to get this fancy! Also, no take-backsies!

• Delete: At the end of each day, the database deletes all entries so the slate is blank for the following morning. 

Normally, to get started, I would brush up on my paltry HTML skills and prepare a sacrifice to the JavaScript gods, but,  in 2023, I can just ask an AI: 

I want to build a single page web app that will allow users to submit their name, their dog's name, and the time they intend to visit the local dog park. After users submit this information, it is displayed for anyone who visits the page. I need front end code and a database on the backend that can store state. 

First - Please explain how you would create this app using pseudo-code examples. 
Second - write the code for this app and share it as a single code snippet.

First, let me outline the general structure and pseudo-code for the app.
Frontend (HTML, CSS, JavaScript)
a. Create an HTML form for user input (user's name, dog's name, and intended visit time)
b. Style the form using CSS
c. Add JavaScript to handle form submission and send data to the backend API
Backend (Node.js with Express and a database like MongoDB)
a. Set up a Node.js server with Express
b. Create API endpoints to handle data submission and retrieval
c. Connect to a MongoDB database to store and fetch the submitted data
Connect frontend and backend
a. Use JavaScript to send data from the frontend to the backend API
b. Display the fetched data on the frontend

Ok, let’s drop the code it generated into my WordPress CMS just to take a peek and…by George, it works.

<!-- index.html -->
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Dog Park App</title>
    <style>
        /* Add your CSS styles here */
    </style>
</head>
<body>
    <h1>Dog Park App</h1>
    <form id="dog-park-form">
        <input type="text" id="userName" placeholder="Your name" required>
        <input type="text" id="dogName" placeholder="Dog's name" required>
        <input type="time" id="visitTime" required>
        <button type="submit">Submit</button>
    </form>
    <ul id="visitors-list"></ul>
    <script>
        // Add your JavaScript code here
    </script>
</body>
</html>

Looks a little janky, though. Let’s clean things up. 

Edit the code above so that the three text fields are centered on the page and there are three line breaks between the name and time fields. 

And just like that, the changes are made. I’ll spare you that code snippet as it’s largely the same.
Looks better, but it’s still missing something. I know! 

Modify the code above so that an ASCII image of a dog appears over the text fields for name and time. The ASCII image of the dog is centered on the page.

<!DOCTYPE html>
<html>
<head>
	<title>My Webpage</title>
	<script>
		// This function clears the text area every 24 hours at midnight
		function clearText() {
			var currentTime = new Date();
			var hours = currentTime.getHours();
			var minutes = currentTime.getMinutes();
			var seconds = currentTime.getSeconds();
			if (hours == 0 && minutes == 0 && seconds == 0) {
				document.getElementById("output").value = "";
			}
		}
	</script>
	<style>
		/* This style centers the text fields and the dog image on the page */
		.container {
			display: flex;
			flex-direction: column;
			align-items: center;
		}
		input[type="text"], textarea {
			display: block;
			margin: 0 auto;
		}
		.ascii-dog {
			font-size: 50px;
			text-align: center;
			margin-bottom: 50px;
		}
	</style>
</head>
<body onload="setInterval('clearText()', 1000);">
	<div class="s-code-block container">
		<div class="s-code-block ascii-dog">
			<pre>
  __      _
o'')}____//
 \"''''''''\\
   \        \
    )________\
    |_|_|_|_|_|    
			</pre>
		</div>
		<label for="name">Name:</label><br>
		<input type="text" id="name"><br><br><br>
		<label for="time">Time:</label><br>
		<input type="text" id="time"><br><br><br>
		<button onclick="displayText()">Submit</button><br>
		<label for="output">Output:</label><br>
		<textarea id="output" rows="5" cols="50" readonly></textarea>
	</div>
	<script>
		// This function displays the text from the input fields in the output area
		function displayText() {
			var name = document.getElementById("name").value;
			var time = document.getElementById("time").value;
			var output = document.getElementById("output");
			output.value += name + " - " + time + "\n";
		}
	</script>
</body>
</html>

How sweet it is. This is what  mean when they say AI is a tool to unlock human creativity and potential, not to replace us.

So far so good, but the input doesn’t persist. I need some kind of backend to record my state. 

I want to add a database to the code above so that I can store the state of the output and display all text submitted to users on the client side. I want the database to contain two rows, one for "name" and one for "time". When users submit a "name" and "time" the database stores this state and it's displayed in the output section. The database clears these rows once a day at midnight Eastern time. 

A few seconds later the AI spits back a code block that uses jQuery and PHP with a MySQL database. For a hopeless programmer like me, it’s fascinating to see that I’ve now got access to building blocks in multiple languages. I can’t even master one on my own!

That said, the AI can’t solve the issue of ME being an intractable vortex of bugs and errors, attracting obstacles to functioning code like a magnet picks up nails. I go through the process of setting up an AWS account and following the online tutorial for creating a MySQL database, but can’t actually get it to connect to the workbench. 

The AI tries to be kind. I ask it to rewrite the code for a NoSQL database and it helpfully switches from PHP to Node.JS without missing a beat. I spin up a MongoDB Atlas cluster, but am lost again in a sea of error messages. 

Eventually I try a clean slate. I start a new dialog with the AI and ask it to create a simple app that will work with Firebase. I’ve learned a few things from my first two attempts, so I understand a little better how the pieces are supposed to fit together. It’s easier for me to work in the terminal and to have a clean file and folder structure. At 9:30pm on a random Tuesday, I’ve got a functioning CRUD app—input your info about your next visit to the dog park, refresh the page and there it is for the world to see. 

You can accuse me of talking my book, but in all honesty, there were some things the AI simply couldn’t solve for. It insisted I needed the code below:

​​<!-- The core Firebase JS SDK is always required and must be listed first -->
  <script src=“https://www.gstatic.com/firebasejs/9.6.8/firebase-app.js”></script>
  <!-- Add Firestore (database) -->
  <script src=“https://www.gstatic.com/firebasejs/9.6.8/firebase-firestore.js”></script>

But it didn’t connect that to an error I kept getting:

Uncaught SyntaxError: Unexpected token 'export'

A quick web search and I found the following question and answer pair. Do I understand why deleting that code fixed the issue? Of course not. But it let me get closer to a working solution. (Ed. note – removing this code fixed the error, but caused an issue with the window refreshing when a user clicked the submit button. I later added it back in, and got refresh working, and didn’t encounter the error a second time.)

The AI also had a bunch of ideas about how to solve for this one:

Uncaught SyntaxError: Cannot use import statement outside a module

I tried three or four of its solutions, but we got nowhere. But the web had what I needed. I fed the context of the answer back in and voila, a working web app.

Index.html:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Palatine Bark Play Scheduler</title>
  <style>
    * {
      box-sizing: border-box;
    }
    .banner {
      background-color: #4CAF50;
      color: white;
      text-align: center;
      padding: 1rem;
      margin-bottom: 1rem;
    }
    .container {
      display: flex;
      justify-content: space-around;
      flex-wrap: wrap;
    }
    form {
      display: flex;
      flex-direction: column;
      align-items: center;
      padding: 1rem;
      width: 50%;
    }
    .data-section {
      width: 50%;
      display: flex;
      flex-direction: column;
      align-items: center;
    }
    .data-container {
      width: 100%;
      padding: 1rem;
      text-align: center;
    }
    img {
      max-width: 100%;
      height: auto;
      margin: 2rem 0;
    }
    /* Media query for smaller screens */
    @media screen and (max-width: 767px) {
      form, .data-section {
        width: 100%;
      }
    }
  </style>
</head>
<body>
  <div class="s-code-block banner">
    <h1>Palatine Bark Play Scheduler</h1>
  </div>
  
  <div class="s-code-block container">
    <form id="data-form">
      <label for="name">Your Name:</label>
      <input type="text" id="name" required>
      
      <label for="dog-name">Your Dog's Name:</label>
      <input type="text" id="dog-name" required>
      
      <label for="time">Time:</label>
      <input type="time" id="time" required>
      
      <button type="submit">Submit</button>
      <img src="https://firebasestorage.googleapis.com/v0/b/palatine-bark.appspot.com/o/good%20boy.png?alt=media&token=XXXXXXXX" alt="Good Boi" />
    </form>
    <div class="s-code-block data-section">
      <h2>Today's Data</h2>
      <div class="s-code-block data-container">
        <div id="data-display"></div>
      </div>
    </div>
  </div>
<!-- Firebase App (the core Firebase SDK) -->
<script src="https://www.gstatic.com/firebasejs/9.6.8/firebase-app-compat.js"></script>
<!-- Firebase Firestore -->
<script src="https://www.gstatic.com/firebasejs/9.6.8/firebase-firestore-compat.js"></script>
<!-- Your custom JavaScript file -->
<script type="module" src="app.js"></script>
</body>
</html>

App.js:

// Initialize Firebase
const firebaseConfig = {
  apiKey: "MYAPIKey",
  authDomain: "coolest-dogpark-scheduler.firebase.com",
  projectId: "Bens Awesome Project Goes Here",
  storageBucket: "Ben's Bucket",
  messagingSenderId: "NUMBERS",
  appId: "LOTTA CHARACTERS"
};

// Initialize Firestore (Database)
const app = firebase.initializeApp(firebaseConfig);
const db = firebase.firestore(app);

// Form and data display
const form = document.getElementById('data-form');
const dataDisplay = document.getElementById('data-display');

// Handle form submission
form.addEventListener('submit', async (event) => {
  event.preventDefault();
  
  const name = document.getElementById('name').value;
  const dogName = document.getElementById('dog-name').value;
  const time = document.getElementById('time').value;
  const timestamp = firebase.firestore.Timestamp.now();
  
  await db.collection('data').add({ name, dogName, time, timestamp });

  // Clear form
  form.reset();
  location.reload();
});

// Fetch data and display
async function fetchData() {
  const today = new Date();
  today.setHours(0, 0, 0, 0);

  const querySnapshot = await db.collection('data')
    .where('timestamp', '>=', firebase.firestore.Timestamp.fromDate(today))
    .get();

  dataDisplay.innerHTML = '';

  querySnapshot.forEach(doc => {
    const data = doc.data();
    dataDisplay.innerHTML += `<p>${data.name} - ${data.dogName} - ${data.time}</p>`;
  });
}

// Call fetchData() initially and whenever the form is submitted
fetchData();
form.addEventListener('submit', fetchData);

Lots of folks are making claims about AI replacing programmers, but for the time being, the effect on my usage was the opposite. The AI was the inspiration to try coding again; it was a springboard that got me going and handled some basics. But it wasn’t enough to get me to the finish line on its own. When things started to get tricky, I was pinging colleagues in chat for help, or searching up error messages and sifting through answers from Stack Overflow, technical documentation, and blog tutorials.

What do you call something that can string together complex concepts and provide useful answers to questions about everything from history to poetry to programming? A friend of mine suggested “thought calculator” and I’ve fallen in love with this phrase. It perfectly captures this new technology’s power, and its limitations. 

The AI isn’t sentient or autonomous, as far as we know, but it has a form of intelligence which you can leverage as a tool. Coding is one of its best applications, because it uses textual instructions to produce an objective output. We can argue forever about the quality of an AI’s marketing copy or literary essay, but with programming, setting aside for a moment the important question of code quality, we can test to see if it produces the desired result. 

As our CEO announced last week, we’re thinking about ways we can use the power of generative AI to continue building a knowledge community that is open and accessible to all. Some folks protested that it would be wrong for us to train AI on Stack Overflow data, but the reality is lots of people are already doing that. We could sit around and do nothing while other organizations ingest the web into an algorithmic black box, or we could try to build a system where human input is still recognized and rewarded, where the output created by the collaboration of a human and their thought calculator is something everyone can then leverage and learn from. 

We are all in the process of figuring out how to use these new tools in a safe and ethical manner. We would love to hear from the community about how you’re approaching the issue. If you’ve been experimenting with AI to help you write code, let us know what you think in the comments. I’ll try to be back in a week or two with an update, once my army of autonomous agents completes my plan for world domination.

The post The worst coder in the world tries an AI sidekick appeared first on Stack Overflow Blog.

At its most basic, Know Your Customer (KYC) is a due diligence process used to verify that the person is who they say they are and that data they have shared is correct (e.g. phone number, email, and address). KYC regulations are in place to prevent criminal activities such as identity fraud, money laundering, and other financial crimes. However, the compliance and implementation of these regulations comes with a host of challenges. Some of these challenges include:

• Customer identification: You’ll need to compare the customer’s likeness with a photo on an approved ID card (usually government-issued like a passport or driver’s license). There is additional friction if this happens in-person.
• Documentation verification: Making sure the picture matches the person isn’t enough; the government-issued ID of the individual must be checked for forgery or tampering. This requires the person carrying out the verification to be trained to recognize these signs and look for the presence of appropriate watermarks, etc.
• Address verification: As an additional check on their documents, the customer must provide proof that they live at the address on the government-issued ID documents (Proof of Address or POA) usually by providing recent utility bills. This is getting trickier as many of us opt for electronic bills, so the customer may have to request one-off letters from approved sources to prove they live at the address they’ve provided.
• Verify contact details: Finally, you’ll need to check that the email address and phone number provided are correct and belong to the customer. A common approach is to support a one-time-passcode (OTP) system that allows a customer to enter the passcode they receive and have it linked to the account profile that is being created.

For software developers, all of these steps slow down your user onboarding. Many of these verification steps are manual and some require the customer to come in to meet face-to-face, which adds friction to the onboarding experience. This manual process creates additional costs for the organization performing the validations. 

KYC can have multiple layers of customer verification such as anti-money laundering checks and risk assessments—for example, using Ekata to spot if an address provided has being used repeatedly before for fraudulent transactions. It can even include checking an individual’s crypto footprint for inappropriate transaction history or if they’re on a sanctions list (Ciphertrace). 

For the purposes of this post, we’re going to look at how Mastercard Open Banking Account Owner Verification APIs can help support the most fundamental step in a KYC pipeline: verify the digital identity of your consumers or small businesses based on their provided name, address, and contact details. 

But the banks have already done KYC for their customers!

Yes, the banks have built a rigorous KYC practice to verify customers before they open any account. Banks adhere to a range of regional privacy, security, and anti-money laundering regulations. Many of us have had to go through that process of providing proof of identity and address as well as completing registrations using OTPs sent to our email and phone. An important consideration is how we as customers also keep this information up to date, so we don’t miss any important communications from our bank.

The timeliness, quality, and accuracy of the customer data that the bank holds is very useful, but also valuable because of the time and effort invested in collecting it and keeping it up to date. The US Open Banking API enables you to unlock this value. Specifically, we can use the Account Owner Verification API to access this valuable data.

Using Mastercard Account Owner Verification APIs to leverage the bank’s KYC efforts

Open banking empowers users (consumers and small businesses) to access, use, and benefit from their own financial data. They can control what accounts can be seen, how long that access can be granted, and for what purpose— things like opening new accounts, securing loans, improving credit scores, and enabling consumer choice in payments. 

The user (your customer) feels safe doing this because they are authenticating directly with their bank using their online banking credentials, giving them confidence that the third-party won’t see or store them.

The US Open Banking API handles a trusted connections to the banks, provides a secure dialog so the account holder can authenticate, and caches the financial data so when you execute a query it comes back quickly (avoiding a round trip to the bank every time).

The US Open Banking API provides a range of features including account aggregation, payment enablement, and confidence scores that can be used for financial services use cases like lending, payments, and account opening. 

To help support your KYC pipeline, we can use Mastercard’s Account Owner Verification API, which returns bank pre-verified data (i.e. name, email, address and phone number) along with identity insights and an identity risk score based on users’ activity pattern and its association to help detect fraudulent behavior, thereby instantly verifying that the user (bank account owners) are genuine and help mitigate online fraud. This makes it exponentially harder for criminals to use real or fake IDs to commit fraud and significantly reduces identity fraud by enhancing the effectiveness of fraud detection systems for account openings, me-to-me (M2M) transfers, peer-to-peer (P2P) transfers, bill payments, and other transactions. 

How it works

1. Your server registers a customer with Open Banking and receives an ID for this customer.
2. With this customer ID, your server can generate a redirect URL for loading the connect experience that will allow the customer to connect to their bank.
3. Your frontend application redirects the customer to connect using the generated URL from step 2.
4. The customer logs into their financial institution using their bank credentials through the connect experience.
5. The customer grants permission for their financial data to be accessed.

Once the customer has granted you access to the account, you make a call to the Get Account Owner Details API.

The Get Account Owner Details API is aligned with the Financial Data Exchange (FDX) standards. The FDX is dedicated to unifying standards for secure and convenient access of user-permissioned financial data sharing throughout the financial services industry. This allows the API to return the payload about the user (consumer or small business) in a standard format with the following details:

• Names: The names of the account holders. It will also indicate whether it is a person, business, or other entity.
• Address: The standard format for a US address, broken into constituent parts, which will also have an entry to say if it is a business or home address.
• Phone numbers: An array of contact phone numbers along with type (Personal/Business).
• Email addresses: An array of email addresses and their type (Personal/Business).
• Documentation: An array of the document proofs provided: tax identifier SSN, government-issued ID (passport number, visa number, driver’s license number, etc.) Quick sidebar: though this element is a required field in some regions, it is not mandatory in the US, and as such, it is at the discretion of the bank.

Example response

{
  "holders": [
    {
      "relationship": "AUTHORIZED_USER",
      "ownerName": "John Smith, PhD",
      "firstName": "John",
      "middleName": "L",
      "lastName": "Smith",
      "suffix": "PhD",
      "nameClassification": "person",
      "nameClassificationconfidencescore": 100,
      "addresses": [
        {
          "ownerAddress": "434 W Ascension Way",
          "type": "Home",
          "line1": "434 W Ascension Way",
          "line2": "Suite #200",
          "line3": "UT 84123",
          "city": "Murray",
          "state": "UT",
          "postalCode": "84123",
          "country": "USA"
        }
      ],
      "emails": [
        {
          "isPrimary": true,
          "email": "myname@mycompany.com",
          "emailType": "Personal"
        }
      ],
      "phones": [
        {
          "type": "HOME",
          "country": "61",
          "phone": "1-801-984-4200"
        }
      ],
      "documentations": [
        {
          "taxId": "123-45-7890",
          "taxIdCountry": "USA",
          "governmentId": "123456789"
        }
      ]
    }
  ]
}

Congratulations! Now you have the data (verified by a bank) necessary to enhance your customer onboarding process.

How are the account holder’s credentials secured?

The US Open Banking Service uses OAuth when integrating into the bank APIs and as a third party in that process, it never sees the account holder credentials. When the account holder successfully authenticates with their bank and selects the account they want to allow access to and for how long, the service will get a token that it encrypts and stores until it needs to access the data. It will be limited to just the data that the account holder has explicitly granted access to.

If you are using the Get Account Owner Details API as part of a one-off customer onboarding verification step, you can set the token up to be a single-use token to further limit access to the account owner data. If, however, it is part of a broader Account Opening process, Account Aggregation or similar Open Banking use case, you can configure the token to live for a longer period of time.

How do I get started?

Check out this Quick Start Guide on Mastercard Developers to get up and running with the US Open Banking API and then use the Get Account Owner Details API call to pull the data necessary to power your KYC process.

The post Instantly verify your customers online with Open Banking APIs appeared first on Stack Overflow Blog.

At the beginning of the year, Shopify took drastic steps to reduce their meeting burden. They automatically canceled all meetings of three or more people, a total of around 12,000 calendar series and events that would have taken up roughly 322,000 person hours. Chaotic and drastic? Maybe. But as Kaz Nejatian, Shopify’s COO and VP product wrote in the email announcing the change to employees, “We’ve unleashed the ‘chaos monkey’ before and have always come away faster—faster at shipping, at making great decisions, at getting to results and impact. No one joined Shopify to sit in meetings.”

Chances are that no matter where you work, you didn’t join that company for the meetings. You wanted to build software. But the meetings became part of that, and the more senior you are, the more meetings you probably get asked to attend. 

In this article, we’re going to take a look at the productivity impact of meetings, reevaluate why we have meetings at all, and consider ways to make meetings better (or avoid them altogether). 

One third of meetings are unnecessary

Unless we’re actively multitasking during a meeting (which humans do badly), we’re blocked from other parts of our work: writing code, debugging processes, or designing new features. So if we’re asking for our coworkers’ time, that time should be spent productively. Professor Thomas Fritz of the University of Zurich ran several studies about how software developers perceive their own productivity across their activities. Perceptions across all three studies found that developers view slightly more than half of their meetings negatively. For more from Prof. Fritz, check out our recent podcast with him:

Plenty of things about meetings can make attendees feel they’re wasting their time: guest lists that spiral out of control, overwhelmingly negative participants, and meeting participants who stray off-topic. Otter.ai and Stephen G. Rogelberg, Professor of Organizational Science, Management at UNC Charlotte, found that developers in bad meetings report feeling “frustrated” and “annoyed.” Even good meetings can turn bad if run poorly. 

In that same study, they found that developers see about one-third of all meetings as unnecessary—they want to decline 31% of meetings, but only nix 14%. Worse yet, bad meetings not only affect how developers feel about their jobs, they cost organizations money—an estimated $25,000 per employee per year. You can estimate how much any given meeting costs your company with this calculator. And that’s just the direct costs of the meeting, regardless of how disruptive it is to the rest of a developer’s day. 

Meetings don’t happen in a vacuum; often they happen back-to-back with other meetings. Microsoft’s Human Factors Lab found that back-to-back meetings cause a great deal of stress and make people worse at meetings. 

This wasn’t just a subjective perception. They strapped an EEG cap that measured brain waves to 14 volunteers and either set them up with four consecutive meetings or four meetings with a ten minute meditation break in between. They found that back-to-back meetings caused stress to build up and caused attendees to lose focus and engage worse over time. Surprisingly, one of the biggest sources of stress came from the transition between meetings, as participants tried to switch gears without adequate time. 

These factors combine to drag down the company as a whole. In a study of 20 organizations in manufacturing sectors, Simone Kauffeld of Technische Universität Braunschweig and Nale Lehmann-Willenbrock of the University of Amsterdam found that bad meeting behaviors were associated with lower levels of market share, innovation, and employment stability. A company that doesn’t have meeting discipline may soon find their best employees fleeing to greener pastures as their balance sheets slowly drift into the red. 

All these downsides to meetings may make you wonder why we have meetings at all. 

Wait, why do we have meetings at all?

Obviously, nobody schedules a meeting to torture their coworkers. Meetings are the standard tool for collaboration in companies, ways to find consensus and make a decision, share knowledge, or brainstorm solutions—essentially, they are pop-up communities of practice. Get everyone in a room and talk it out. 

The Harvard Business Review quotes an unnamed pharmaceutical executive who sums up the pro-meeting bias that many people in leadership positions hold: “Our abundance of meetings at our company is the cultural tax we pay for the inclusive learning environment that we want to foster…and I’m OK with that. If the alternative to more meetings is more autocratic decision-making, less input from all levels throughout the organization, and fewer opportunities to ensure alignment and communication by personal interaction, then give me more meetings any time!”

In fact, these are all things that I’d wager every one of us wants from a job. We may lionize the enlightened dictator CEOs of the past, but those who attract praise for their decisive style are a perfect example of survivorship bias. Those successful autocratic leaders still need to inspire and motivate their workforce to do the work behind their decisions. If you’ve ever worked for a determined and bull-headed leader who gave orders instead of direction, then you know how painful it is to work in a culture without collaboration. 

Many people see reluctance to attend meetings or outright rejection of a meeting as an insult. They hold the attitude of our mystery pharma exec: by rejecting the meeting, you are rejecting an opportunity to collaborate. For one person, maybe that meeting would have been better as an email—you want something done, so send me the requirements/brief so I can do the thing. But another person may see the meeting as a way to feel out an idea, to get a better solution by working with an expert—you. 

Those meetings that come during points in the software development lifecycle when collaboration is most important—planning, designing, and setting scope—end up being the ones that feel the least disruptive. At these moments, meetings aren’t taking you away from other work; they are the work. Without all stakeholders coming together and determining what needs to be done, engineering orgs would be coding solutions blind. 

Finding ways to have better meetings (or skip them altogether)

But let’s put some big asterisks on “all stakeholders” and “what needs to be done.” The right attendees can make a meeting feel productive for everyone. “I have heard in our studies that there are often meetings that have a lot of participants yet require only a few,” said Professor Fritz. “In my opinion, the most important part of a meeting is to take the time and reflect on who is really necessary for a meeting and even examine whether or not I should participate, and provide an environment in which it is OK to make that decision yourself.” Everyone at the meeting should have a sense of what to do next: Only 56% of participants leave meetings knowing what actions they need to take.

If you need to call a meeting, the research suggests that you need to be a good steward of the meeting, same as you would any other project. More than half of SurveyMonkey participants say two things would make meetings better: a clear agenda and a short meeting time. All participants can improve meetings by being direct, clear, and communicating in a way that leads to a decision. While we all have different communication styles, understanding which ones work best for video calls can make those better, too. 

When meetings happen can also make a huge difference in whether people see them as valuable or not. One of the big changes that came with Shopify’s meeting-ageddon was that Wednesdays became meeting-free and Thursday had a block allocated for meetings of 50+ people. In a statement, Nejatian said, “Uninterrupted time is the most precious resource of a craftsperson, and we are giving our people a no judgment zone to subtract, reject meetings, and focus on what is most valuable.”

As the Human Factors Lab research above showed, breaks between meetings are absolutely necessary. But they also found that days without meetings improve overall collaboration, which is the whole reason we have meetings in the first place. And Professor Fritz and his students found that self-reported productivity declined if a developer had more than two meetings in a day. 

All this research is a great excuse to have fewer meetings. Meetings will still occasionally be necessary, but limiting them improves developers’ lives. Dropbox uses a framework to identify when something needs a meeting that they call the 3 Ds: decisions, debates, and discussion. These are the business actions that need people to collaborate in order to get them done. Everything else can be served in other ways. 

It’s worth thinking about how we can use other tools to take the place of those meetings we call reflexively. Status check-ins like the standup meeting have traditionally been served by actual meetings where everyone reports the status of their projects one at a time. But if companies genuinely want to embrace remote options (and preferably, asynchronous working), there are plenty of tools that will let you do that without turning on your camera. Better yet, there are tools that can automate status updates for you. 

Especially for companies with remote options, finding ways to replace the visibility of office life can make a lot of the pushes for video meetings moot. Working in public as much as possible can help share information across teams and give everyone access to the company’s domain experts. Greater transparency reduces the need for meetings, which can build trust and improve overall morale. When we spoke with the folks at 84.51° about how they use Stack Overflow for Teams, Michael Carrico, director of data science, told us, “It’s a way to help spread institutional knowledge through that entry point person and make connections. Otherwise it would have been seen as more intrusive to directly ping or email somebody.”

Meetings, especially post-pandemic as more people have gone remote, have become the de facto way we collaborate, and it’s making us all more stressed, less productive, and worse at actually collaborating. 

There is hope, but it takes thought, care, and the right tools. We have meetings for genuinely good reasons, but they take a toll on us and our organizations. You don’t have to nuke the calendars from orbit, but finding alternative ways to collaborate will win out in the end. And if you’re interested in improving our overall understanding of meetings, providing the data we need to change company cultures for the better, well Professor Fritz would like to speak to you. 

The post Are meetings making you less productive? appeared first on Stack Overflow Blog.

Of course, plenty of developers do have college degrees in relevant fields. If you’re looking at colleges in the hopes of landing a coding job, you may have to decide: Computer science or software engineering? Both are great fields to study for a career in technology, so what’s the difference?

A formal education in either subject will contain a large amount of overlap, particularly in the first half. Both fields require a solid understanding of math, logic, and basic computer programming skills and concepts. After this, the two diverge significantly.

This article will discuss the subjects that each of these majors will cover, where they overlap, and which you might want to pick depending on where you’d like your career to go. 

What is computer science?

Computer science is the study of algorithms, information, and automation. This is a domain separate from programming; computer science builds off of the theory of computation, which has deep roots in logic, mathematics, and philosophy from hundreds of years before computers existed. In fact, the first computer science departments grew out of mathematics. The founder of the first computer science department, Purdue University professor Samuel D. Conte had a PhD in mathematics. 

You can get a sense of what a university computer science degree will teach you by the questions you should be able to answer in the curriculum. What is the most efficient way to sort a list of random numbers? How can we transmit information between two people privately and how do we mathematically prove it is secure? Is there an algorithm that will sometimes return an answer, but other times continue endlessly? Much like how material science seeks to understand the fundamental properties of the things that civil engineering uses to build a bridge, computer science explores how we can organize and compute information as the foundation to writing software.

Take a look at our Computer Science Stack Exchange to get a sense of what sort of questions the field covers. Where the questions on StackOverflow.com cover the ins and outs of using programming languages and tools that build software, the computer science site is almost entirely about algorithms.  

What is software engineering?

However, knowing how to calculate things doesn’t mean you can build the operating systems and computer programs that have become ubiquitous in modern life. Software engineering is the study of how to design, build, test, and maintain software. How do you coordinate thousands of programmers to work together to build a new version of the operating system on your phone and make sure that millions of people can install the update successfully? How does a social media website organize its code so that people can use the same program in dozens of different languages? Software engineers need to understand the algorithms they use to build a product, but they focus their attention on designing and building a working product for thousands or millions of people.

In fact software engineering is less about the actual code that gets written and more about the processes one goes through to write the code. Ensuring that code is properly tested, deploying code changes to production are reliable and automated, and teams are working together with a common set of standards and practices are paramount to running a successful software project. Here at Stack Overflow, we have an Architecture Guild that regularly meets so that we can come up with standardized practices for all of engineering that ensure that our teams are working together as best as possible. As Yogi Berra once said, “In theory, there is no difference between theory and practice. In practice, there is.” Software engineering is about ensuring that coding practices get as close to theory as practically possible.

However, there is a specific difference between these degrees in certain countries like Canada (and technically in many other places). Canada has very strict laws regulating who can call themselves an engineer: to do so, you must be licensed by the local engineering board where the title will be used, similar to how there are rules around calling yourself a medical doctor or a lawyer in many countries. While this requirement initially came about because of a bridge collapse, it applies to anyone who calls themself an engineer, including software engineers. If you want to say you are a software engineer (or anything else with “engineer” in the title) in Canada, you need to be properly certified; otherwise you risk being fined.

Again, to see what the field discusses, check out the Software Engineering Stack Exchange. You’ll see a lot of questions about things like software design, bug tracking, and deploying code.

What else could I study if I want to code?

Computer science and software engineering alone aren’t enough to build a software product. Computer engineering works to design the CPUs, GPUs, and data storage devices that enable our digital worlds and none of those devices would even turn on without electrical engineering. Mathematics builds the theory and understanding of numbers that underpin the foundation of cryptography and statistics builds the tools to process and understand the information we gather. Newer fields like data science are starting to get their own degrees as artificial intelligence and advanced statistical modeling blurs the line between math, statistics, and computer science. Many other fields in science and engineering use a combination of computer science, software engineering, computer engineering, and mathematics to design planes, develop vaccines, and create animations for TV and film. In fact, many of the best programmers I know have degrees in mechanical engineering or bioinformatics and use their skills to develop robotic systems or medical drug treatments. Even medical degrees like neuroscience and audiology are now requiring programming knowledge so clinicians can compare brain images or sound recordings in scientific programming languages like R or MATLAB.

Put together, computer science research creates the building blocks that software engineering uses to design and build a working computer program. Here at Stack Overflow, software engineering practices have helped us bring together a community that serves around 500 million pages a month. Our team uses these practices to coordinate over 200 programmers to put together the website you see today. What most people might not know is that this website runs on just five servers together small enough to fit in your living room. A solid understanding of computer science principles allowed our team to efficiently store and compute all the data we keep to help people get answers to their questions. As we work to move our company fully into the cloud, we rely on software engineering principles to allow us to continue to smoothly run all our websites while we redesign our software architecture and move our code from physical servers to cloud services.

There is no right path for how to become a developer, so keep asking questions to help you find the best path for you!

The post What’s the difference between software engineering and computer science degrees? appeared first on Stack Overflow Blog.

The modern tech product company wants to be data driven—and they’ll do whatever they can to get their hands on that data. 

Compare that to the approach that produced the unicorn companies of the late nineties: your eBays and your Amazons, for example. The archetypical founding story? Some guy—admittedly it’s usually some wealthy, well-connected guy, but some guy nonetheless—thought up a thing that he wanted for himself. It turned out that the world wanted it, too, to the tune of billions of dollars. 

Why aren’t the unicorn companies of 2023 talking about how some dude in his garage wanted a thing for himself in 2013?

The answer derives from who built the internet of today, and who they built it for. 

Innovation serves unmet needs

An innovative product breaks the market for existing solutions. And this is important: it usually doesn’t just serve a need better than other solutions serve it. Instead, it serves a need that other solutions don’t serve at all. 

Let’s look at some examples.

eBay used reputational scores to connect buyers and sellers who did not know each other and have stuff actually get paid for and delivered—across time and space! Until the early internet, sales had to be colocated, and even after the internet they carried the risk of some faceless website stealing your money and never sending the thing you ordered. At the time, no mainstream solution did this.

Amazon added a logistical system that made ordering things over the internet not only reliable, but also relatively fast. No more waiting three weeks for John in Ohio to ship your CrazyTown album sticker via snail mail. To this day, for better or worse, Amazon offers a more streamlined, reliable online purchasing infrastructure for many products than even the sites of the shops that create the things they’re selling.

When Apple released the iPhone in 2007, no other phone offered the opportunity for different apps to have different input interfaces. Its introduction more or less broke the market for existing phones. 

In the days of the early internet, unserved needs were easy to find: until then, commerce had to happen either in person or via snail mail and phonecalls. Quick, self-service, asynchronous options simply did not exist. The web made it possible for them to exist. It more or less dumped a whole vat of krill into the waters of commercial demand for companies to snap up without having to be that creative.

Now, those krill have largely been eaten. The easy wins have been won. Online commerce is a saturated business. The commercial web’s early garage founders built the things they wanted, and now the things that that demographic wants have been built. 

Focus groups, surveys, segmented customer data, segmented competitor customer data, smartphone data—today’s product and marketing teams lean heavily on these resources to find the tiniest capillary opportunities for monetization within that demographic.

Innovation means, pragmatically, serving a heretofore unserved need. Those are hard to find among the people that the internet was built for. The problem lies in the fact that the product and marketing teams’ sources of data—people who do focus groups and fill out surveys from their smartphones, people who already use the product or a competitor’s product or a variety of smartphone apps—come from the people that the internet was built for. These people have few unserved needs online. 

In search of the unserved need

So where can we find the unserved needs—the opportunities to innovate? It will sound too basic to be true when I say it: we have to talk to folks that don’t enjoy an internet built for them. We can’t find them among folks already well-represented in boardrooms, product teams, and pools of active customers. Instead, visionary products derive directly from centering people at the margins of modern technology. 

That doesn’t sound like it would be a profitable strategy, does it? Our instinct on this tends to get hindered by two flawed assumptions.

Flawed Assumption #1: “Marginalized folks constitute a minority of our potential customers.”

This is just flatly incorrect in a lot of cases. Take a look at the difference in uptake by race for online money transfer companies CashApp and Venmo. These two companies do not differ in size by orders of magnitude. A Venmo employee might say of the company’s numbers “That’s just who wants/needs a money transfer app.” And they’d be, candidly, wrong.

Flawed Assumption #2: “Pursuing solutions in underserved populations will cost us demand from existing customers or larger demographics.” 

This would be true if the needs of the two groups were mutually exclusive, but they’re often complementary. My favorite example of this is the same 2007 iPhone introduction I mentioned above. The star feature of that phone—the multi-touch capacitative screen that facilitates different interfaces for different apps—was not an Apple invention. Apple got that screen by acquiring FingerWorks, a small company founded by John Elias and Wayne Westerman to distribute a computer input device Westerman had designed to help his mother, who lost much of her fine motor function, to continue to do the activities she wanted to do on her computer. Indeed, you’d be floored by the number of your favorite technological features that started as accessibility features for people with disabilities. 

Now, reams have been written about how to get marginalized people into the room on product decisions. Tech companies individually tend to experience intermittent hiccups of trying really hard on that, but as an industry as a whole, the situation ain’t great. No number of DEI trainings seems to manage to keep a diverse team. And I maintain that the reason for this is that, as not-racist, not-sexist, or not-ableist as individuals want to be, we run into two things:

1. There is so much to learn about so many axes of marginalization, that a single corporate training could not possibly completely inform and educate about all of them.
2. An equitable environment that welcomes diversity requires its members to have a complement of skills, but most training on the matter focuses on values or demographic groups rather than skills, leaving folks unequipped to uncover and examine unserved and underserved needs that technology could address.

Teams can facilitate an inclusive environment and increase their capacity to sniff out unserved needs by explicitly practicing that complement of skills in their work. Some of these get touched on in DEI trainings, but they form the foundation of creating a team where everyone experiences positive changes in their opportunity to contribute. I have written before about building an employee evaluation rubric for what I see as the critical five:

1. Moderation

This is the skill of giving folks in a meeting the opportunity to listen by protecting their opportunities to speak. Without this, teams run into all kinds of icky demographic patterns governing who gets to talk in meetings, and it’s often not the people with the most thought-out ideas. That’s why moderation is such a critical skill—especially for leadership of an innovative team. You need people’s ideas, and that requires the skill of uncovering them in groups. 

2. Soliciting opinions

There’s also skill associated with uncovering ideas individually, and identifying who to ask for those ideas. It includes thinking specifically about who will be affected by a decision, and seeking them out for input. It also includes identifying which perspectives would disagree with those making the decision, and explicitly soliciting those objections to avoid running into obstacles later on that could have been predicted. 

3. Attribution

This often feels like a small skill, but it has a huge impact. It’s common in tech for folks to misattribute ideas, forget where they heard them, or fail to recall who taught them something they know. In the aggregate, this makes folks less likely to share their ideas out of concern that they’ll lose credit for them. It’s especially true among underrepresented groups, but it can happen to anyone. A team that cares about attribution, and practices making it regularly, facilitates earlier and more open idea sharing.

4. Most advanced assumption

It’s easy to accidentally condescend to others in the workplace and give the impression that we think other people are less capable than we are. We want to share what we know, and sometimes the unintentional result of that is that we lecture someone who knows more than we do. This makes folks feel like we don’t respect their valuable expertise. And if we don’t respect it, why should they share it? 

This is another one that feels small but makes an outsize impact. It’s also tricky to broach a topic when we don’t know the level of expertise of others in the conversation. But we can learn to ask questions about that. We can also learn to share knowledge with explicit consent to be stopped if an interlocutor already knows, or if we assume too much knowledge and they have questions. 

5. Capitalizing on alternative perspectives

We’re a conflict-avoidant culture: we’ll go to great lengths to skirt disagreement. But disagreement often signals and precedes conversations that result in more complete solutions that better serve unserved needs. Developing a more positive attitude about disagreement requires tools for navigating it more comfortably and with more grace for each other. And that, like the other four items discussed above, is a skill that can be taught, learned, and practiced.

Those skills all have a lot of nuance in them, and I have not explained exactly where to get it. I’m working on an online, self-paced workshop with exercises in it to help technologists like you create and lead teams that can, in fact, innovate. You can pre-order that course right here (and, if you want proof of my teaching chops, this course on tech debt is already up and available).

But the takeaway here revolves around where innovation comes from. We can employ skills—individual, interpersonal skills—to create the kind of environment that makes innovation possible, even in a tech sector that often looks bleakly saturated. There’s hope for us, hope for our teams, and hope for our industry if we can learn to execute on our curiosity about the unmet needs we can build for. 

The post “Data driven” decisions aren’t innovative decisions appeared first on Stack Overflow Blog.

Of those who lost their jobs in the most recent round of layoffs, 45% were women—which doesn’t sound bad until you remember that less than a third of tech industry roles and less than a quarter of tech leadership roles are filled by women. Other underrepresented groups, especially Black tech workers, have also been impacted at outsize rates. And the layoffs have revealed cracks in an immigration system that hasn’t been overhauled since LISTSERV was born.

Women and people of color are more likely to have jobs perceived as expendable

Women and people of color aren’t being laid off at higher rates because we were dead-weight DEI hires in the first place. According to Sarah Kaplan, director of the University of Toronto’s Institute for Gender and the Economy, it’s because “the roles that historically underrepresented groups are hired into tend to be seen as the most expendable.”.

This includes less technical roles and ones perceived as less prestigious or farther from the product—like field and customer support, human resources, communications, and marketing. “Overall, definitely nontechnical roles are more affected, women are more affected,” Reyhan Ayas, a senior economist at Revelio Labs, told The Washington Post.

The rise of remote work during the pandemic allowed more women and people of color to enter the tech workforce because remote work made barriers like childcare and unaffordable housing within commuting distance of the office easier to overcome. At Meta, for instance, US hires for remote roles in 2022 were more likely to be people of color, while global hires were more likely to be women. But when companies make cuts, remote workers may be more likely to lose their jobs—they certainly feel more anxious about it, according to Harvard Business Review. Since companies often follow the “last in, first out” rule when determining which jobs to cut, recently hired remote workers—more likely to be women and people of color—are often the first to be laid off.

The reasons why roles seen as less technical and/or less prestigious tend to be stacked with representatives of underrepresented groups are manifold and complex, including structural barriers like historical access to education and economic resources, geographic location, social conditioning in school, and cultural and familial expectations. But the result is that industry-wide layoffs fall disproportionately on people who are already underrepresented in tech.

Our immigration system is failing H-1B visa holders

For foreign-born tech employees in the US on H1-B visas, a layoff isn’t just a job loss: it can uproot their entire lives, and their families’ lives. The H-1B visa program allows people with specialized skills who are sponsored by an employer to come to the US to live and work. H1-B visa holders can stay in the US for no longer than six years unless an employer sponsors their permanent residency (their green card). 

The number of H-1B visas awarded is capped at 85,000, and big tech companies account for a hefty percentage of these, with nearly 70% of the visas going to people in “computer-related” roles. Because only a limited number of employment-based residency applications can be granted every year, people can wait decades for a green card, tying H1-B visa holders to the same employer for years and making them especially vulnerable in the event of layoffs.

When someone on an H1-B visa is laid off, they have 60 days to secure sponsorship with another employer or leave the country. “These visa holders have built lives here for years, they have a home, and children, and personal and professional networks that extend for years,” Linda Moore, president and CEO of TechNet, told WIRED. When the companies responsible for sponsoring most H1-B visas are the same companies laying off workers, the system fails the workers (and the companies) it exists to serve.

Part of the problem is we’re using a legacy system from the 80s. The H1-B system hasn’t changed substantially in more than 35 years, writes Anna Kramer for WIRED, but in those decades, the US has become a dominant presence in science and technology, a rise fueled in large part by foreign-born talent. The immigration system hasn’t evolved along with the reality of the industry, and that creates problems—not just for individuals, but for companies and the industry as a whole.

“Tech companies have invested decades and millions of dollars into lobbying for kinder rules and an increase in the number of visas available, and in sponsoring hundreds of thousands of workers,” writes Elliott. “Yet the process remains unchanged, and layoffs mean some skilled workers that companies may want to hire from competitors either now or in future will instead leave the country.” And that’s our loss.

A problem for everyone

The erosion of diversity in tech is a problem for everyone, not just individual members of underrepresented groups. “If you don’t have a diverse workforce, you’re going to get technologies that exacerbate inequalities in our society,” Kaplan told Fast Company, referring to technologies like AI-powered facial recognition or credit score assessment. “We should care that the tech sector is not diverse, because it’s creating technologies that shape our lives.”

The post The people most affected by the tech layoffs appeared first on Stack Overflow Blog.

Coming off the heels of 2022, it may be difficult to assess where web3 technologies stand in 2023. Bitcoin rose to $47,000 and fell to $16,000. NFT trading volumes peaked at $17B in January 2022 and a year later collapsed to a mere $143M. “Blockchain” and “digital currencies” became everyday terms in the mainstream media. We saw the collapse of FTX and all its cascading consequences.

It was a tumultuous year in the world of web3—full of speculation, crashes, and scandals. But does this mean that web3 is dead and the underlying technologies made obsolete? Hardly.

Though mainstream enthusiasm for NFTs and cryptocurrency has ebbed and flowed, the community is still very much alive and actively invested in not just the technology, but in ensuring the promises of a decentralized internet are realized. The world at large is frustrated with the data collection practices of the tech industry heavyweights. The global reach of eCommerce needs trustworthy payment systems that can operate worldwide. While much of the discussion around NFT collectibles focused on high profile acquisitions and losses, NFTs themselves have only scratched the surface of what’s possible.

Web3 is here to stay

We are still in the early days of blockchain. Keep in mind that we’ve been using the term “web 2.0” since 1999 (24 years ago!) but blockchain quietly entered the market as an underpinning technology for Bitcoin in 2008 (15 years ago). That difference of nine years may sound small, but consider that nine years ago most large companies were just starting to move to the cloud.

Today, blockchain technologies power much more than basic cryptocurrency transactions. Banking and finance applications support cross-border payments that settle in seconds, not days. Multi- and cross-chain transactions via DeFi applications allow for increased crypto liquidity and improved exchanges with fiat currencies. Blockchain developers can build their own customized sidechains (more on those later) to support integration with real-time, low-cost transactions in video games and other use cases. SDKs are available in nearly every popular language, making it easy for today’s web2 developers to take their existing coding capabilities and embrace decentralized technology.

Emerging applications of blockchain and crypto include:

• Cross-border payments
• Real-time tracking of goods in supply chain and logistics 
• Electronic health record storage
• Energy supply transaction tracking, including renewable energy certificates
• Citizenship and credential tracking across borders
• Documenting legal agreements, such as real estate and carbon credits

Despite everything that’s been reported in the news about crypto and blockchain this past year, their potential is still largely untapped. Blockchain advances are bringing economic and technical utility to both users and developers. It’s truly an emerging technology with seemingly endless opportunity.

The tech behind the headlines

The technology comprising a blockchain is rather sophisticated. In the most simplistic sense, a blockchain is a database: it stores data in an ordered fashion. However, a blockchain doesn’t act as a simple database with all data on a single server, but rather as a distributed ledger: multiple computers across the world store redundant copies of all the data in the blockchain and share the work of confirming transactions, without needing a central authority or intermediary.

In a blockchain, each node has a copy of the blockchain ledger and participates in the transaction validation process. New transactions are broadcast to the network, and nodes work together to verify the transaction data and add it to the blockchain. This process is known as consensus, and it ensures that all nodes on the network agree on the state of the blockchain and that it remains secure and tamper-proof.

While some blockchains are centralized and managed by a single organization, most are open source and decentralized, meaning they are managed and maintained by a community of developers. For example, the XRP Ledger is a public, permissionless blockchain, meaning anyone on the internet can set up a validator and join the network. The reference implementation of the protocol is open source and any developer can propose amendments to this software. Because of the XRP Ledger’s decentralized nature, no singular authority can make decisions for the network. Instead, network changes are determined by a specific subset of validators, who vote on behalf of the XRP Ledger’s best interest. That being said, in order for amendments to pass, at least 80% of the validator community has to vote “yes” and that minimum threshold must be maintained for at least two weeks. If both of those conditions are met, then amendment proposals can be passed.

Consensus protocols run cryptographic functions to ensure the integrity of the network and its ledger. These usually include:

• Hash functions: Create a unique digital fingerprint of each transaction on the blockchain. They are one-way functions that take an input (e.g. a transaction) and produce a fixed-length, unique output based on that input (SHA-256 is an example of a hash function). Hash functions ensure the integrity of data because any error in transmission or other change results in a totally different hash value. If you get the same output from the hash function, you know you have the same input data.
• Public-key cryptography: Used for enabling secure communication between nodes on the network. Each node on the blockchain has a public key and a private key. The public key can be shared with anyone, while the private key is kept secret. Digital signatures are for ensuring the authenticity and integrity of transactions on the blockchain. Each transaction on the blockchain is signed using the sender’s private key, which creates a digital signature that can be verified using the sender’s public key.

Validator nodes execute the consensus protocol and can often run on commodity hardware (depending on the energy and computation requirements for the specific blockchain). Different blockchains use different consensus protocols to compute the final state of a transaction on the ledger. 

Because the XRP Ledger is open source, anyone can learn how it works, contribute to the code base, and report issues. Or they can simply write and consume apps; mint, manage and otherwise interact with NFTs; and much more.

Consensus algorithms, energy consumption, and transaction times

The two most popular consensus algorithms have long been Proof of Work (PoW) and Proof of Stake (PoS). 

In PoW algorithms, every node on the network competes to solve cryptography problems in order to validate a transaction. That’s fine for small networks of a few dozen computers, but multiply this computational cost over 100,000+ nodes and it adds up very quickly. This is compounded by the fact that the fastest nodes to validate transactions often receive financial rewards, hence a competitive arms race to deploy thousands of powerful, electricity-hungry GPUs to solve these cryptographic puzzles faster than other nodes in the network.

PoW methods are what led China to ban cryptocurrency mining altogether, the White House to issue a press release about energy concerns, and the Ethereum community to push for and switch to the more energy-efficient PoS methodology in 2022.

In PoS algorithms, instead of solving a cryptographic puzzle on every node, nodes that hold a larger stake in the network (i.e. the greater the number of tokens, the greater the stake in the blockchain) are the ones to validate transactions. They still perform a cryptographic validation process, but it’s only a fraction of the nodes on the network with the biggest stake. The algorithms are no less complex and the validation mechanisms are similar to PoW, which is why PoS transactions can also take minutes or hours to be validated.

Ethereum moved to PoS “because it is more secure, less energy-intensive, and better for implementing new scaling solutions compared to the previous proof-of-work architecture.” It was a tremendous shift in how that chain operated and resulted in more than 99.9% reduction in electricity consumption. So tremendous, in fact, that they termed it The Merge. According to CoinTelegraph, Ethereum on PoW was using 112 TWh per year and on PoS is now using 0.01 TWh per year. For reference, Bitcoin is still using tremendous energy—more than many countries on earth.

There are many alternatives to PoS and PoW algorithms, with various tradeoffs to speed, centralization, and efficiency. Chains such as the XRP Ledger and Stellar use “federated consensus” or “proof of association” algorithms where a subset of nodes collectively build and agree on the next block of transactions. Other chains, such as Ignite, use hybrid systems that combine elements of federation and PoS. These systems are far more efficient than PoW and faster than both PoW and PoS because they eschew the wasteful work of competing to solve cryptographic puzzles. For example, transactions on the XRPL take 3-5 seconds to be validated, rather than minutes or hours.

Additionally, both PoW and PoS typically let the winning validator build a block however they like—which leads to miners and validators gaming the system to get the maximum extractable value (MEV) from each block. Federated consensus algorithms are typically less susceptible to these problems because they always arrange each block of transactions in a canonical order.

Making developers’ lives easier with abstractions, dApps, and smart contracts

Web2 brought us rich application experiences, cloud computing, asynchronous communication, and plenty of centralization. It’s practically impossible to develop a web2 app without paying corporations and being subject to their privacy policies, terms and conditions, and fiduciary responsibility. Web3 gives developers the ability to write and run apps that are fully-independent, widely-available, and decentralized. No limits and no corporate dependencies.

To make this a reality, most major blockchains are working hard to attract and onboard developers to their platforms with easy-to-use SDKs and high-quality documentation (e.g. Solana, Cardano, XRPL). Open-source blockchains are widely available and provide fertile ground for innovation. Each has built-in support for financial transactions using their native tokens (e.g. SOL, ADA, XRP), ensuring that people can pay and be paid.

Many chains support the development of dApps—decentralized applications. They can be written in a variety of programming languages, depending on what the chains support. Generally speaking, the larger the developer community of a given chain, the more languages it supports. For example, Ethereum supports .NET, Go, Java, JavaScript, Python, Ruby, Rust, Dart, and Delphi. The XRPL supports Python, JavaScript/TypeScript, C++, Java, React.js and Ruby.

Some blockchain apps are backed by or written as smart contracts. Smart contracts are tamper-proof, immutable pieces of code that live on the blockchain and facilitate interactions or agreements between the app, the user, and the chain. Blockchains offer simple abstractions and SDKs so developers can get up and running quickly with app development. For example, Ethereum offers a variety of application development tools to help people experiment, build front ends, and test their dApps and smart contract implementations. The downside to smart contracts is that, since they’re immutable and shared online, if anyone finds a bug in the contract’s code, they can exploit it to their advantage, and the developer can’t easily patch the vulnerability away. This makes developing smart contracts a delicate task with higher stakes than many other projects.

The XRP Ledger supports programmability through a number of protocols and standards. It includes native transactors that provide out-of-the-box functions which are already battle-tested and standardized. The Hooks proposal would further extend programmability on the Ledger. Hooks are small, efficient pieces of code that allow for the quick and easy execution of logic before and after a transaction — all native to the Ledger. This is important because standard smart contracts can be complex and difficult to navigate, especially for developers that are new to web3.

Unlike other protocols, the XRPL also has native support for NFTs, which means developers don’t need to build or maintain a smart contract in order to bring their NFT projects to life. This lowers the barrier to entry for developers, creators, and anyone else who wants to interact with NFTs on the XRPL. Additionally, automatic royalties are enforced at the protocol level which helps ensure maximum value for creators and developers. Core operations such as minting and burning are native to the Ledger to promote ease-of-use regardless of experience level.

An upcoming amendment, XLS-30d, proposes a native Automated Market Maker (AMM) on the XRPL. The proposal will include bid and vote features, allow for simple token swaps, and should create deep liquidity between token and currency pairs. The AMM’s functionality allows application developers to create interfaces for traders and liquidity providers (LPs) and introduces a novel auction mechanism that incentivizes arbitrageurs while reducing the impact of impermanent loss faced by LPs.

Developers make the chain better—for everyone

The XRPL community is also currently testing sidechains. Sidechains allow developers to build and experiment with customized features in a sandbox-like environment—connected to, yet distinct from the mainnet—enabling innovation without disrupting or compromising the mainnet. Sidechain features could eventually be proposed as amendments and be merged into mainnet if voted on by the community. There is also ongoing development and testing of an Ethereum Virtual Machine (EVM) sidechain to bring Ethereum’s native Solidity-based smart contracts to the XRPL ecosystem.

As developers do more work on blockchains, we’ll inevitably see improvements in utility, security, scalability, cost and sustainability. The more adoption, the greater the improvements, and the greater the likelihood that more developers (and users) will further adopt this technology. The network effect and a fast-growing list of innovative features are already appealing to developers who want to move on from web2 conventions.

How developers can upskill and start building

The innovations underpinned by blockchain and advantages over web2 are getting hard to ignore. Web3 protocols are making it easier than ever to build on decentralized technologies. Web3 tech isn’t just “an upgrade” or “a step up” from web2—it’s a whole new paradigm of working on applications. They’re decentralized, permissionless, scalable, and stable. Developers can use what they already know and upskill to web3 technologies. For once, they can have skin in the game with full ownership of their assets and intellectual property. Using the programming languages they already know, they can increase their domain expertise and take advantage of decentralization. 

When choosing a chain to start on, developers should consider:

• Adoption: Do you want to build on a prime-time chain with lots of users, an up-and-coming chain with a growing user base, or get in early on something brand new?
• Ease of development: Is there sufficient documentation, fully-featured and supported SDKs, an ecosystem of existing dApps to explore, and low-friction onboarding?
• Ledger functionality and transaction time: How does consensus work? Is it efficient and quick?
• Environmental impact: Are energy consumption and sustainability priorities for the blockchain?
• Time to first dApp: How long does it take to build an app? Minutes? Hours? Weeks?
• Community: Is there a living, vibrant user and developer base? Are they passionate about the chain, its growth, and web3?

Blockchain and crypto have the power to enable a better future, and there is a vibrant community of developers that are building, testing and iterating on top of the technology to help uncover future use cases and applications. Ripple is just one contributor among many to the XRP Ledger; as members of this developer community we are deeply committed to helping it grow and thrive. 

There are a number of programs like grants and bounties to help developers of all levels get started with the funding and resources they need to bring their web3 projects and applications to life.  The XRP Ledger also recently launched an online learning portal where developers can learn more about the basics of crypto and blockchain, or dive straight into coding on the XRPL with courses in languages such as React.js (currently in beta).

For additional information or to join the community, check out the developer Discord, view open source code and repos on GitHub, and follow @RippleXDev on Twitter where we regularly share updates, projects, new features, and fixes from the XRPL community. 

The post From Web2 to Web3: How developers can upskill and build with blockchain appeared first on Stack Overflow Blog.

Is there anything more annoying to hear when you’re elbows deep in a hairy bug?

When I was working as a software engineer a few years ago, I remember putting on headphones with no music, just to keep people in the office from disturbing me when I really needed to focus. It’s frustrating, and disruptions waste a lot of time. Gloria Mark, a professor of informatics at the University of California, Irvine points out that it can take over 20 minutes to get back into a complex problem after being jolted out by a distraction.

While remote work might make it easier to carve out focus time, it depends entirely on the company. Some still expect software engineers to be ready to respond to chat or email messages at any moment, and those who are slow may suffer professionally for it. Others bring developers into so many meetings that they rarely have an hour away from video calls to get any deep work done.

So when I started my own business a couple of years ago, I decided we would do things a bit differently. I wanted to build a company that allowed me to have time for deep work, time to take walking breaks in the middle of the day, and time to pick up my son from school. Even more, I wanted to create a work environment that gave my employees the same privilege.

The working model we developed is a form of asynchronous remote work. In an asynchronous environment, all or most of the work is done at the employees’ preferred working time. Employees use asynchronous work tools to communicate, and there are typically very few real-time meetings.

Draft.dev remains almost completely asynchronous, even as our team has grown to over 12 full-time people and 300 technical writers. We don’t use real-time chat tools, and by letting people work in their own timezones, we are able to hire employees and contractors in over 50 countries around the world.

I’m not alone in embracing asynchronous work though. There is a growing trend in companies of all sizes to adopt this style of work because of the productivity, lifestyle, and hiring benefits it affords.

For this piece, I interviewed several founders and leaders of asynchronous-first companies. You’ll hear more about why they adopted this work style, what they’ve learned about making it effective, the tools they use, and some of the challenges that arise in asynchronous work environments. Whether you’re looking for a job and curious about what it might be like to work at an asynchronous company, or you’re a leader who is considering rolling this out with your team, there will be some good takeaways here.

Why asynchronous work?

Initially, my reason for making Draft.dev asynchronous was personal. My partner and I had recently had our first child and with daycares closed during Covid, I needed to be home to watch him most days. I would try to schedule sales calls during his naps or in the evenings and spend most of the day watching him, responding to emails when I had little gaps of time.

As I started hiring more people though, I realized there were other benefits to being all remote and mostly asynchronous.

Productivity time

“The original reason we went asynchronous was the timezone distribution and the need to respect each individual’s productive time,” Sameera Perera, a manager at CloudExtend told me. He pointed out that his company of 44 is located around the world and some are night owls, while others are early risers. They have a few real-time meetings per week, but 90% of their workday is done asynchronously.

My team’s experience has been similar. While we have some roles that need to be available for meetings with clients, most of our team spends their day focused on deep work like writing, editing, or coding.

I’ve also found that in some regions of the world, it’s almost impossible to work at certain times of the day. For example, rolling blackouts in South Africa often force our team members there to move their workdays based on when they have power.

Access the global talent pool

Probably the most common reason technology companies move to asynchronous work is that it gives them access to the growing global talent pool. This was certainly a factor for me when I started Draft.dev. We grew from 0 to 80 clients in just two years, so I couldn’t find enough software engineers in the United States to write the volume of content we needed.

Rishabh Kaul, Head of Marketing at Appsmith, had a similar experience as they had to hire quickly too. “After the first five employees, we decided to hire globally to access a broader talent pool,” he told me. “We have users from 140+ countries, with the top six all being from different time zones,” he said, adding that by hiring globally, they have support employees ready nearly 24/7.

Improve documentation and collaboration

A less obvious benefit to working asynchronously is having a record of almost every conversation that happens in the company. While private email or chat threads aren’t ideal for documenting decisions, there are a few purpose-built tools I’ll discuss later for building documentation or training material from past discussions.

“We end up documenting or having a trail for all discussions,” Rishabh Kaul told me, “which makes it easier for people to help themselves.”

“We encourage documentation and regular updates in a public forum like Slack or Notion,” Ronak Ganatra, Marketing Director at Lano added, saying that they “try to minimize ‘important’ projects being kicked off in 1-on-1s,” because they don’t want any details to get lost.

A common objection to asynchronous work is that collaboration is harder, but Brian Casel, CEO of ZipMessage said that he sees the opposite.

“We actually collaborate much better because we are async,” told me, pointing out that everyone gets time to think about and organize their thoughts before they speak or write. “All messages are logged,” he told me, “so it’s really easy for us to link back to something someone said.”

You might forget what someone said in a video call three months ago, but if you can go back through your message history, you can avoid losing important organizational knowledge.

Increase team autonomy

One of the benefits my team talks about most often is the high level of flexibility and autonomy they get thanks to our asynchronous environment. Employees are able to work around their lives, but also in the way they work best.

“People are happier when they have flexibility, and when they are happier they are more productive,” Kiran Shahbaz, founder of GrowWell Ventures and Goodwork told me. “Our team really values the freedom and autonomy that remote and async work provides.”

As a manger, having an asynchronous team means that I really can’t rely on metrics like hours worked. Whenever possible, I try to use output-based metrics, which encourage team members to get more efficient at their jobs rather than maintain the status quo.

How to make asynchronous work

One thing that I find challenging about asynchronous work as my company grows is that I can’t get answers to all my questions immediately. I know that’s probably a good thing (because I’m not constantly interrupting my teammates), but it’s required an adjustment to my expectations around work and forced me to get better about processes.

In addition to the mindset shift required, there are a few other things you’ll need to do to go async:

Strengthen your writing skills

“[We ask everyone to] have everything posted on public channels to increase transparency,” Rishabh Kaul told me, adding that this means they often have to “train people to emphasize the written word.”

Brian Knoles of Bellawatt’s advice was similar. “Writing is more time consuming than speaking,” he said, “but it also helps to clarify thinking in a different way. We’ve all had an ‘aha!’ moment while typing up something we were confused about.”

I’ve always thought that writing was an incredibly important skill for software developers, and Harvard neuroscientist, Juliette Han says the same is true for everyone:

“The most underrated skill that successful people, especially introverts, have is the ability to write clearly. It doesn’t matter what industry you’re in. If you are a thoughtful and strategic writer, you’ll be more confident in your interactions—in emails, public speaking or even just small talk.”

So whether your company is asynchronous or not, it’s worth investing time to become a better writer, but if you join an async company, this is doubly true.

Build processes and documentation

As writing becomes more ingrained in your daily life at an asynchronous company, it’s only natural that some of that writing will turn its way into documentation. “We document everything that is done more than once,” Kiran Shahbaz told me. “Because we don’t have the luxury of ad-hoc questions and answers, these need to be prepared beforehand.”

Of course, not all documentation is written. “We record a lot of calls,” Rishabh Kaul said, “Many meetings are recorded, especially meetings that have more than two or three people.” These recordings then serve as a library of context for people who couldn’t make the meeting and want to review it afterwards.

Finally, we’ve found that automated processes also really help our team at Draft.dev. Using tools like Zapier and Airtable, we’ve been able to automate many of our internal communications, allowing our teams to work more efficiently at their own time.

Choose the right tools

While there are a handful of tools that just about every office worker is familiar with (Zoom, Slack, Google Docs, etc.), there are some tools that work especially well for asynchronous teams.

For example, Stack Overflow for Teams offers a collaborative place for teams to ask and answer questions without interrupting each other in real-time chat platforms. Similarly, Twist is organized around threads and works more like a traditional forum than a real-time chat tool.

My team uses a daily standup tool called Status Hero to record what we’re working on each day without all having to hop on a call in real-time. Other teams I spoke to use tools like Metro Retro to capture and share their agile retrospective notes both synchronously and asynchronously.

Finally, I’ve gotten to be a big fan of leaving async video messages through ZipMessage. While real-time calls are useful for collaborating quickly, a lot of meetings could be replaced by video messages.

The challenges asynchronous work introduces

While I think a highly asynchronous environment can work for the right type of team and business, it introduces its own set of challenges. Having tools and metrics will help you, but ultimately, it’s not the right work environment for every company in every kind of business.

Trust is key

First, if employees are allowed to work when, where, and how they want, you have to be able to trust them. “Async only works with 100% reliable team members,” Kiran Shahbaz told me. Ronak Ganatra agreed, adding that everyone must be “accountable and independent.”

That last part can be tough for some managers to accept. I have worked for a few managers who didn’t believe people working from home were “really working,” and async won’t work in a low-trust environment.

For me, the key is having good work and result-tracking measures in place. For some roles, employees log and report their hours weekly, while other roles are paid on an output basis. This allows managers to compare performance objectively, even if they’re not able to meet with each employee face-to-face every day.

Hiring employees who are used to working asynchronously also helps. I look for people who have worked remotely and independently in the past. For example, a lot of freelancers tend to succeed in an asynchronous environment because they know how to motivate themselves and stay focused without a boss looming over their shoulders.

But employees have to be able to trust each other too. “Team members should remember that asynchronous means someone might not be available when you want them to be,” Anthony Eden of DNSimple told me. “Team members must accept and respect their fellow team members’ schedules and work habits.”

If you can’t trust your fellow team members or your employees, then asynchronous work is not going to be a good fit.

The mindset shift

“Regular sharing is a habit that takes time to build.” – Anthony Eden, DNSimple

Almost every new employee we hire goes through a period of time where they’re still trying to wrap their head around this asynchronous work thing. For example, we use Trello for project management, so if someone has an idea for a new project, our company’s best practice is to add it to the correct backlog and tag anyone you’d like to review it for you.

But it usually takes new employees three to four months to remember that this is how they’re supposed to do it, so I find myself stopping long email threads with a reminder to move the discussion to the proper forum.

Some people struggle to get into a consistent work routine. One of our core values is that “we work at a sustainable pace,” so I often have to remind team members not to put in extra time on the weekends or evenings unless that’s when they prefer to work. Just because work is always available doesn’t mean you have to be constantly plugged into it.

Real-time tasks and roles

Finally, some tasks and roles are simply not suited to asynchronous work. Everyone I talked to for this mentioned how challenging brainstorming in an asynchronous workplace is. Many of them use real-time meetings for that work and regular team check-ins that require more back-and-forth at a quicker pace.

Our account management and sales teams are also much more synchronous than our production team due to the nature of their work. Clients expect availability in their timezones, and responding to a client question quickly can often build a lot of trust. While we encourage these teams to set boundaries, we also want to serve our clients the best we can.

The key thing is to be honest with new hires about the amount of real-time vs. asynchronous time their role entails. Most people will understand as long as you’re open about it.

Conclusion

For technology workers, remote work is becoming the norm, but asynchronous workplaces are still relatively rare. This working style requires total buy-in across the company and a high degree of trust between employees and managers, so it may never become the standard. That said, it has advantages, especially for engineers, writers, and other knowledge workers who need to spend a lot of time in deep work.

What do you think? Have you ever worked in an asynchronous environment? Did you like it or not? Find me on Twitter to continue the conversation.

The post Building a collaborative asynchronous work environment appeared first on Stack Overflow Blog.

Raw startups are often chaotic and scattered. “Move fast and break things!” was repeated in the standups of hundreds of startup engineering orgs. Typically, startups tend to build monolithic systems to reduce friction in creating and changing features. As time goes on, the monolithic architecture begins to show strain, and almost inevitably begins to be broken up. 

Teams begin to take ownership of the new services, usually adhering to Conway’s Law. Since each service is only owned by a single team (or should be), questions inevitably arise related to the technologies used for each of them.

Should the backend administrative system, used by a dozen internal employees, be written using the same frameworks and stack as the tight, performance-critical end-user system? Should batch processors have the same technical footprint as stream processors? Should analytics databases use the same solution as the one used for ad-hoc queries?

Some engineers love to experiment with new, bleeding-edge technologies. Others warn of the perils and demand adherence to the tried-and-true. As the company grows ever bigger, the same sorts of problems emerge over and over again to be solved. Do you solve those problems the same way you did earlier because it’s already done? Or do you go with a new approach that offers more benefits?

How these problems get solved becomes more critical as a startup transitions to a medium-sized company.

Terminology

This article discusses all kinds of technology from a software engineering perspective. This can include programming languages, frameworks, database systems, ways to store and transfer files, schema definition systems, etc. I will use the word tech as a shortcut to describe this list of solutions, systems, and projects.

Option 1: The Wild West

At one extreme, we could simply allow every team to choose whatever tech they wanted to use with full autonomy. There’s no oversight committee, no red tape, and no blockers to doing what they want how they want.

The benefits

• Teams are empowered to make their own technology decisions based on the information they have.
• The “best” tech for the problem being solved can be used.
• Teams are not tied down by previous technical decisions made either by themselves or other teams.
• Morale is high since engineers get to use new, fun tech rather than being stuck with old or out-of-style choices.
• Engineers feel that their voices matter in technology decisions.

The downsides

• It is more difficult to reuse previous solutions written in other languages or frameworks.
• It is harder to generalize solutions across similar problems since reuse is harder.
• There is risk of “technology sprawl”: a single engineer has to know multiple technologies in order to succeed at their job.
• This can result in superficial broad knowledge of languages or systems rather than deep knowledge of one tech, which can affect the ability to debug problems or make more advanced changes.
• Hiring and internal training becomes more onerous. Either the company has to hire engineers who already know the full set of tech or spend more time training them once they are hired.
• Internal support becomes more fractured since there will be fewer experts in any one tech to help less experienced developers.
• Building internal tools may become more difficult since it has to work with all the tech that the team (or company) supports.

Option 2: Lock it down

At the other extreme, whatever choice was made at the company’s inception becomes the one and only solution that must be used. If the first application was made using Python and Postgres, then every service must use only Python and Postgres.

The advantages and disadvantages are swapped here.

The benefits

• Reusing previous solutions and generalizing them becomes much easier since you are guaranteed to use the same tech stack.
• Each engineer can be trained on a single stack and can more easily learn deep knowledge about it, and therefore help support other engineers.
• Hiring is easier since only one set of skills is necessary.
• Internal tools are simpler since they only have to deal with one set of tech.

The downsides

• Engineers become demoralized since they feel stuck with old tech that may not be well suited to the problem at hand.
• Trying to squeeze a tech into solving a problem it isn’t designed for can result in increasingly hacky and costly kludges.
• Engineers do not feel empowered since they have no agency to make technological decisions.

The golden mean

As you can guess, the “right path” is somewhere in the middle of these extremes. At Flipp, the term used is the Tech Toolbox, and it works like this.

• The team or company curates a list of approved tech. This list should be very small.
• The contents of the list should start with whatever the company is using at the moment.
• Each tech on the list should be given an overall status of approved, pending approval, discouraged, or not allowed.
• Further, each tech should be specific as to what use cases it is approved for. For example, Postgres might be approved for ad-hoc queries and internal tools, while MongoDB might be used for more performance-critical uses where the queries are well-known.
• All new projects must by default use tech on this list, and all other tech is not allowed.
• Approved tech are given official support by internal tools and our engineering guilds. All other tech is not supported.

Crucial to the success of this framework is that it should have three processes:

• Adding a tech to the toolbox
• Auditing and removing tech from the toolbox
• Allowing one-off exceptions to the policy

There should be a committee of senior, staff, or principal engineers who manage these processes. This sounds heavy-handed, but in reality, once the toolbox is set and disseminated properly, these processes happen very rarely.

Adding a new tech

If a team feels that none of the tech in the toolbox currently solve its problem set and that a new tech could be broadly useful across the team or company, it can petition the committee to add it to the toolbox.

This petition needs to be presented as a business case. This doesn’t mean that the team needs to do research with actual dollars attached to it. They need to be able to argue one or more of the following:

• The tech allows us to significantly reduce infrastructure cost by being more performant.
• The tech allows us to reduce engineering costs and go to market faster by making it significantly faster to develop with.
• The tech saves on testing and quality costs by reducing errors.
• The tech increases morale by providing a less stressful development environment. (This could be a broad category).

Generally, there should be some kind of data attached to this. In the latter point, it might simply be surveys of the engineers on the team.

Once the committee is satisfied, the tech is considered pending approval. The requesting team should proceed with a proof-of-concept implementation of the new tech so that any kinks or surprises can be ironed out. Once this is deemed a success, the tech can be approved in the toolbox.

If the tech fails the process, that doesn’t mean the proposal is dead in the water! Proposals can always be revisited if there is new information or use cases. And the proposing team always has the option of using it for one-offs (see below).

Auditing and removing tech

Periodically, the committee should speak to the engineers who manage the systems they own and see if a particular tech has fallen out of favor, and if so, why. Sometimes a new tech entirely supplants an older one and is deemed unwise to use since the new one is better in most respects for the use case in question.

If the committee and engineers agree, the tech in question then becomes demoted to discouraged. This means that no new projects should be built with it, and all existing projects using it should have some kind of plan to move off of it if possible.

Once as many services as possible have moved off it, the tech can be moved to not allowed. There can be legacy systems using the old tech that are grandfathered in and allowed to stay, but these are “special cases” and don’t affect the overall status of the tech in the toolbox.

One-off exceptions

No one likes working for tyrants, and the tech leadership committee is no different. There are always cases where a tech that didn’t make the cut or is marked as discouraged still needs to be used.

At Flipp, we moved Node.js from approved to discouraged for our end user-facing systems, and effectively replaced it with Go. However, we identified that there are some cases that Go is not well suited for—data munging arbitrary JSON files, for example, or using third-party SDKs that  often are better maintained in JavaScript. These projects were able to argue their case and got a pass, with the caveat that these teams were on their own for support and tooling.

The result

With a tech toolbox, we have a single central document we can point to explaining what we use at Flipp and why. We have a “paved road” to make it easier to create projects using this technology, such as an app generator, shared libraries, deployment support, and more. We have guilds that meet regularly and take on projects to improve the usage and documentation for how we use that tech within the company. And we have an explicit process to make changes to this list so engineers don’t feel disenfranchised.

As your company grows, technology decisions become more costly. Having a framework like this in place will provide a defined path forward.

The post Your tech toolbox: The middle ground between tech chaos and rigidity appeared first on Stack Overflow Blog.

SRE principles vs. SRE organizational structure

To begin with, we need to distinguish between fulfilling the SRE principles and an organizational structure for SRE. The SRE principles are:

• Operations is a software problem
• Work to minimize toil
• Automate this year’s job away
• Move fast by reducing the cost of failure
• Share ownership with developers
• Use the same tooling, regardless of function or job title

It is vitally important to understand that the SRE principles do not dictate any organizational structure. Rather, the SRE principles can be followed by teams embedded in several different organizational structures.

An SRE practice where the SRE principles are followed can succeed either with a central SRE team, without a central SRE team, or with several central SRE teams comprising an SRE organization. With this, what are the options to organize well for SRE? 

Who builds it? Who runs it?

Organizing for SRE must start with a fundamental decision: “Who builds and who runs the services?” This gives rise to several options ranging from the traditional “you build it, ops run it” to the modern “you build it, you run it.” The main options in-between are “you build it, you and SRE run it” and “you build it, SRE run it.” In Establishing SRE Foundations, these options are aligned on the so-called “who builds it, who runs it” spectrum. The spectrum is shown in the figure below. 

(Image attribution: “Establishing SRE Foundations”)

What is important to understand about the options on the spectrum are the incentives they provide for the development teams to implement reliability. With “you build it, you run it,” the incentives are maximized because developers are on-call and do not want to be woken up in the middle of the night due to reliability issues. This will prompt the developers to do everything possible to implement reliable services, though it does add yet another responsibility to developers. These incentives diminish with every other option. 

With “you build it, ops run it,” the incentives are minimal and can lead to the notorious chasm between development and operations teams. The chasm results in developers throwing their code over the wall to operations engineers. In this case, neither the code is written with operability in mind nor the operations engineers possess the knowledge to operate it. We therefore exclude this option in the considerations below. 

Other differences between the options on the “who builds it, who runs it” spectrum include knowledge synchronization between teams, incident resolution times, service handover for operations, establishment of an SRE organization, etc. 

Setting up an organizational structure for SRE

Once an organization selects an option from the “who builds it, who runs it” spectrum, they can set up an organizational structure for SRE. To do so, the following questions need to be answered:  

• Which teams are in the development organization?  
• Which teams are in the operations organization?  
• Which teams are in the SRE organization, if it is to be created?  

The cross product of  

• you build it, you run it 
• you build it, you and SRE run it 
• you build it, SRE run it 

and 

• development organization 
• operations organization 
• SRE organization 

yields nine sensible SRE Team Topologies. These are described in detail in Establishing SRE Foundations. In the next section, we provide an overview of the topologies. 

SRE Team Topologies

The SRE team topologies are embedded in the development, operations, and SRE organizations of an enterprise. To avoid ambiguity, here are the primary responsibilities of the three organizations: 

That is, a selected SRE team topology determines to a great extent the primary responsibilities of the development, operations, and, if it exists, SRE organization. Below is the list of nine SRE team topologies from Establishing SRE Foundations. 

SRE Team Topology 1: 

This is a classic “you build it, you run it” SRE team topology as followed by Amazon, for example. 

SRE Team Topology 2: 

This SRE team topology introduces a dedicated SRE role in the development team. That is, unlike the SRE team topology 1, not every developer is an SRE on rotation here. 

SRE Team Topology 3:  

This SRE team topology is a combination of the SRE team topologies 1 and 2. There is a dedicated SRE role in the team that runs the product together with another developer on rotation. 

SRE Team Topology 4 

This SRE team topology introduces a dedicated SRE team placed in the development organization. The members of the SRE team run the product in a shared on-call together with the developers from development teams. 

SRE Team Topology 5 

This SRE team topology places a dedicated SRE team into the operations organization. Like in the SRE team topology 5, the members of the SRE team run the product in a shared on-call together with the developers from the development teams. 

SRE Team Topology 6 

This SRE team topology introduces a dedicated SRE organization. The SRE team running the product together with the development teams is in the SRE organization. The SRE infrastructure team building and running the SRE infrastructure is in the SRE organization too. The shared on-call is the same as in the SRE team topologies 4 and 5. This is roughly the SRE team topology employed by Facebook with their production engineering organization. At Facebook, it is called the “centralized reporting, embedded locality” model. 

SRE Team Topology 7 

This SRE team topology places the responsibility of running the product onto a dedicated SRE team placed in the development organization. However, if the services fall below an agreed service level, the SRE team “returns the pager” to the development team until the agreed service level is reached again. 

SRE Team Topology 8 

This SRE team topology places the responsibility of running the product onto a dedicated SRE team placed in the operations organization. As in SRE team topology 7, if the services fall below an agreed service level, the SRE team “returns the pager” to the development team until the agreed service level is reached again.

SRE Team Topology 9  

This SRE team topology places the responsibility of running the product onto a dedicated SRE team placed in the SRE organization. As in SRE team topology 7, if the services fall below an agreed service level, the SRE team “returns the pager” to the development team until the agreed service level is reached again. This is the SRE team topology employed by Google. 

In addition to the differences in organizational structure, different SRE team topologies vary in other areas such as knowledge synchronization between teams and organizations, effort for service handover for operations, incident resolution times, and more. An often overlooked difference is the SRE cultural identity created by an SRE team topology. 

SRE cultural identity 

An SRE cultural identity is based on three identity dimensions: a product-centric identity, an incident-centric identity, and a reliability user experience-centric identity. A product-centric SRE identity is when SREs strongly identify themselves with the product they run. They are not just SREs, they are (for example) Microsoft Office 365 SREs taking pride in the product. This is typical when SREs are placed in the development organization. 

An incident-centric identity is when SREs are focused on having as few incidents as possible in products they run. These SREs pride themselves in metrics like only having just a few incidents a year. This is typical when SREs are placed in the operations organization. 

A reliability user experience-centric identity is when SREs are focused on achieving the user experience of reliable products for the products they run. These SREs pride themselves in having SLOs tracking the user experience well, having the SLOs fulfilled by the products they run, etc. This is typical when SREs are placed in a dedicated SRE organization. 

An SRE team topology spawns an SRE cultural identity triangle with the vertices: product-centric identity, incident-centric identity, and reliability user experience-centric identity. A particular SRE team topology will lean more towards one of the vertices on the SRE identity triangle. 

Transition to the selected SRE team topology 

Once an SRE team topology has been selected, the question of transitioning from the current setup to the selected one becomes important. If a new SRE organization gets established during the transition, it needs to be positioned within the overall product delivery organization. 

The SRE organization can be viewed as a cost center, an asset, a business partner, or a business enabler. The goal of the newly minted head of the SRE organization is to position the organization as much as possible to be the business enabler. 

Within the SRE organization, an SRE career path needs to be established to provide a proper career ladder for SRE professionals as they grow their skill and practice. A defined SRE career path also helps attract SRE talent to the company. 

Summary

SRE principles can be fulfilled by many organizational structures. In this article, nine SRE team topologies were presented, which can be widely found in the industry. A decision to choose a particular SRE team topology needs to be made taking into account the current organizational setup and culture, the envisioned target organization and SRE cultural identity, knowledge synchronization requirements between teams, and other factors. 

More details on how the decision can be made are available in the talk “Establishing SRE Foundations: Aligning The Organization On Ops Concerns Using SRE Team Topologies” from the DevOps Enterprise Summit US 2022 and the corresponding book Establishing SRE Foundations: A Step-by-Step Guide to Introducing Site Reliability Engineering in Software Delivery Organizations by the author. 

The post Who builds it and who runs it? SRE team topologies appeared first on Stack Overflow Blog.

This is scary. People have lost their livelihoods. Thousands of people in the United States on H-1B work visas, along with their families, face deportation unless they can find another job within 60 days. Diversity gains in tech have been dealt a serious blow. These layoffs have spotlighted the tenuous and unsustainable situation the US immigration system creates for foreign-born workers; the disproportionate impact of tech layoffs on women, people of color, and parents; and the still-shifting landscape of the post-pandemic economy.

Many of us have been through layoffs before, sometimes several times. My career at tech companies began in 2014, and in that time I’ve been laid off once. My colleague Ryan Donovan recently wrote about his experiences with tech startups and how to handle industry-wide layoffs, whether you recently lost your job or you’re just afraid you might.

As the discouraging headlines and meta-narrative about what the layoffs really mean continue, we thought it was worth revisiting how our core community of developers has been experiencing and coping with this ongoing reality—and exploring what sets this economic situation apart from previous dips and busts.

The post-pandemic economy isn’t what we expected

Any conversation about tech layoffs in 2023 has to account for the fact that, as The Atlantic’s Derek Thompson put it, “the post-pandemic economy has been much weirder than most people anticipated.”

In 2020, Thompson writes, people noted our rising dependence on technology like streaming video and food-delivery apps and predicted an “acceleration” of the rapidly digitalizing pandemic economy: “In this interpretation, the pandemic was a time machine, hastening the 2030s and raising tech valuations accordingly.” In response, hiring across tech jumped. By 2022, it was clear that the pandemic had produced less of a steady, sustainable acceleration and more of a…well, bubble. And we all know what bubbles tend to do.

But the current economy has less in common than you might think with the wreckage of the dot-com bubble or the Great Recession. Overall, it’s still a good time to work in tech, and the hiring market remains robust: One survey found that almost 80% of people laid off in tech found new roles within three months of launching their job search. There are more open tech positions than people to fill them (about 375,000, according to one estimate), and job listings between January and October 2022 were up 25% over the same period in 2021.

Company see, company do?

If the job market isn’t as dire as we think, why does this round of layoffs feel so widespread, affecting companies often perceived as more recession-proof than their peers? Part of the answer may be what organizational behavior experts have termed “copycat layoffs.” 

“Laying off employees turns out to be infectious,” writes Annie Lowrey in The Atlantic. “When executives see their corporate competitors letting go of workers, they seize what they see as an opportunity to reduce their workforce, rather than having no choice but to do so.” Organizations seeking to reduce risk in the face of an anticipated economic downturn may jump on the opportunity to trim costs without raising a ruckus. Companies that lay off employees while everyone else is doing it also reduce their risk of reputational damage: they’re not the only ones doing it, which suggests that layoffs are due to external economic factors, rather than company-specific shortcomings.

The jobs aren’t gone—they’ve just moved

In many cases, workers laid off by household-name tech companies have found new jobs outside the traditional parameters of the tech industry, where their skill sets are in high demand. As Matt McLarty, global field chief technology officer for MuleSoft, told CNBC, businesses that have long needed tech professionals to upgrade their stack or guide a long-delayed cloud migration can now scoop up freshly laid-off tech workers (and those for whom Silicon Valley has lost its luster). Companies in energy and climate technology, healthcare, retail, finance, agriculture, and more are hiring tech pros at a steady clip, even if FAANG companies are less bullish. It’s been said before that every company is a tech company, but in 2023, that’s truer than ever. 

In fact, the biggest difference for tech workers this year, reports The New Stack, is that “the greatest opportunities may not lie exclusively in the FAANG companies anymore, but in more traditional industries that are upgrading their legacy stacks and embracing cloud native.”

Some of those opportunities also lie with startups, including ones helmed by Big Tech veterans ready to turn their layoffs into lemonade. And efforts are underway to build the leading generative AI platform and an expanding ecosystem of related tools. “There’s a lot of investment firms that are still bullish about the startup space,” Lindsay Grenawalt, chief people officer at Cockroach Labs, which raised $278 million in Series F in late 2021, told The New Stack. 

So whether you’ve been affected by the recent spate of layoffs or not, it’s worth expanding your list of potential employers to include companies—even industries—you’ve never considered. You might find that they’re thrilled to have you.

One place to start is Indeed’s layoff support resources, offered in collaboration with Stack Overflow, Glassdoor, and Tech Up For Women. You’ll find free, automated tools to optimize your job search; paid professional services like career coaching and resume building; and articles and webinars to help you navigate things like negotiating a severance package, understanding unemployment eligibility, pivoting to a new career, and more.

We’re also teaming up with Indeed to provide a 45-minute learning and development webinar where experts at Indeed will share best practices for job hunting and professional development while answering your career questions. Register below to get your questions answered by the experts!

The post What’s different about these layoffs appeared first on Stack Overflow Blog.

We spoke with Marco Palladino, CTO and co-founder at Kong, all about APIs past and future and what engineers need to do once their API endpoint has been built.

We’ve edited this conversation for clarity. This Q&A is also available as a video. 

Ryan Donovan: How did you get involved in the world of APIs?

Marco Palladino: When we started our journey with APIs, we started with an idea in 2009 to build an API marketplace like Amazon but for APIs. We imagined a world where APIs would be the main foundation block for every application that anybody creates in the world. In 2009, that was just about to get started, so people were asking us, “What is an API?” We built our first business called Mashape, which was an API marketplace. If the world runs on APIs, then we need to have a marketplace for APIs. That product was the beginning of the Kong journey, because the Mashape marketplace didn’t work very well for us but the technology we built was very good in this new microservices and API world. We built it for ourselves and we open sourced it, so we extracted it and we pivoted into Kong as part of a transition we made in 2015. 

RD: That’s very much ahead of the game. You must be excited about the innovations in Jamstack these days. 

MP: Yeah, I mean there’s innovations that are happening pretty much across the board. Now in my space, which is the API space, what we’re looking at is APIs as fundamentally running pretty much every digital experience we can think of. 83% of the world internet traffic today runs on APIs. APIs are powering everything, as we all know, in our daily lives, in every category and every industry that we normally interact with.

RD: What makes a good API?

MP: Well we should think of APIs as user interfaces, except the user is a developer. Good APIs are easy to use, easy to understand, and not convoluted, and fundamentally they provide a nice abstraction on top of the service or the data that we want to access through the APIs. The ones that are bad are the ones don’t have any of these properties. They’re ugly, they’re hard to use, they’re inconsistent. There is no documentation whatsoever, therefore, it’s hard to consume them and hard to test them and debug them. 

Ultimately, the criteria that separates the good APIs from the bad APIs is the consumption. At the end of the day, APIs are as valuable as the consumption that we’re able to create on those APIs. And if these APIs are not being consumed, it doesn’t matter how good the service is or the data is that’s behind that API. If the API is not being consumed, that API, quite frankly, is useless. 

RD: Do you have an opinion on the various architecture styles or frameworks like the REST versus GraphQL, or even SOAP from back in the day?

MP: It’s funny to see the evolution of API protocols over time. We started with SOAP, but some in the audience may think we started earlier than that with CORBA. APIs as a concept have permeated our industry since forever. 

Now with SOAP APIs, we have the emergence of web service for the first time. SOAP APIs were notoriously hard to use, hard to consume, very verbose, and so when mobile came out in 2007-2008, everybody needed to create mobile applications. It turns out that consuming RESTful APIs is a much easier endeavor, and we can leverage most of our existing knowledge and clients to be able to do that so we don’t need to have specialized SOAP clients to consume those APIs. 

The problem is, as the number of APIs increases over time, it becomes very computationally and network expensive to make lots of requests to all the RESTful APIs that we have. So we started to see new patterns emerge, like GraphQL, which allow us to essentially get multiple responses for multiple APIs in one request and one response. That allows us to save in bandwidth which is very important, especially for mobile, and also improve the latency because we’re not sending 50 requests across all the APIs but only one request. In GraphQL, the gateway is going to be responsible for aggregating all those other responses. 

GraphQL is obviously one of those trends, but we’re seeing a lot more. Internally especially we’re seeing adoption of GRPC, where we want to use faster protocols that do not require computationally intensive serialization and deserialization in JSON. We’re seeing events being used as a way to create asynchronous microservices by propagating state changes in the data, not via a service-to-service synchronous requests, but an asynchronous event that we can store in a log collector like Kafka. We’re seeing that APIs were SOAP only for a very long time, then REST came in, and then now it is many different protocols depending on the use case and the requirements that you have.

RD: When we’re talking about a large number of APIs, we’re usually talking about microservices. How do gateways, service meshes, and other architecture-level applications help manage microservice overload?

MP: Building an API is half of the job. Once we have an API, we need to expose the API and govern how we want these APIs to be consumed, either internally or externally. There’s lots of controls that we have to build in the API infrastructure that allow us to manage access to or revoke access, monitor and capture analytics, document the API, and create an onboarding flow for the API. All of these complimentary use cases are critical for that API to be successful. Having an API sitting somewhere does not mean that API will be successful. This is very important at the edge where we want to expose our API to partners, to a developer ecosystem, to mobile applications. 

We want to have that whole product journey to the API to be very nice. APIs are products in a way, right? So we have to treat them with the same lifecycle that we treat every other product. How do we version them? How do we decommission them? How do we make them better? API gateways are great at this. API management is a function that allows us to productize an API, either externally or internally, and it allows us to create all these flows and highways to the consumption of the API. Now some of these APIs are going to be consumed internally within the applications themselves—so not across different applications, but within the application itself. There we don’t need to have this higher level management of the API, but what we need is a lower level that’s faster, lower level network management of the API, and that’s where service mesh comes in. 

With service mesh, we can reduce and remove that extra hop in the network that we would have by having a centralized ingress. We can remove that and go from service to service via a sidecar model in such a way that we make that performance much quicker because there is less networking hops we need to do, as well as it allows us for a more fine grain, lower level management of the underlying networking. This allows us to implement zero trust. It allows us to implement observability. It allows us to implement across data centers, across cloud failovers. If you experience problems in one cloud, we can automatically redirect to the other cloud. Now the reality is we need both. We need to have a service mesh to create this underlying network overlay that’s secure, that’s reliable, that’s observable, and then some of these APIs we want to expose at the edge or to another team or another application. That’s when API management comes into the picture to provide all those other capabilities. So the way I see it, these are complementary technologies.

RD: What’s the security risk with lots of APIs?

MP: Yeah, as a matter of fact, APIs are the biggest attack vector for pretty much every product that anybody is creating these days. Every product runs on top of those APIs, so APIs become a great source of problems if we do not secure them properly. Security means many things in the world of APIs. Security means securing the protocol and the underlying transport, so we want everything to have an identity and we want everything to be encrypted over a secure HTTPS connection in the case of RESTful APIs. 

We want to secure access to the API, so we want to make sure that we can create tiers of access for those APIs. We can assign clients and consumers to these tiers in such a way that we can control who consumes the APIs, but we can also then apply specific rules to a specific tier of consumers, such as, “This type of consumer can make x number of requests per second, but this other tier cannot.” There is a third level of security where we are looking at all the traffic that anybody’s making through our APIs and trying to identify patterns that are suspicious, for example, a developer trying to send random fields to an API to see if it breaks or not. Every attacker is going to be exploring and using APIs in ways that were not intended in such a way that they can find a vulnerability. Being able to detect these types of traffic patterns becomes very important to identify suspicious behavior.

RD: What’s the most work you’ve seen a single API do? (the largest number/volume of processes behind it)

MP So I’ve seen it all. There’s different types of APIs. There are APIs that are high frequency so there’s lots of value to those APIs, but fundamentally each response is not as valuable so we can afford to lose some of that traffic because it doesn’t really matter. For example, I’m sure that Twitter has lots of API requests whenever somebody wants to open a tweet or send a new tweet. It’s not a big deal if somebody cannot see a tweet; they can just retry. That is high volume but low value for each transaction. 

Then there are low volume but high value transactions, for example, when we send a tax return using one of those tax return services. We are never going to use that app and that service ever throughout the year but that one time that we’re going to be submitting our report, and that request happens once a year for each user but it’s very high value. So in my experience working with enterprise organizations and customers, Kong today is the most adopted API gateway in the world in the open-source community, but we also work with great enterprise organizations around the world that are building their API infrastructure on our technology. And I’m seeing all of these use cases so it’s very hard to pinpoint a specific one, but I’ve seen responses of gigabytes of data. So you make one request, you get gigs back, you get this huge response back. I’ve seen APIs taking days to be processed because those APIs probably should have been replaced with a job queue system. There’s pretty much everything out there. 

RD: For those high-value APIs, how do you ensure reliability without sort of duplicating effort? 

MP It’s very important to provide the right API infrastructure. This is why building an API is only half of the job. The other half is to make sure that these APIs are reliable. How do you make them reliable and secure? Well, we need to build that for every API that we have. And there is a series of things that have to happen to make sure that APIs are reliable, but first and foremost, reliability intended as security that has to be in place. Reliability intended as low latency and performance, we need to be able to trace the full stack of our requests to determine where potential bottlenecks could be located in such a way that we can fix them. And then there is reliability intended as being able to measure the API response status codes and response parameters in such a way that we can detect those types of anomalies and then act upon them. For high-value APIs that are low frequency, we’re working with customers where every 500 error is an open investigation that may take two or three weeks to be resolved, because they cannot lose any API request because it would create harm in their reputation and to the final end user. There are different levels of reliability that we want to achieve. 

Being able to also replicate our infrastructure across multiple clouds and multiple regions in such a way that we can tolerate unpredictable failures in the underlying infrastructure becomes very important. When we have lots of APIs, it’s very hard to think of these problems on an ad hoc basis for each one of these APIs and it becomes much easier to provide this reliable infrastructure for APIs to the whole organization in such a way that we can cater to everything that’s creating APIs and not just a subset of it.

RD: In the next five to ten years, how will the ways that software services talk to each other change?

MP I am speaking with customers that are telling me in the next five years they’re going to be creating more APIs in the organization than all the APIs they’ve created up until now. So what we’re going to be seeing in the next ten years is an incredible amount of scale. And scale is both exciting and frightening. Scale is exciting because it allows us to build faster and better, and this is why we’re adopting APIs. APIs allow us to turn every product and every silo into a platform. There is lots of value in that because we can build products faster on top of that, we can create ecosystems that are much more efficient, like partner ecosystem across the globe. There is lots of business value in that scale that we’re going to be creating, but there is also a requirement to have the right infrastructure in place so that that scale can be enabled in the first place. If we are not making the application teams that are building all of these APIs extremely productive whenever they ship a new API, then the application teams are going to be worrying about all these complementary concerns that they shouldn’t be worrying about. That’s not their job. So it’s very important that as we prepare for this type of scale we make sure that the application teams are builders of APIs but not builders of infrastructure. We want them to be consumers of infrastructure and builders of APIs. 

RD: Was there anything else you wanted to touch on before we sign off?

MP No, this has been a fantastic conversation. APIs are fundamentally changing and shifting the way we think of software. The way I see it, APIs are providing us the opportunity to create an assembly line of software where you pick and choose different pieces like an assembly line, and put them together to ship new applications in a better way, in a faster way. They are fundamentally changing how we are building software in the digital world. So thinking about APIs really is thinking about the future of the business, because without an API vision there is not going to be a business vision that is going to be successful, because that business vision has to rely on an API to be successful. So it’s becoming very strategic for every organization these days.

The post Building an API is half the battle: Q&A with Marco Palladino from Kong appeared first on Stack Overflow Blog.

We wanted to see how developers feel about the tech making headlines, so our latest pulse survey asked developers to think about nascent trends in technology and tell us how they felt about them. Despite many of these technologies having been around for quite some time, the conversations about them and their applications are evergreen and always evolving. With AI-assisted technologies in the news, this survey’s aim was to get a baseline for perceived utility and impact of a range of buzzworthy technologies in order to better understand the overall ecosystem. 

The survey results’ matrix below shows four quadrants of sentiment where technologies are grouped into areas of “Positive-Emergent”, “Positive-Proven”, “Negative-Proven”, “Negative-Emergent” and in the opposing “Negative-Emergent” and “Positive-Proven” quadrants is where this analysis has placed a lot of focus due to the strength of sentiment shown in this survey’s results. Open source is clearly positioned as the north star to all other technologies, lighting the way to the chosen land of future technology prosperity.

Experimental to proven: Who’s ready for prime time?

Technologies such as blockchain or AI may dominate tech media headlines, but are they truly trusted in the eyes of developers and technologists? On a scale of zero (Experimental) to 10 (Proven), the top proven technologies by mean score are open source with 6.9, cloud computing with 6.5, and machine learning with 5.9. The lowest scoring were quantum computing with 3.7, nanotechnology with 4.5, and low code/no code with 4.6. 

By grouping frequency of scores into three sentiments and looking specifically at a handful of interesting standouts elsewhere in the survey, a pattern emerges: those without strong feelings about whether a specific technology is definitely experimental or proven take a larger share of the responses for those technologies that also have larger proportions of emergent ratings.

One hypothesis for this pattern is that technologies considered experimental may also be technologies that developers have less experience with and therefore absence of strong feelings. Those neutral feelings appear to be correlated with the question asked in the survey about whether respondents agreed on which technology would never be widely used in the future: blockchain and low code/no code both received more than 10% making them the top two choices in that category. It makes sense that you may grade a technology lower on the experimental-to-proven scale if you don’t believe it will be used much in the future. 

Conversely and against expectations, those technologies that scored high proven scores were not necessarily the same that were chosen from our list of technologies as those that developers believed would be widely used in the future, however they were very close with one main exception. AI comes in at the top of the list by a large margin, but our three top proven selections (open source, machine learning, cloud computing) follow after. Technologists are willing to concede that AI isn’t a proven technology as of today but seem to be very positive about the direction it’s going.

The hero we need: the Positive Impact Score

It’s one thing to believe a technology has a prosperous future, it’s another to believe a technology deserves a prosperous future. Alongside the emergent sentiment, respondents also scored the same technologies on a zero (Negative Impact) to 10 (Positive Impact) scale for impact on the world. The top positive mean scoring technologies were open source with 7.2, sustainable technologies with 6.6 and machine learning with 6.5; the top negative mean scoring technologies were low code/no code, InnerSource, and blockchain all with 5.3. Seeing low code/no code and blockchain score so low here makes sense because both could be associated with questionable job security in certain developer careers; however it’s surprising that AI is not there with them on the negative end of the spectrum. AI-assisted technology had an above average mean score for positive impact (6.2) and the percent positive score is not that far off from those machine learning and cloud computing (28% vs. 33% or 32%). 

Possibly what we are seeing here as far as why developers would not rate AI more negatively than technologies like low code/no code or blockchain but do give it a higher emergent score is that they understand the technology better than a typical journalist or think tank analyst. AI-assisted tech is the second highest chosen technology on the list for wanting more hands-on training among respondents, just below machine learning. Developers understand the distinction between media buzz around AI replacing humans in well-paying jobs and the possibility of humans in better quality jobs when AI and machine learning technologies mature. Low code/no code for the same reason probably doesn’t deserve to be rated so low, but it’s clear that developers are not interested in learning more about it.

What can tech of the future learn from the journey of open source?

Open source software is the overall choice for most positive and most proven scores in sentiment compared to the set of technologies we polled our users about. That’s a win any way you slice it. Open source is not new, but this survey is and one has to consider that open source was not always on top. The top tags for open source for non-commercially backed technologies on Stack Overflow are Python, Java, and Ruby, while the larger network has many sites dedicated to specific open source technologies: Ask Ubuntu, Unix & Linux, Blender, Drupal, Raspberry Pi, and more. 

How did open source defy the assumed necessity of paid support in order to become regarded as the proven technology it is today? A big part of the history of open source that overlaps with the user base for our survey is the love of collaboration: open source and Stack Overflow’s public platform and Stack Overflow for Teams product don’t exist without it! 

Collaboration isn’t just an unpaid internship type of gig, it’s important at the office. In our last pulse survey about jobs, almost one in four developers say collaboration is what retains employees and there is reason to believe that spirit of collaboration comes from experience with open source; in the same survey, 27% of developers said contributing to open source at work made a job more appealing. 

Open source isn’t just about collaboration, and neither is the developer experience. Another reason to believe open source has persevered as a proven technology today is that it is a platform for learning. Proprietary software and programming languages are not going to be the first choice for educational institutions with low thresholds for cost, and definitely not for self-taught programmers completing online courses. Learning is the backbone of coding, something we can see on Stack Overflow, as well. Python, an open-source licensed programming language, has been one of the top tags for questions as of late and all of those questions being answered equate to tangible learnings. New tech talent going through traditional education paths or self-moderated online learning modules are coming into the workforce with skills in open-source tech and sometimes through open-source platforms. From our 2022 Developer Survey, Python is almost tied as the most popular language for people learning to code. Open-source removes friction in the learning process for developers, and what’s not to love about that.

Perhaps one of the biggest contributors to the growth and goodwill behind any of the technologies listed here is salary. Over half of respondents in our December survey agreed a better salary is still the largest motivator when considering a new opportunity (54%). The potential to grow one’s personal capital by learning and using open source most certainly is a large contributor to the positive and proven sentiment we see in this survey. Software engineering is projected to have the largest average growth in salary in the next 10 years by the BLS. This is likely due to the large return on the relatively low investment of time when it comes to entering the developer industry. 

One of the main takeaways from our last developer survey found that blockchain developers garnered a comparable salary to people managers but with less experience. Given the sentiment scores in this survey, it’s very interesting to watch the progression of blockchain over time. While small, question share on Stack Overflow almost tripled from 2021 to 2022 (0.05% to 0.14%) despite nearly a quarter of respondents to our Web3 pulse survey believing blockchain was all hype or a scam. Post-FTX scandal, it’s clear that most developers do not feel blockchain is positive or proven, however there is still desire to learn as more respondents want training with blockchain than cloud computing. There’s a reason to believe in the direct positive impact of a given technology when it pays the bills. 

AI-assisted technology should take notes from the way open-source technology has cultivated collaboration, learning, and take-home pay if it wants to make gains in sentiment around positive impact and be regarded as a proven technology. We can see the possibility of this already with machine learning, which overlaps and is a component of AI; machine learning is the third-highest scoring for both proven technologies and positive impact tech. Machine learning is already utilizing open-source programming languages and libraries, and the median salaries of data scientists/machine learning engineers worldwide are mid-range according to the developer survey. The trajectory of machine learning shows that there is a path for AI to become more proven and positive and the core characteristics of open source lay the foundation for AI to achieve that goal.

Organizations and business leaders everywhere should heed these lessons. Open source shows that collaboration and learning around emerging tech is key to wider adoption. If you’re part of an organization trying to implement new technology, Stack Overflow for Teams can help by encouraging the healthy cycle of knowledge sharing, collaboration, learning, and ultimately, knowledge reuse. While the topics, technologies, and sentiments discussed in this survey will shift over time, we’re confident that the role of collaboration and learning will not. We look forward to seeing more discussion and more learning take place on our platform in the years to come. 

The post After the buzz fades: What our data tells us about emerging technology sentiment appeared first on Stack Overflow Blog.

AI has become the core of everything we do at Intuit. 

A few years ago, we set out to embed AI into our development platform with the goal of accelerating development velocity and increasing individual developer satisfaction. Building AI-powered product features is a complex and time-consuming process, so we needed to simplify it to enable dev teams to do so with speed, at scale. We found success in a blended approach to product development—a marriage of the skills and expertise of data, AI, analytics, and software engineering teams—to build a platform powered by componentized AI—what we at Intuit refer to as Reusable AI Services (RAISE) and Reusable AI Native Experiences (RAIN). These allow developers to deliver new features for customers quickly and build and integrate AI into products without the typical pain points or knowledge gaps.

Today, it’s not just our customers who benefit from our AI-driven technology platform; our developers do too. Whether it’s building smart product experiences, or keeping design consistent across multiple products, our investment in a robust AI infrastructure has made it possible for technologists across the company to build AI capabilities into Intuit products at scale for our more than 100 million global consumer and small business customers.

In this article, we’ll share Intuit’s journey to democratizing AI across our organization, along with lessons learned along the way. 

Simplifying the path to integrating AI

In the beginning, when our developers wanted to add AI features to their projects, they couldn’t just plug in a library or call a service. They had to reach out to our data scientists to create or integrate a model. Most machine learning (ML) models are built on a bespoke basis because data is typically specific to a process or domain and doesn’t translate well outside of the identified scenario. While this is changing with multi-modal AI, in practice most systems still train on a specific corpus where they are expected to perform (images, text, voice, etc). 

We realized that in order to make it easier for our developers to integrate AI just as they would with any other feature or component, we had to overcome three key challenges:

1. Cross-domain communication
2. Data quality standards
3. Process improvements 

Cross-domain communication: Getting devs and data scientists on the same page (and tech stack)

Because product development teams work in different ways, aligning on an inclusive, common language when discussing how to integrate AI into the development process was key to fostering collaboration.

Software engineers and data scientists use different vocabulary in their day-to-day work. Data science terminology, for example, is very precise, especially around concepts like model performance, and can be difficult for non-experts to understand. Data teams might use terms like ROC (receiver operating characteristic curve), macro-F1, or hamming loss. Similarly, software engineers are usually focused on durability, scalability, and behaviors of distributed systems. Such technically-specific language can lose meaning in translation.

Simplifying such technical terminology—and having good documentation to explain what it means—made it much easier for developers and data scientists to communicate. Over time, developers will pick up new knowledge as their domain-specific comfort level improves. But we don’t want every developer and data scientist to have to learn a whole new set of jargon just to get started.

To address this, we adjusted the way we communicated based on context: using precise language when necessary (for accuracy purposes), and more approximate verbiage when the same message could be conveyed through more accessible terms. For example, when data scientists described data entities, we found that it was faster for engineers to understand once these were translated into rows, columns, and fields, as well as into objects and variable values. 

We also found that mapping complex topics to business-specific terminology helped get everyone on the same page. For example, translating terms like classification, regression, and propensity scores into business use cases, such as pricing predictions or likelihood to resubscribe, made the concepts more accessible. Ultimately, we found that investing in finding a common ground and devising a more inclusive approach to communication resulted in better collaboration. 

Equally pivotal to our success was bridging the worlds of software developers and data scientists by seamlessly integrating AI into existing processes. We had to find a way to support technology stacks our developers were accustomed to, so we mapped interfaces in the world of AI onto constructs they were familiar with. We built continuous integration/continuous delivery (CI/CD) pipelines, REST (Representational State Transfer) and GraphQL APIs, and data flows to build confidence in the platform’s integration across various domains.

With everyone speaking the same language and working in the same workflows, we turned our attention to the data we rely on to create AI-driven features.

Data quality: Being good stewards of data means aligning on standards of quality

As a fintech company that deals with customers’ sensitive information, we have a higher bar for data access than may be the standard in other industries. We abide by a set of data stewardship principles, starting of course with the customer’s consent to use their data. 

While technologists are eager to leverage AI/ML to deliver its benefits to customers, using it to solve the right problems in the right ways involves nuanced decision-making and expertise. While traditional API integration and state management in a distributed microservices world is already enough of a challenging task for most engineering teams to handle, AI-driven development requires a different level of complexity: identifying the optimal use cases, making sure the data is available, and capturing the right metrics and feedback. 

But at the heart of AI/ML is data, and that data needs to be good to get good results. We aligned on a process of storing and structuring data, creating feedback loops, and systematically building data quality and data governance into our platform.

Having clean data was a non-negotiable—we couldn’t allow our core data to be polluted. At the same time, speed was crucial. These two factors can sometimes come into conflict. When they did, we decided to handle things on a case-by-case basis, as it quickly became clear that a blanket policy wouldn’t work.

Once an ML model has been trained and put into production, that isn’t the end of its need for data. ML models need a feedback loop of data signals from the user to improve their predictions. We recognized that this was new territory for some of our developers, and that they needed to account for more time for the models to gather results. Once developers got used to this, feedback loops became better integrated into the process. 

However, the developers creating those loops also needed to have access to data. Most of our data scientists are used to dealing with writing big, complex SQL queries. However, you can’t expect an engineering team that wants to leverage ML in their daily work to train an algorithm to write highly complex SQL queries against a back end Hive table, as they may not have the same experience. Instead, we set up GraphQL or REST API endpoints that allowed developers to use a familiar interface.

We had a shared language, and we had an understanding of how to use data in our features. Now we needed to tackle the hardest and most time-consuming portion of feature development: development processes and the people in them. 

Process deficiencies: This meeting could have been an API

In the past, when a developer wanted to build a new feature with AI, the process went something like this:

• Developer has an idea (e.g., an AI-powered autocomplete).
• Developer speaks to the product manager to see if it’s something customers would benefit from.
• Product manager speaks to a back-end data scientist to find out if the data is available.
• Product manager speaks to front-end and back-end engineers to see if the relevant text field can be modified.
• Back-end engineer speaks to the data scientist to find out how to connect the data.
• Developer builds the feature.

We set out to streamline the process, enabling dev teams to build AI-powered features in a fraction of the time, as follows: 

1. Introduced rigorous standards for software integration, including proper syntax and semantics for describing how different types of software interact with each other.
2. Built self-serve software components and tooling to make it easy to consume and implement those standards.
3. On an ongoing basis, we’re building discovery mechanisms so that these components can be easily found and consumed.

So how does this improved process work in practice? Using the same example of an AI-powered autocomplete, we would provide the developer with a UI component that automatically takes user inputs and feeds them into our data lake via a pre-built pipeline. The developer just adds the UI component to their front-end code base, and the AI immediately starts learning what the user has typed to begin generating predictions.

Today, if an engineering team thinks a feature is valuable, data science leadership provides access to the data, algorithms, facilities to train the algorithm, and anything else they need from an AI or data perspective. No more waiting for months on a Jira request—developers can just go in, do the experiment, get the results, and find out quickly whether their feature will deliver value to the customer.

After AI integration, solving for scale

Once we managed to successfully integrate AI into our development platform, the next question was: How do we scale this across our organization? It can take several months to develop a complex ML model from end to end. When we looked at our processes, we realized that we could make improvements and optimizations that would bring that down to weeks, days, and even hours. The faster we can build models, the more experimentation we can do and the more customer benefits we can deliver. But once we began to scale, we ran into a new set of challenges.

The first challenge was reusability. As mentioned previously, a lot of AI/ML features developed today aren’t reusable because models trained on data specific to a single use case don’t tend to generalize outside of that domain. This means developers spend a lot of time rebuilding pipelines, retraining models, and even writing implementations. This slows down the experimentation process and limits what an organization can achieve. 

On top of that, since development teams don’t necessarily know what has already been built, they might end up building something that already exists. We uncovered our second challenge: duplication. By the time we had dozens of teams building data pipelines, we realized a lot of duplication was going on, and solutions that worked well for one group couldn’t scale across an entire organization.

This is how we arrived at Reusable AI Services (RAISE) and Reusable AI Native Experiences (RAIN). Software developers reuse components all the time. There’s no need to reinvent the wheel if there’s a method, class, or library that does part of what you’re trying to do. How could we apply reusability into our platform to solve for scale with AI?

Eventually, we realized the level of AI adoption and scalability we wanted was only feasible with a platform approach. We set out to identify solutions with potential for a broader set of applications, and invited teams to collaborate as a working group to develop scalable solutions. Getting the right people in the same room enabled sharing and reuse to drive innovation without duplication. We started building cross-cutting capabilities to be used across a range of different use cases for any team focused on building innovative new AI-driven products and features. 

A truly AI-driven platform: making it RAISE and RAIN

The objective was simple: create the foundational building blocks developers need to build AI into our products with speed and efficiency, while fostering cross-functional collaboration and simplifying approval processes. After addressing the roadblocks that were slowing us down—the different ways our teams spoke about their work, improving data quality, and streamlining processes—we were able to take our componentized AI services and turn them into RAISEs and RAINs that our developers could then integrate into Intuit’s end products, building smart and delightful customer experiences.  

Our new platform, with AI at its core, provides developers with a marketplace of data, algorithms, and models. We standardized the metadata that developers and data scientists contributing to every model, algorithm, and service so that they are visible and understandable through our discovery service. We even use this metadata to describe the data itself through a data map, making it easy for developers to search the platform and see if what they need is already available. The platform also picks up updates and new releases and continuously prompts the development process to ensure AI-powered features provide the best possible customer experience.  Today, AI-driven product features that used to take months can now be implemented in a matter of days or hours. 

Our journey to democratized AI has not been a fast or simple one. It has required a complete change of mindset and approach. Has it been worth it? Absolutely. Aside from the compelling customer and business benefits, our data scientists have become better software engineers and, in turn, our engineers have developed a richer understanding of the limitations and possibilities of AI and how it can make an impact.

Fundamentally, we believe that democratizing AI across an organization empowers development teams to build products that deliver outstanding customer experiences. However, the journey to reach democratized AI is not a fast or simple one: it requires a complete change of mindset and approach for most organizations.

Was it worth it? Absolutely. Without our commitment to democratized AI, it would not have been possible for our development teams to deliver smart product experiences. It removes barriers to collaboration and ultimately leads to a virtuous cycle for developers and data scientists alike that’s driving innovation with speed at scale for our consumer and small business customers. 

The post How Intuit democratizes AI development across teams through reusability appeared first on Stack Overflow Blog.

One developer got caught in a framework update. Another got stuck reorganizing the feature flags. A third needed to spelunk a long-abandoned repository to initiate the database changes. The team is underwater. Every feature release will feel like this until we get a few weeks to dig ourselves out of tech debt. We have no idea how to even get the business to consider that.

Does this sound familiar? It’s a disheartening conversation.

But we often predispose ourselves to this situation. How? We try to get businesspeople, designers, product managers, and engineers onto the same page by using the phrase “tech debt.” But that phrase puts us on completely different pages.

Ask someone in tech if they’ve heard of tech debt, and they’re likely to respond with a knowing sigh. Now ask them what it is. Do this ten times, I dare you. How many different answers do you get? Three? Four? Seven?

Everybody associates the term with a feeling—frustration, usually—but they don’t have a precise idea of where that feeling comes from. So they slap the term onto whatever happens to bother or frighten them. Designers say it means the design can’t look the way they planned it. Product folks lament how it means they lose three weeks and get no features out of the deal. Engineers? Their answers vary the most, but often they’ve got something to say about “bad code.” We’ll return to why “tech debt equals bad code” is such a scourge, but first we have to address the effect of a bunch of different people defining the same term differently in the first place.

Here’s the effect: the minute we trot out the term “tech debt,” everyone is upset but no one is listening. Each conversant assumes they know what we’re all talking about, but their individual pictures differ quite a bit. It sounds to the business like the engineers are asking for three weeks free from the obligation to release any features. They remember the last time they granted those weeks: within a month the team was underwater again. When businesspeople don’t want to grant a “tech debt week” because they saw with their own eyeballs that the last one improved the team’s capacity zero percent, how can we expect them to grant us another one with alacrity?

We, the engineers, have to examine our terminology. And we can find that terminology by dissecting what we mean when we say “tech debt.”

Tech debt is more than just bad code

Equating tech debt to bad code allows us to fall into traps. It allows us to assume that the prior developers just sucked at their jobs—which is uncharitable, but fine, until we realize that there was actually a constraint we didn’t know about. This constraint explains the loathsome characteristics of this code, and it also prevents us from doing our own genius solution. 

I once worked on a team that complained ad infinitum that customer information required a query that drew from two different tables. The team assumed that the structure remained in place because of inertia or because changing the database structure had backward compatibility implications. After spending a non-negligible amount of time bashing the database design and dreaming up ways to fix it, the team learned that their plan was…illegal. For privacy reasons in their industry, it’s illegal to store these two particular pieces of personally identifying data in the same table. Luckily, a product manager happened to mention the situation to a lawyer at the company before the engineering team got very far, or it might have been a showstopping compliance issue.

Equating tech debt to bad code also allows us to believe that if we just write good enough code, we won’t have tech debt. So we don’t spend time eliminating any. There’s no need to revisit, document, or test this code; it’s just that good. A year later, we’re back where we started. Whoops.

Equating tech debt to bad code also allows us to conflate “this code doesn’t match my personal preferences” with “this code is a problem”—which, again, is fine, until we’re under a time constraint. We spend “tech debt week” doing our pet refactors instead of actually fixing anything. Engineers love tech debt week because they get to chase down their personal bugaboos. The thing is, those bugaboos rarely intersect with the code’s most pressing maintenance challenges. So when each engineer finishes their gang-of-four-fueled refactoring bender, the code is no easier to work in than it was before: it’s just different, so no one besides the refactorer knows it as well anymore. Fantastic. A+. No notes. 

In all seriousness, this is a huge reason that spending three weeks paying down tech debt, carte blanche, often does little or nothing for the team’s velocity after those weeks have ended.

To fix these problems, choose something measurable to evaluate the quality of the system. My recommendation: maintenance load. How much time and effort are developers spending on tasks that are not adding features or removing features? We can talk to folks outside the engineering team about that number. If we have six developers but half of our work is maintenance work, then our feature plan can only assume three developers. Business people think of engineers as expensive, so this framing motivates them to help us decrease our maintenance load.

We can also track that number and determine how fast it grows over time. The faster our maintenance load grows, the more frustrations we can expect. Zero growth means that we can always maintain the system with the same proportion of our engineering team. 

Reclaiming your time

How do we minimize maintenance load growth? With good code stewardship practices. We rarely reward, recognize, or teach code stewardship the way that we do feature development skills. But code stewardship skills—documenting systems, recovering context from code, and designing for future changes—make the difference between a team that hums along for a decade or more and a team that repeatedly mires itself in declarations of code bankruptcy, rewrites, and despair. 

The Holy Grail? Negative maintenance load growth: the kind of growth that makes our code more maintainable over time instead of less. The Grail requires even more of the team than a healthy quotidian code stewardship routine. It requires us to look at individual maintenance tasks, track their origins, and address those problems at the source. These chores, backed by empirical evidence, give us something concrete to discuss in meetings about tech debt.

Are we performing lots of routine library or framework updates right now? Maybe we need to explicitly set aside time on a recurring basis to complete those. The more these pile up, the harder it becomes to debug the interactions between releases of different libraries. And the less programmers perform these, the more out of practice they remain—which makes the update rockier and more painful at the last possible second, when the update becomes mandatory.

Are we reaching into abandoned repositories and figuring out how to make a change? Maybe we need to devote effort to recapturing information about how those repositories work. It’s common for development to become much harder after some seminal team member leaves because it turns out they knew a lot of critical information that wasn’t written down or organized anywhere. I call this a context loss event, and we have no idea how maintainable a code base really is until it survives one of these. After a context loss event, developers need to proactively assess and repair the damage done to the team’s shared knowledge before unfamiliar parts of the code base become dark and scary. 

Are we constantly working around an architectural choice from the past based on assumptions about our domain that are no longer true? Maybe we need to create and prioritize a ticket for rearchitecting that. A resilient code design considers what types of changes the team expects to make in the future, and it allocates flexibility in those parts of the code. As with any task that involves predicting the future, sometimes we get it wrong. In those cases, we might need to change our design. That may require a dedicated effort.

How do we identify and prioritize chores like these? I have a whole self-paced online course about that, but even focusing on maintenance load in units of specific chores, rather than a unitary towering thundercloud of “tech debt,” gives us a better place to start addressing it.

We want feature development to feel smooth and effortless. The longer we put off maintenance work, the less smooth and effortless feature development will be. Rather than sweep all of those tasks under a rug called “tech debt” and then occasionally ask for time to deal with it as one unit, we can track what specific elements of the system force feature development to take longer, measure them in terms of the amount of developer effort that they require, and then negotiate their completion as individual tasks with attractive outcomes for developer capacity. We’re no longer framing them as an opaque and uncertain cost. We’re instead framing them as clearly circumscribed investments in our ability to produce impactful features. That conversation puts folks on the same page. It also increases the likelihood that:

• Engineers can allocate specific time to do the maintenance work
• Engineers will even be recognized for doing the maintenance work
• The maintenance work, having been selected from the real reasons that feature development slowed down, will actually improve the feature development experience for the future

And that makes the conversation about tech debt a lot less disheartening; It might even make it hopeful. 

The post Stop saying “technical debt” appeared first on Stack Overflow Blog.

“We’ve started 2023 staring down the barrel of a confluence of challenges unlike any other in our lifetimes,” said United Nations Chief António Guterres during the General Assembly in New York on February 6. “Wars grind on. The climate crisis burns on. Extreme wealth and extreme poverty rage on.”

Meanwhile, the Doomsday Clock, established by the science and security board of the Bulletin of the Atomic Scientists—an organization established by Albert Einstein and others in 1945—moved 90 seconds to midnight this past January.

Against the backdrop of these existential challenges, it’s unsurprising that “open source attracted unprecedented attention from governments and the global policy community,” wrote Mike Linksvayer, head of developer policy at GitHub, back in the Octoverse 2022 report.

No matter what’s going on around us; however, it’s up to us to remain optimistic in charing the path forward.

Naturally, a lot of people in the tech sector—and developers in particular—are thinking about the power of our work in building a better future for our world. Does our industry’s open source roots serve a purpose in debugging the issues that have led humanity to this point in the first place?

COVID-19 was—and continues to be—a learning experience

Consider the enduring challenge of COVID-19 vaccine inequity, which was a contributing factor to the rise of the Omicon variant that impacted people all over the world—not just people in poorer countries. 

The problem persists. As of January 2023, just 24.6% of people in low income countries have received at least one vaccine dose.

Scientists are talking about open source as a pathway for vaccinating people against the disease in regions lacking vaccine access, demonstrating the power of continued, persistent, and enduring collaboration. Regardless of what else was going on, the scientific community had humanity’s back from day one of the outbreak.

“While political leaders [locked] their borders, scientists have been shattering theirs, creating a global collaboration unlike any in history,” wrote Matt Apuzzo and David D. Kirkpatrick for The New York Times in April 2020.

“Never before, researchers say, have so many experts in so many countries focused simultaneously on a single topic and with such urgency.”

Within a matter of weeks from the onset of the pandemic, more than 200 clinical trials were launched. By August 2021, FDA approved the first COVID-19 vaccine, with the first vaccines beginning in December 2020. And that was only the beginning.

The power of bottom-up thinking

Mike Volpi, general partner at Index Ventures, pointed out in a 2019 article for TechCrunch that with open source projects, the community effectively acts as a QA department and product manager. The absence of hierarchy enables ingenuity to proliferate in a compounding, decentralized fashion.

“Open source software also ‘keeps up’ with change and executes on the promise of continuous improvement more effectively due to breadth and depth of contribution to design, development and defect correction,” elaborates Craig Heartwell who leads the Chief Architects team in the public sector for North America at Red Hat.

In December 2022, when The Linux Foundation Training and Certification Team announced a partnership with Rancher Government Solutions to address security and operational needs of the U.S. government and military, the stated objective was to work through long-standing issues related to cloud adoption and legacy application modernization.

“Government sector entities are facing many fundamental challenges in IT including cloud adoption, legacy application modernization and new application development followed by continuous integration/continuous delivery (CI/CD) in a DevSecOps environment,” explains Heartwell.

Open source is about many minds converging to establish a technical superbrain that can outpace a seemingly never-ending onslaught of problems.

“The world is incredibly dynamic, and technology and threats are evolving faster than we have ever seen before,” explains DARPA Director Stefanie Tompkins. “Status quo is a losing strategy.”

So what challenges are in the way?

Regulations. Compliance. Barriers to collaboration across and within institutions. 

It’s a long-standing challenge of technical innovation and policy to achieve congruence and balance with each other.

According to Code.mil, an experiment in open source at the U.S. Department of Defense (DoD), “code written by U.S. Federal government employees typically does not have copyright protections under U.S. and some international laws” due to the current regulatory picture in the country. That makes it difficult to issue open-source licenses. Governments also run into challenges motivating people internally to make open source a part of their professional responsibilities.

The good news is that there’s ongoing discussion about how to get open source right in government, with legislation opening new doors, particularly in the security industry.

With more bottom-up thinking, solutions to problems can truly come from anyone, anywhere, in spite of company layoffs, venture funding downturns, and whether or not the boss is in a good mood.

Regardless of what’s happening from an economic or commercial perspective, humanity cannot afford for knowledge sharing to slow down. There are a lot of smart thinkers, everywhere in the world, whose brilliance has a place in steering our collective future.

To learn more about how leading technology companies, of all sizes and types, use Stack Overflow to support knowledge-sharing and problem-solving at high velocities, take a look at our customer stories.

The post Why governments need open source more than ever appeared first on Stack Overflow Blog.

Edge computing brings computation power and data storage closer to the end user, resulting in faster processing and response times. The concept of edge computing is not new; it has been used for many years in manufacturing, transportation, and oil and gas industries to process data locally and reduce dependence on centralized servers. It gained more attention as some cloud providers such as Amazon Web Services (AWS), Microsoft Azure, Google Cloud and others, have introduced edge functions. As a result it is now possible for hobby developers to take advantage of edge computing using edge functions.

One of the key ways to take advantage of edge computing is through the use of edge functions. In this article, we’ll dive deep into the world of edge functions, exploring how they can be used to unlock the full potential of edge computing.

Edge functions

Edge functions are small, lightweight pieces of code that can be deployed at the edge of the network to perform specific tasks or functions—serverless functions deployed on the edge. They can be used anywhere to process data with low latency or trigger actions based on specific events or conditions. They allow for faster processing times, reduced latency, and improved reliability. Edge functions operate similar to a content delivery network (CDN), but they are able to run dynamic code rather than just static hosting.

Many cloud providers support edge functions in most programming languages, including JavaScript, TypeScript, Python, Java, Go, and PowerShell. However, the introduction of edge functions is not limited to cloud providers; many companies are developing and deploying their own edge functions to meet the specific needs of their industries. Each cloud provider has their own unique way of handling edge functions, so check out their documentation to learn how to take full advantage of your specific provider.

In the next sections, I’ll dive into some of the most common ways edge functions are being implemented in the software development and IoT industries, specifically with an emphasis on utilizing Netlify’s edge functions in JavaScript.

Serving static and localized content

Edge functions make it easy to serve up content like static files, HTML, and images using JavaScript. All you have to do is create a direct HTTP response from the edge function. In the example below, the edge function returns an HTTP text/html response for serving a static HTML page.

export default async (Request, Context) => {
	// render the page generated using JavaScript
	const html = generateHTML()

	// send our response
	return new Response(html, {
		headers: { 'content-type': 'text/html' },
	})
}

When you use edge functions, they’re deployed on the edge network, meaning the data you’re viewing comes from a server close to you. You can access the user’s approximate location from the geolocation data of the edge function. This is a great way to serve up localized content, like a translated version of your webpage, to the user. The following example uses the geolocation API to serve localized weather forecasts based on the location of the user.

export default async (request, context) => {
	// Retrieve geolocation data from context object
	const countryCode = context.geo?.country?.code || 'US'
	const countryName = context.geo?.country?.name || 'United States'

	// retrieve weather data from the API
	const weather = await getWeatherForecast(countryCode)

	return new Response(`The weather in ${countryName} is: ${weather}`, {
		headers: { 'content-type': 'text/html' },
	})
}

Serverless backend

Edge functions are great for building faster and more efficient serverless backend systems or API gateways that handle network requests. For instance, you can handle things like authentication, managing cookies, and A/B testing, which allows your main application to focus on providing a better user experience.

export default async (request, context) => {
	// Retrieve user data from request object
	const { email, password } = request.body
	// do the authentication
	await authenticateTheUser()
	return Response.json({ message: 'User authenticated', status: 'success' })
}

Many cloud providers let you use Node.js (or its successor, Deno) functions to create your serverless backend, but it’s important to note that the functionality you have access to is limited. Most providers have their own APIs for common use cases, like geolocation.

Real-time data processing and analytics

Edge computing allows you to process data in real time by a server closer to the source of the data. Edge functions are really efficient when it comes to real-time data processing because of low latency and faster processing times. This has a wide range of applications, particularly in the IoT field. For example, you can use edge functions to analyze security video footage as it’s being captured.

export default async (request, context) => {
	// Get the video stream from the request object
	const video_stream = request.body

	// Perform video processing
	const result = doSomethingWithData(video_stream)

	// Return the response
	return Response.json({
		statusCode: 200,
		body: JSON.stringify({
			result: result,
		}),
	})
}

Another great use case for edge functions is data analytics and security reinforcement. With the fetch API provided by Netlify, you can retrieve data from remote sources within the function itself, meaning you can process data and get insights in real time.

export default async (request, context) => {
	// Retrieve request data
	const source = new URL(Request.body.source)
	const data = await fetch(source)

	// Perform analysis on the data
	const result = doSomeDataAnalytics(data)

	// Return success response
	return Response.json({
		statusCode: 200,
		result: result,
	})
}

Process automation

Edge functions are a powerful tool for automating processes in the industrial and manufacturing industry as they provide faster and more reliable automation of processes. They can be used for various applications such as quality control, maintenance, and automated decision-making. For instance, you can use the below edge function to keep an eye on inventory levels and send out alerts via email when stock is running low.

export default async (request, context) => {
	// Retrieve inventory data from database
	const inventoryData = await retrieveInventoryData()

	// Loop through inventory data and check for low stock levels
	for (const item of inventoryData) {
		if (item.quantity < item.reorderThreshold) {
			// Send reorder alert email
			sendEmail(item.name, item.quantity)
		}
	}

	// Return success response
	return Response.json({
		statusCode: 200,
		body: 'Reorder alerts sent successfully',
	})
}

While these are just a few examples, the possibilities are endless when it comes to utilizing edge functions for faster and more efficient processing.

The drawbacks of using edge functions

Just like any other technologies, edge computing and edge functions have their downsides, no matter how awesome they are for fast processing. Before you start using edge functions for your next project, here are a few things to keep in mind:

• Remote data source: A centralized database increases latency, complexity, and security concerns. If you need to fetch data from the database, the distance between the edge server and the database can slow down data retrieval and increase latency. In these cases, it might be more beneficial to opt for distributed database solutions such as AWS DynamoDB Global Table and Azure Cosmos DB Global Distribution, which replicate data to multiple regions, along with edge functions. Caching data locally at the edge can help to minimize the delay reducing the number of requests to the centralized database.
• Code and time limits: Edge functions are similar to CDN networks where identical pieces of code are deployed in different locations. So, the code size is limited based on the cloud provider. They’re not ideal for larger computational models or longer requests with response times limited to a few milliseconds.
• Vendor lock-in: Cloud providers have their own way of doing things regarding edge functions. If you switch to a different provider, you’ll probably have to rewrite your code to fit their way of handling things.
• Some security concerns: As edge computing and edge functions rely on a distributed network of servers, the number of potential attack vectors increases. This makes it more challenging to secure multiple servers and protect against cyber attacks when compared to centralized systems. But the security of edge functions mainly relies on the cloud provider you select.

Wrapping up

In conclusion, edge computing and edge functions are rapidly becoming popular in the technology industry because of their ability to provide faster and more reliable processing by bringing computation power closer to the end user. Edge functions, when implemented correctly, can significantly decrease latency, provide a more efficient serverless backend, automate processes, and improve performance in a variety of industries. As technology continues to advance and demand for edge computing increases, edge functions will become even more essential for businesses and developers looking to stay competitive in the industry. With the advancements in technology and the increasing demand for edge computing, it’s the perfect time for developers and cloud providers to explore and take advantage of the benefits of edge functions.

The post How edge functions move your back end close to your front end appeared first on Stack Overflow Blog.

During my three years working on the Stack Overflow podcast and blog, I’ve gotten the chance to talk with lots of interesting folks working with a wealth of microservices and containers. I’ve seen the push to build infrastructure as code and the appeal of going serverless. At the same time, I’ve chatted with lots of folks in the areas of observability and service meshes, who have found a new business supporting the sprawl of interconnections that exists in modern applications.

Recently, however, I’ve noticed a new trend growing in parallel. Yes, adoption of public cloud continues to grow, with many companies still in the process of deciding what to migrate off local servers. Tons of folks gathereveryday to share knowledge about AWS, Azure, and Google Cloud across our Stack Overflow Collectives.

At the same time, however, a growing number of organizations are also carving out space to repatriate work from public providers to on-prem private clouds and, for the growing world of edge computing and machine learning, going back to the future of actually owning and operating their physical hardware on-site. 

A cloud to call your own

According to a 2022 report by Bessemer Ventures, there has been a significant uptick in the adoption of virtual private clouds. The report suggests that, “it is becoming easier to package SaaS products and deploy them inside a customer’s virtual private cloud (VPC). This is due in part to the standardization around Kubernetes as the operating system of the cloud. This makes it easier for SaaS companies to serve a wider range of customers that may prefer to keep certain sensitive data or applications in a VPC.” 

A lot of our customers using Stack Overflow for Teams Enterprise edition opt for this approach. We build and upgrade the platform for knowledge sharing and collaboration, but it’s installed in a private or on-prem location and the conversations about their proprietary code clients are discussing stay privates on-prem.

Tom Limoncelli, a technical product manager on our site reliability team, has strong opinions on this trend. “Here’s what’s really happening,” he wrote to me: 

a. Running your own datacenter encourages bad practices due to lack of governance and other reasons.

b. The cloud forces/encourages better practices, such as strict governance, infrastructure as code, and fully automated CD/CD pipelines.

c. People are building on-prem clouds, which emulate those best practices because they got a taste of in the cloud.

Another way to say that is… People aren’t returning to the datacenter; they’re returning to on-prem clouds because datacenters can be frustrating.

The way Limoncelli sees it, a fully DIY approach encourages bad practices. Owners tend to treat servers more like pets than cattle, adding their own customization. After decades of that, you get a datacenter that is just one big mess of bad configuration ideas, mismatched technologies, and political barriers that prevent any of that from being fixed. The smart way to move to private cloud is to be strict about standardization. In other words, no, you can’t request a special machine with a weird ethernet connection because one person thinks it’s cool.

Cloud systems, on the other hand, provide an API to request new virtual machines in minutes, instead of a manual purchase process that took months. They install racks and racks of the same hardware configuration. They use standardized machine configs, guiding users away from bespoke configs. Costs are accounted for, which often doesn’t happen in datacenters. On the software side, the move to the cloud is an opportunity to adopt automation like a CI/CD pipeline, weaning people off of manual deployments.

Users acquire resources via an API, not by a purchase order. Governance and automation are established from the start. The combination of accessing resources by a standardized API and automatically enforced governance results in a system that is more maintainable and enforces more modern practices. 

Containers will set you free

As Nick Chase argues over at The New Stack, Kubernetes has been a powerful enabler for companies looking to gain more control over their usage of the cloud. It’s relatively agnostic about the real or virtual hardware you choose to employ because it can do a wealth of different things with the Linux kernel as its foundation. 

Kubernetes was designed to make it simpler for a small group of people to manage a large constellation of applications by abstracting the underlying hardware. Barring a setup that confines your system to a scarce resource, it offers a level of resilience that customers ten years ago turned to public cloud providers for. Along with allowing users to update systems without going offline, it also offers capabilities for monitoring of services up and downstream, something that many microservice heavy organizations are now turning to third-party observability providers for.

“Kubernetes’ superpowers change the game in important ways,” writes Chase. “If you can deploy, scale and manage the lifecycle of Kubernetes, you can use it to pave over public and private cloud infrastructures, optimize costs and overheads aggressively, and treat everything underneath Kubernetes as a commodity.”

As my colleague Ryan Donovan pointed out during a recent conversation, “being able to abstract infrastructure has enabled a lot of cloud providers, but it’s also allowed folks to have those containers located anywhere — within a public cloud, on a prem server in Lithuania, or a private cloud replicated across multiple locations.” Just because your infrastructure has moved to the cloud doesn’t mean you don’t care about proximity to users and the advantages that might provide you in terms of cost or latency.

Stack Overflow has taken advantage of some of these superpowers. Max Horstmann, formerly a Staff Software Engineer at Stack Overflow, now a principal software engineer on the Azure Kubernetes Service (AKS), wrote in depth about why Kubernetes might be a good choice and how we took advantage of it inside our organization. You read his article on it or listen to his podcast below. 

“If you’re starting a new project from scratch — a new app, service, or website — your main concern usually isn’t how to operate it at web scale with high availability,” writes Horstmann. “Hence, when it comes to choosing the right set of technologies, Kubernetes — commonly associated with large, distributed systems — might not be on your radar right now. After all, it comes with a significant amount of overhead.”

Despite all this, he sees value in adopting it from the start. “When you’re launching something new, your focus is typically to move fast and iterate quickly based on early feedback. Scaling is something for later. K8S is a tool that, in my view, allows you to do just that because it can accelerate your build/test/deploy loop, allows you to easily deploy and instrument different instances of your app, e.g. for split testing, customer demos etc.”

If you’re lucky enough to find product market fit and start to see a surge in customer demand, Kubernetes proves valuable in this area as well. “The problems that come with scale — fault tolerance, load balancing, traffic shaping — are already handled,” says Horstmann. “At no point will you hit that moment of being overwhelmed with success; you future-proofed your app without too much extra effort.” 

This comment from a HashiCorp’s forum sums up the advantages well: “A Kubernetes cluster is a good example of an abstraction over compute resources: there are many hosted and self-managed implementations of it on different platforms, all of which offer a common API and common set of capabilities.”

A bridge between public and private clouds

The Bessemer report cites another emerging technology trend that pairs increased cloud adoption with on-prem data. “Emerging middleware platforms are making it easier to bring the power of the cloud to the data, wherever it may be. This has played out in industries like financial services, where a wave of modern fintech infrastructure helped build bridges between the cloud and legacy banking systems. We are seeing similar bridges being built in other large industries like supply chain, logistics, and healthcare to bring the power of the cloud to these on-premise data sources.”

It’s important to define what we mean by “middleware” here. As Red Hat points out, the term dates back to a 1968 NATO conference on software engineering, where it referred to code that sat between the assembler/compiler at the bottom of the pyramid and the application logic at the top. In the world of hybrid cloud, middleware refers to an evolved version of this same idea. As Asanka Abeysiinghe, Chief Tech Evangelist at WSO2 explains in a blog, this can look like, “mega clouds that provide infrastructure as a service (IaaS)-enabled middleware capabilities via APIs, which have become the new DLLs. So, for example, message queues, storage and security policies are open for developers to consume in applications running on the IaaS (Infrastructure-as-a-Service).”

Outside the big public cloud providers, Abeysinghe sees other alternatives catching on. “Kubernetes addresses the issue of cloud lock-in by bringing an open standard to the cloud-native world, and it enables basic middleware capabilities as components. In addition, the Cloud Native Computing Foundation (CNCF) brings a rich set of Kubernetes-centric middleware, and you can find them in the CNCF technology landscape. However, if the middleware capabilities provided by Kubernetes and the CNCF are not enough for your application development, you can add custom resources by defining them in a custom resource definition (CRD) because Kubernetes is built using open standards.”

When I spoke with Abeysinghe for this article, he was quick to point out that there was no data to indicate a trend of companies moving fully away from the cloud, far from it. There are still more folks migrating onto the public cloud than off it. He estimates that 80 percent of activity is still focused on the traditional shift from local to public cloud, with another 20 percent moving in the opposite direction. But that 20 percent is important, precisely because it flows against the prevailing tide we’ve seen over the last decade. 

Abeysinghe believes that there is a realization, especially at organizations with a lot of legacy hardware infrastructure, that they now have a lot of machinery sitting idle. If you’re a big bank with decades of mainframes at your disposal, utilizing only five percent of that doesn’t make much sense. “Kubernetes lets you run a private cloud that better utilizes your existing on-prem hardware.” Cloud bursting technology lets you shift to third party resources when your local hardware is maxing out. 

Not to be left out of the game, public cloud providers now offer physical server racks to clients who have jobs that are more efficient on-prem, or need to remain in-house for security and compliance reasons. Companies that once helped to migrate companies off local hardware now offer server-racks-as-a-service bundled with your public cloud offering, a truly full circle moment for the evolution of compute.

Bringing AI models in-house

One area where this trend seems particularly strong is among companies focused on artificial intelligence that work with large data sets and have created their own models. “Big cloud GPU compute is very expensive, whether it’s for training or for inference,” says Dylan Fox, founder and CEO at Assembly AI, a startup that provides AI-as-a-service to companies that are seeking natural language capabilities in their offerings but don’t want to build the models or hire a team in-house.

“We do most of our training in on-prem instances. We have a couple hundred A100 NVIDIA cards, and we recently just purchased like a couple hundred more that we have for on-prem instances used to train.” The crypto winter has been a blessing for this market, as a glut of GPUs has come onto the secondary market and prices for new and used hardware have fallen.

As David Linthcium wrote over at InfoWorld: 

Companies are looking at other, more cost-effective options, including managed service providers and co-location providers (colos), or even moving those systems to the old server room down the hall. This last group is returning to “owned platforms” largely for two reasons.

First, the cost of traditional compute and storage equipment has fallen a great deal in the past five years or so. If you’ve never used anything but cloud-based systems, let me explain. We used to go into rooms called datacenters where we could physically touch our computing equipment — equipment that we had to purchase outright before we could use it. I’m only half kidding.

When it comes down to renting versus buying, many are finding that traditional approaches, including the burden of maintaining your own hardware and software, are actually much cheaper than the ever-increasing cloud bills.

Second, many are experiencing some latency with cloud. The slowdowns happen because most enterprises consume cloud-based systems over the open internet, and the multi-tenancy model means that you’re sharing processors and storage systems with many others at the same time. Occasional latency can translate into many thousands of dollars of lost revenue a year, depending on what you’re doing with your specific cloud-based AI/ML system in the cloud.

It’s not just small AI startups that want to crunch a lot of data at a low latency with homegrown models. Here’s an eye-opening quote from Protocol. “The on-prem trend is growing among big box and grocery retailers that need to feed product, distribution, and store-specific data into large machine learning models for inventory predictions, said Vijay Raghavendra, chief technology officer at SymphonyAI, which works with grocery chain Albertsons.” 

Raghavendra left Walmart in 2020 after seven years with the company in senior engineering and merchant technology roles. “This happened after my time at Walmart. They went from having everything on-prem, to everything in the cloud when I was there. And now I think there’s more of an equilibrium where they are now investing again in their hybrid infrastructure — on-prem infrastructure combined with the cloud,” Raghavendra told Protocol. “If you have the capability, it may make sense to stand up your own [co-location data center] and run those workloads in your own colo, because the costs of running it in the cloud does get quite expensive at certain scale.”

Chick-fil-A had a similar experience. In a blog written by Brian Chambers, the company’s head of Enterprise Architecture, he noted that, “In researching tools and components for the platform, we quickly discovered existing offerings were targeted towards cloud or data center deployments. Components were not designed to operate in resource constrained environments, without dependable internet connections, or to scale to thousands of active Kubernetes clusters. Even commercial tools that worked at scale did not have licensing models that worked beyond a few hundred clusters. As a result, we decided to build and host many of the components ourselves.”

Their solution allowed a DevOps Team and Smart Device Support to deploy, build, and update to thousands of restaurants. 

Cloud, with control

Total spending on cloud computing is already enormous and still projected to grow over 20% this year, closing in on a half a trillion dollars. But it will be a far more varied and nuanced period of growth. “Cloud is the powerhouse that drives today’s digital organizations,” said Sid Nag, research vice president at Gartner. “CIOs are beyond the era of irrational exuberance of procuring cloud services and are being thoughtful in their choice of public cloud providers to drive specific, desired business and technology outcomes in their digital transformation journey.”

After a decade or more spent moving away from server racks, companies are finding there can be advantages to running local infrastructure for certain kinds of compute. There is also, perhaps, a generational shift at work. The engineers who cut their teeth building big public clouds inside large tech companies see now moving on to create startups or take senior roles at smaller companies that specialize in a subset of cloud offerings. What’s old is new again, but with a vast variety of new flavors and permutations to choose from. 

The post Are clouds having their on-prem moment? appeared first on Stack Overflow Blog.

ADHD diagnosis rates are on the rise for both adults and kids, though as you might expect it’s tough to know whether this rise is attributable to a higher incidence of ADHD or simply an increase in the number of diagnoses made. Either way, more people are understanding their experiences and abilities through the lens of ADHD, and this includes many people who code. But is there really a connection between programming and ADHD? And could it be that people with ADHD are particularly well-suited to programming careers? 

A perfect fit?

Many developers with ADHD feel their job is a perfect fit for how they think and approach problems. “Coding can give ADHD brains exactly the kind of stimulation they crave,” explains full-stack developer Abbey Perini. “Not only is coding a creative endeavor that involves constantly learning new things, but also once one problem is solved, there’s always a brand new one to try.”

In addition to a revolving door of fresh challenges that can keep people with ADHD engaged, coding can reward and encourage a state of hyperfocus: a frequently cited symptom of ADHD that developer Neil Peterson calls “a state of laser-like concentration in which distractions and even a sense of passing time seem to fade away.” It’s easy to draw parallels between hyperfocus and the flow state, a distraction-free groove in which programmers, writers, musicians, artists, and other creators produce their best work (occasionally while forgetting to eat). Our paid platform, Stack Overflow for Teams, is popular with developers in large part because it helps them avoid distraction and protect the productive sanctity of their flow state.

But for every quality that makes coding perfect for people with ADHD (or vice versa), there’s another that could represent a particular hurdle. For instance, ADHD can make people more vulnerable to inattentive mistakes, missed deadlines, or unfinished projects. A perennial question on Reddit is some variation of “Programmers with ADHD, how do you stay on track?”

Combating stigma with candor

The reality is that while some forms of neurodivergence might lend themselves to certain careers (I’ve always thought that my obsessive-compulsive disorder makes me a better copyeditor, for example), individual results will continue to vary.

But it’s good news that many programmers with ADHD seem emboldened to share their experiences, offer advice, and ask for support and accommodation when they need it. And as we discussed on a recent podcast episode, more developers (and their managers) are having conversations about how to support the success of neurodiverse team members.

An open dialogue about ADHD and other forms of neurodiversity is a crucial step in dismantling the remaining stigma around neurodiversity in tech. These conversations happen best in psychologically safe environments—something for managers to take to heart, especially in a time when many of us are already feeling increased pressure thanks to industry-wide layoffs. 

Making work and hiring more accessible to neurodiverse people benefits everyone in the organization. Neurodiverse people can contribute unique problem-solving approaches, an affinity for hard skills like data analysis, and a tendency toward perfectionism that can elevate overall quality, says Mariann Lowery, Product/UX Research Lead at Stack Overflow. And making the workplace more inclusive can increase employee engagement and give us a greater sense of purpose at work—whether we identify as neurodiverse or not. 

The post Developer with ADHD? You’re not alone. appeared first on Stack Overflow Blog.

From everything I’ve seen these last few years, few software organizations have achieved this level of monitoring. Most teams I’ve encountered have told me they do not have the monitoring coverage they would like across the surface area of their app. I’ve seen many startups go surprisingly far with almost no monitoring at all. To those who still struggle with the monitoring basics: you are in good company.

Today, it’s easier than ever for a team to monitor software in production. There are more monitoring and observability tools available than ever before. We’re also seeing more collective understanding about monitoring and observability best practice across the industry. So why is there such a gap between monitoring ideals and monitoring reality?

What’s happening is that teams are falling into monitoring debt more quickly than they are able to pay it back. In this article, I’ll talk about what monitoring debt is, why it’s easier than ever for teams to build towards monitoring bankruptcy, and what there is to do about it.

What is monitoring debt?

Most software engineers are familiar with the concept of technical debt, a metaphor for understanding how technical tradeoffs have long-term consequences. People typically talk about tech debt in terms of how the cost of refactoring, redesigning, or rewriting tomorrow allows a team to ship faster today. Tech debt, like financial debt, can be taken judiciously and paid off responsibly.

In some ways, monitoring debt is analogous to tech debt: teams can choose to underinvest in monitoring at the time of shipping code at the cost of having to go back and invest in monitoring later. From a technical perspective, monitoring debt behaves similarly to tech debt. It costs more to clean up later and requires intimate knowledge of a system that a developer may have context-switched out of. And this is assuming the same developer is around to pay back the debt!

The costs of monitoring debt are even more insidious. When a team chooses to ship code without thorough monitoring, here are the the immediate costs:

• The team needs to accept limited ability to catch issues ahead of customers, meaning the customer is the monitoring plan. This may work for some tools, but may wear on the patience of paying customers.
• The team has chosen to give up at least partial ability to quickly localize issues when they arise, meaning they are less likely to be able to fix issues quickly. This means that customers could be waiting for up to hours or days for their issues to get resolved.

What’s worse, paying back monitoring debt is often harder than paying off technical debt, since it requires both intimate knowledge of the code (what kind of behavior is normal; what are the highest-priority events to monitor for), as well as facility with the monitoring tools (how to find and fix the issues of interests given what the tools support?). 

Often, the reasons teams decide to take on monitoring debt—not enough expertise; not enough time—cause them to fall deeper and deeper into debt.

Why it’s easier than ever to build towards monitoring bankruptcy

Today, there are a few reasons it’s easier than ever for teams to quickly and  build towards monitoring bankruptcy.

Monitoring is a highly-skilled activity

Monitoring a system requires a nontrivial amount of knowledge about both the system under monitoring and how that system should be monitored using the tools.

• Setting up monitoring and observability requires knowledge of the underlying system. If you’re using something like Datadog APM, it can look like all you need to do is include a library, BUT this may often involve updating other dependencies. Even in our ~1-year-old code base with three microservices, it took an extremely senior engineer a week to hunt down the dependencies across multiple languages. And even after we set it up, though, my developers don’t have the bandwidth to set up all the dashboards we need to properly use this data. We are perpetually behind!
• The tools themselves have a learning curve. Many tools require some understanding of how to use the tools: how to instrument; how to customize the graphs. Using OpenTelemetry outside a framework that provides automatic instrumentation support has a decently high learning curve because you have to learn how to implement spans. Other observability tools advocate writing code to anticipate that you will consume the logs or traces, which requires understanding and discipline on the part of the developer. Tools that require custom dashboards often require developers to understand how to access the data they need and which thresholds mean something is wrong. Like manual transmission cars, most monitoring and observability tools today trade ease of use for control and flexibility; these tools require some facility and basic understanding of both clear monitoring goals and the underlying monitoring mechanisms.

Monitoring is best done fresh

The longer a piece of software goes without monitoring, the exponentially harder it gets to monitor. First, hooking up the monitoring tool is harder for a system that is not completely up-to-date and paged-in. Any monitoring system that requires the use of a library means that there are likely compatibility issues—in the very least, somebody needs to go around updating libraries. More high-powered tools that require code changes are even harder to use. It’s already hard to go back to old code to make any updates. Context-switching it back in for tracing is just as tricky! 

Second, consuming “old” monitoring data is tricky. Even if you can rely on your framework to automatically generate logs or add instrumentation, what is considered “normal behavior” for a system may have gotten paged out or left the company with past team members.

Finally, with software teams being more junior than ever before and experiencing more churn than in recent history, the chances are increasing that a different, more junior developer may be tasked with cleaning up the debt. These developers are going to take longer just to understand the codebase and its needs. Expecting them to simultaneously pick up skills in monitoring and observability, while retrofitting logging and tracing statements onto a code base, is a big ask.

Better tools have made it easier to take on monitoring debt

Finally, the rise of SaaS and APIs have made it a lot harder to monitor systems. Monitoring is now no longer about seeing what your own system is doing, but how your system is interacting with a variety of other systems, from third-party payment APIs to data infrastructure. I would also say that a legacy subsystem nobody on the team completely understands also falls in this category. While traditional monitoring and observability practices made sense for monoliths and distributed services completely under one organization’s control, it is unclear how to adapt these practices when your distributed system has components not under your team’s control.

What teams need to pay off monitoring debt

My take: let’s get new tools. But in the meantime, let’s also rethink our practices.

Today’s monitoring tools are built for a world in which the developers who built well-contained systems of manageable size can get high-fidelity logging across the entire thing. We live instead in a world where software services run wild with emergent behaviors and software engineering is more like archaeology or biology. Monitoring tools need to reflect this change.

To meet software development where it is, monitoring tools need debt forgiveness.  My proposed improvements:

• Make it easier to set up monitoring and observability black-box. I know, I know: the common wisdom about finding and fixing issues is that you want to understand the inner workings of the underlying system as well as possible. But what if there’s too much code in the underlying system to make this possible? Or if there are parts of the system that are too old, or too precariously held together, to dive in and add some new logs to? Let’s make it easier for people to walk into a system and be able to monitor it, without needing to touch code or even update libraries. Especially to make it possible to set up new monitoring on old code, we want drop-in solutions that require no code changes and no SDKs. And with more and more of system interactions becoming visible across network APIs and other well-defined interfaces, blackbox monitoring is getting closer to reality.
• Make it easier for teams to identify what’s wrong without full knowledge of what’s right. Today’s monitoring tools are built for people who know what they’re doing to do exactly what they need to do to fix what is wrong. Teams should not need to understand what their latency thresholds need to be, or what error rates need to be, in order to start understanding how their system is working together. The accessible monitoring and observability tools of the future should help software teams bridge knowledge gaps here.

Across monitoring and observability, we have great power tools—but what don’t we need in a “for dummies” solution? This is something we’ll need to think about collectively across the industry, but here are some starting ideas:

• We’re not building for teams that are optimizing peak performance. They’re trying to make sure that when load is high, the system does not fall over. Answering the basic questions of “is anything errors?” and “is anything too slow?” often does not require precise machinery.
• Being able to precisely trace requests and responses across the system is great, but it’s usually not necessary. For a point of reference: a friend once told me that less than five Principal Engineers at his FAANG-like company used their tracing tools.
• What is the minimum information we need for monitoring? When root causing issues, is it simply enough to get a unique identifier on which process the issue came from? I would love to see more discussion around “minimum viable information” when it comes to finding and fixing issues.

Talking more about the answers to these questions can help establish minimum, rather than ideal, standards for monitoring using the existing tools.

Bringing down the debt

In order to help teams pay off monitoring debt, we need a mindset shift in developer tools. We need more tools that are usable in the face of tech debt, monitoring debt, and very little understanding of the underlying system or tools.
Today, it would be a heroic feat for a new junior person to join a team and successfully address an incident. But we’re already seeing accounts of this in the news—and tech workplace dynamics mean we should only expect more of this to happen. Why not make it easier for a junior dev to handle any incident?

The post Monitoring debt builds up faster than software teams can pay it off appeared first on Stack Overflow Blog.

For applications aiming for high availability without the hassle of managing infrastructure issues, serverless computing has emerged as the next level of hardware abstraction. 

Cloud computing and containers virtualized the physical servers that web and mobile apps ran on, but you still needed to provision those resources and manage how they scaled. While serverless doesn’t eliminate the server (obviously) it does mean that you don’t have to think about how your infrastructure will have to scale to support your application’s fluctuating traffic, allowing for faster development cycles by lessening the management burden. 

Serverless technology is most commonly implemented via serverless functions, or functions as a service (FaaS), which are used to run business logic in stateless containers. But if you’re building an application that needs to persist data, you’ll need to connect your serverless functions to a database. 

The benefits that come from serverless computing can be lost if you have to spend your time provisioning hardware or worrying about capacity planning and management as your application scales. In fact, traditional databases, (even fully-managed DBaaS), are generally not well-suited for managing the frequent yet disposable requests that come from serverless functions.

Fortunately, there are some databases that can handle the sort of workload that a serverless application produces, as they themselves are built to operate in a serverless manner. In this article, we’ll dive into the perils of working with a database that isn’t serverless and just how much difference the right database makes for running serverless applications. 

What is serverless?

With all the buzz about serverless, it’s a good idea to step back and make sure that we’re talking about the same thing. Serverless is an umbrella term. At its core, it’s a computing model where you don’t need to think about the underlying infrastructure: you don’t provision resources, you don’t manage scaling, and you don’t worry about high availability. All that happens automatically. 

With serverless, you only pay for resources that you use while your requests are being handled. When demand increases during periods of high traffic, the resources available scale accordingly. When traffic drops, you no longer pay for those resources. Alternatively, a pre-provisioned cloud-based service can run up your bill sitting idle and listening for requests. You can still encounter high bills with serverless computing, but only because you’ve had unexpectedly high usage. Serverless has a lot of benefits for applications with variable or sparse workloads. Think about a betting company that allows live wagers during the World Cup, as an example. During a match, bets may come in at any time, but you can assume that the number of new bets will spike when one team scores. Unfortunately you can’t predict when a goal is going to be scored. With serverless, the infrastructure behind the betting application would instantly auto-scale to support the traffic. In the absence of auto-scaling, you would need to manually monitor your resource consumption to determine when it was necessary to scale up — assuming you have the available resources to do so. 

Serverless databases essentially work the same way. When you send them data, the databases automatically scale to handle increased connections and storage requirements. You pay for the read/write traffic and the storage that the data occupies. That’s it. 

The challenges with traditional RDBMS and serverless computing

When you mix a serverless compute platform with a traditional relational database (RDBMS), you’re likely to run into some issues. The most obvious (although not limited to relational databases alone) are around provisioning infrastructure resources. While you’re saving time by using serverless computing, you still need to worry about infrastructure for your database. Even if the database is hosted on a managed platform, you still need to provision those resources — the “managed” part refers to the infrastructure that it’s hosted on, not necessarily the resources available. Databases will grow over time as new data is stored, so your storage resources, in addition to compute, also need to be able to grow with it. 

Another common challenge is the risk of overwhelming the database with too many connections. In a standard database interaction, a service or application opens a connection to a database, maintains it to read and write data, then destroys the connection when finished. Serverless functions may spin up a new connection for every request, creating dry connections: open connections that aren’t sending or receiving data, but add to the number of total open connections and can potentially block new connections. Choosing a serverless database eliminates this challenge entirely and will simply scale (up or down) in accordance with your application needs without any work required on your part. 

You can pool connections to improve performance and avoid running dry connections, but you still risk read/write conflicts if multiple connections are accessing the same data. A database built for serverless typically has pooling or some form of load balancer built in to help mitigate this problem. 

Document-model databases work better with serverless 

The biggest drawback to selecting a relational database for your serverless architecture is the structural rigidity of the database schema. In an RDBMS, each record stored must conform to the structure that the table implements without exception. All data must be written to a table, and every table has a set number of columns. If you have data that has values that don’t fit into any of those columns, you’ll need to add a new column. Each piece of data already entered into the table will require an update to include a value for that column and indexes will need to be refactored. 

The cost of schema changes makes it harder to iterate on and evolve your database structure alongside your application logic. You lose out on one of the core values of serverless —– speed to market.

Instead, opting for a document model database deployed on the cloud, like MongoDB Atlas lets you store data as it comes to you. Whereas relational databases impose strict data structures, document model databases define the structure of each document separately, giving you a great deal of flexibility while you develop.

For fast moving products, change happens often. If you’re locked into a specific data schema, that limits the kinds of changes that you can make.  Document databases allow any new document structure you want to apply to exist alongside all previous structures. If you need to adjust existing records, you can backfill data or modify the structure as a separate process. This makes it easy to iterate on and evolve your data model as you develop. 

Achieving serverless-style scalability from a database

Beyond just the benefits of the data model, MongoDB Atlas also removes much of the pain associated with provisioning and scaling infrastructure. Our fully managed data platform makes it easy for you to get the database resources you need when you need them, with automatic, elastic scaling to take the burden off of development teams. This is especially important when working with serverless compute services (or any serverless architecture for that matter) when you need your database to be reactive to unexpected spikes in traffic. 

This type of scaling can be achieved with an Atlas dedicated cluster with compute and storage auto-scaling enabled, which will give you more control over setting minimum and maximum scaling thresholds, or you can opt to deploy a serverless database. A serverless database in Atlas, referred to as a serverless instance, is an on-demand endpoint that scales automatically to meet your application needs without any upfront provisioning or capacity management required. They are based on an operations-based pricing model that charges only for the resources and storage used and will scale down to zero if there is no traffic—giving you the full benefits of building on top of serverless infrastructure.  

However, as with serverless computing, a sudden spike in traffic can sometimes lead to a surprisingly large bill. To minimize this sticker shock, we’ve implemented a tiered pricing system for read operations in which each tier is progressively less expensive. You still pay for what you use, but you’ll pay less for that traffic the more traffic you get.

Getting started with a serverless database in MongoDB Atlas

Because all infrastructure is provisioned and all scaling is handled, getting started with a serverless database is a matter of naming the database instance and pointing it to your cloud provider of choice (available in regions on AWS, Google Cloud and Azure). Seriously:

All you need to do is pick your provider, region, and desired backup option, then give your instance a name you’ll use to refer to it in your application. That instance name is probably the most important part of this setup; you’ll need to include it in the database connection string that opens a connection to a database, so make it memorable and manageable. 

In contrast, creating a database on a dedicated server requires you to select the infrastructure requirements and a host of other configuration options that have been abstracted in serverless, helping to save you time. 

“Thanks to Atlas serverless instances we are spending less time wrestling with infrastructure and configuration and more time developing solutions for our customers.” – CEO, Small Business Computer Services Company. 

Build full-stack serverless applications with MongoDB Atlas + Google Cloud Run

If you’re looking to get started with a serverless compute and database stack, we’ve found good results running MongoDB Atlas with Google Cloud Run. It’s a fully-managed serverless solution that performs seamlessly with our database instances.  

Cloud Run’s automatic scalability, combined with MongoDB Atlas’s fully managed and highly available database, allows for a seamless and cost-effective way to handle incoming traffic. As Cloud Run automatically spins up additional containers to handle the load, your serverless database will also provide more resources to handle the database load. This ensures that your application is always able to handle incoming traffic, even during periods of high and unexpected usage, without you having to worry about managing the underlying infrastructure. So you can focus on developing and deploying your application, instead of worrying about maintaining servers or databases.Both Cloud Run and MongoDB Atlas are pay-per-use services, which means that you only pay for the resources you actually use. Depending on your use case, this may help you save on cost as you are not paying for resources that you don’t need.

Finally, Cloud Run’s stateless and event-driven nature and MongoDB Atlas’s automatically scaling and highly available database make them a great fit for a wide variety of use cases, from simple web applications to complex microservices architectures. Developers can take full advantage of serverless technology across the entire stack, making their development process more streamlined and cost-effective.

Move fast and don’t worry about infrastructure

For individuals or organizations looking to get a cloud-based application up and running quickly without having to worry about provisioning and scaling infrastructure, serverless has emerged as a solid option. But you could miss out on the full benefits of serverless if part of your stack — your databases — aren’t optimized for flexibility and scale. 

If you’re looking to improve your efficiency and take advantage of the added benefits of a serverless database, sign-up for MongoDB Atlas on Google Cloud Marketplace to try Atlas serverless instances today. After signing up and completing the setup wizard, you’ll be ready to start storing data within minutes. If you’re not quite ready for that, you can also visit our website to learn more about how to build serverless applications with MongoDB Atlas to make sure you’re prepared when your next project arises. 

The post Serverless scales well, but most databases don’t appeared first on Stack Overflow Blog.

In the 1980s, personal computers became common in the workplace. With the release of the Apple II in 1977 and the IBM PC four years later, a tool once reserved for scientists and the military proliferated in accounting offices, universities, and hospitals. You no longer had to be an engineer to use a computer.

Something similar has happened with programming over the past decade. In everyday offices, classrooms, and laboratories, code is now being written and maintained by people who don’t think of themselves as programmers.

The good news is that it’s easier than ever to write code that works. There’s no shortage of beginner-friendly programming tutorials, and high-level languages today are more readable than ever. So, if someone at work hands you a Python script to run over some data, between Stack Overflow and Codecademy you can probably figure out how to get it to work.

The bad news is that after you write the code you just learned to write, someone is going to have to read it (that someone might be you in six months). To a beginner, getting a working solution is the finish line. To a developer, it’s only the starting line. And since more of us are writing and collaborating on code than ever before, it’s becoming crucial to write better code.

There’s a lot to learn about basic programming beyond “making it work.” For professional developers, writing maintainable code is fundamental to the job. But these practices aren’t usually discussed in “Coding 101”. Beginner tutorials usually focus on syntax.

So we’ll assume you already know how to pull together a working solution — it’s time to  level up to Coding 102: writing code with and for other humans.

Prerequisites

We’ll keep our examples short and simple and write them in pseudocode (which should seem familiar if you’ve written in a popular language). This post is aimed at people who’ve spent time writing code, but don’t write it professionally full-time. It helps if you are familiar with variables, functions, and if/else statements.

Why writing better code matters

There’s an adage in programming that maintaining code is 75% of its total cost. Once you write the code, other people (including future you) will invest two to three times as much time, on average, in reading, updating, and fixing it. This doesn’t mean your code was badly written or buggy! Requirements evolve, other parts of the code change, and sometimes… well, yeah, sometimes there are bugs.

At one of my first (non-programming) jobs, I wrote a script that switched between browser tabs with user accounts and pasted in discount codes. At the time, this task was done by hand for hundreds of users each month.

I hacked the script together in a few hours, and, to my manager’s delight, it pasted hundreds of codes in a few minutes. But the following week, the script went haywire and applied the code to 300 wrong accounts. It took me two hours to code the script, but eight hours to undo the damage it caused and fix my convoluted code. That’s the maintenance cost.

Good design “reduces the cost of future changes.” Let’s say my company changed the layout of the discount code page. How long would it take someone to update my script to work with the new layout? If another developer offered to help, how long would it take them to get up to speed on my work? That depends on how well the code was designed.

In order to make code easy to maintain, update, and fix, it should be modular, easy to follow, and easy to reason about. Writing “clean” code (as it’s often called) will help you think through the problem. If other people you work with can embrace these practices, you’ll spend less time struggling and more time collaborating productively.

Coding 102

Now that we understand why maintainable, clean code matters, let’s talk about how to write it.

1. Write small functions

Aim to write small functions that do one thing. Let’s say you’re writing a function that takes a list of widget prices and returns some text with the average of those prices. For example, you give it [13, 28, 17, 143, 184, 72.3] and it should return “The average price for 6 widgets is $72.22.”

A first attempt at this code might be a function called get_price_average(data) that:

1. Takes some data
2. Gets the length of the number list
3. Gets the sum of the prices
4. Divides the sum by the number of widgets to get the average
5. Rounds the average
6. Assembles all the stats into a string of text and prints it

Depending on the language, some of these operations might be built in. You put these steps in one function, and it works. This is a fine, typical first version.

On the plus side, the code is cohesive: everything we’re doing related to the stats text is in one place. So if you need to change anything having to do with this text, you know where to look.

The problem is that if you do need to make a change (add a stat, clarify which numbers should be averaged, or change the decimal points on the average), you’ll need to read and understand the whole function. And adding the change will just balloon it further.

The solution is to break our oversized get_price_average(), which does many things, into several small functions that each do one thing. A good guideline is:

> A function should be small, do just one thing, and ideally shouldn’t impact anything outside the function

Let’s try to see this by example. Here’s how we might break the function up.

1. get_widget_price_stats(data): we’re renaming our wrapper function, get_price_average(). It now takes some data, extracts the relevant stat and returns texts with that stat. We’ll talk about names shortly!
2. get_max(list_of_numbers): gets the largest number in a list. Depending on the language you use, this might be part of the standard library.
3. get_min(list_of_numbers): gets the smallest number in a list. Depending on the language you use, this might be part of the standard library.
4. get_sum(list_of_numbers): gets the sum in a list of numbers.
5. get_average(list_of_numbers, decimal_points): takes a list of numbers and returns the average, rounded to decimal_points.

Did you find some of those function descriptions obvious? For example, it might be pretty clear that get_max(list_of_numbers) gets the largest number in a list. Good! That’s by design. When you write small functions that have clear single responsibilities, you can give them names that clearly convey what they do. Then, your future readers don’t have to read all the code in the function to understand what it does.

How does this design respond to changes? Let’s say instead of the average, you need to return the standard deviation. In the initial version, we’d need to find the section of the code that calculates the average and overwrite it with a standard deviation calculation.

In the small-function version, we could create a new function, get_standard_deviation(list), then go into get_widget_price_stats() and replace the call to get_average() to get_standard_deviation().

To recap, the functions we break out should be small, self-contained, and do one thing. This makes it much easier to read and change your code. This practice alone will improve your code significantly.

2. Pick names carefully

There’s a running joke that “there are two hard things in computer science: cache invalidation and naming things.” We’re going to leave cache invalidation for another post and focus on naming things.

A related idea is that “good code is self-documenting.” In essence, this means that good code conveys what it does. Names are a big part of this: a good name is precise, easy to understand, and gives the code context without having to read every line.

Here are some good naming practices:

1. Describe intent: Try to describe the variable or function you’re naming as specifically as you can. For example, the function we looked at above was called get_price_average. However, since the string we’re returning is actually an average of widget prices, we could name the function more precisely: get_widget_price_average(). Later, we wanted it to serve as a wrapper function that returned some stat, so we renamed it to get_widget_price_stats() From this function, we extracted a general average function that takes a list of any numbers and returns the average.
   
   Try to avoid general names like date, slice, part, numbers. Instead, depending on context, date might become created_at, slice might become outdated_prices, part might become part_to_be_fixed, and numbers might become current_row.
   
2. Be consistent: Whichever convention you decide to use, stick with it. If the codebase uses get_ for functions that return values, each function should be get_[stat]: get_average, get_max, etc. If you name a function that returns standard deviation standard_deviation, another dev will wonder how it’s different from all the get_[stat] functions.
   
3. Give the right amount of information. A variable called first_name conveys a good amount of information. The alternative, name is too vague (first? last? middle? full?) and n or fn is downright indecipherable. These are examples of under-explaining; it’s not clear what the variable does.
   
   It’s also possible to over-explain: user_first_name doesn’t offer any additional information, since user is fairly generic, and it’s not clear what the alternative is (does the app have bots with first names?)  Similarly, first_name_for_user_record, or get_price_average_by_dividing_prices_by_quantity offer redundant information.
   
   Deciding on the right level of specificity is a skill you’ll build over time. It requires you step back and think deeply about what the variable is doing. This’ll make your code better in the long run and make you a better developer with it.

Speaking of how you shouldn’t call your variable fn, here’s a list of naming practices to avoid:

1. Don’t include type: Focus on the purpose of the variable, not its implementation. Common names that break this rule are array, number/num, list, word, string. Syntax is usually clear from the code, so you don’t need to add function or the type to the variable name: get_average_function or array_of_prices. As a general rule, your name shouldn’t include the type of the variable (this is called “Hungarian notation”). If you need to say admission_numbers, keep looking for a better name (daily_admissions.)
2. Don’t use abbreviations: Is det “detail”, “detrimental”, “deterministic”, “Detroit”, or something else? Our earlier fn falls into this category.
3. Don’t use one-letter names: These are abbreviations on steroids—they’re completely incomprehensible! Instead, describe what the variable’s purpose is.
   
   There are a few uncommon exceptions to this:
   1. Loops, where it’s common to refer to the index as i. 
   2. Scientific and mathematical conventions, as long as you’re certain they’ll be clear to other developers.
   3. Domain-specific conventions (for example, in JavaScript, webpage events are often referred to as e instead of event).

These are exceptions—99% of your variables should have descriptive, multi-letter names!

3. Avoid reassigning variables

The following problematic function takes some input, transforms it with seasonal_multiplier, adds a mystical 14, and returns result.

function get_result(list, seasonal_multiplier) {
	int result = list;
	result = list.sum / list.length;
	result = result * seasonal_multiplier;
	result = result + 14;
	
	return result;
}

Putting aside the opaque variable names, result represents four different things during the course of this function. It starts out as the input, then becomes input per participant, then input per participant adjusted for the seasonal multiplier, and is then finally returned with a magical 14 tacked on. It’s doing too much work!

Most languages allow you to assign a new value to a new variable. In practice, this is usually not a great idea. A variable should represent one thing, and change its identity during its lifespan. When you look at a variable name, you should have a good sense for what its purpose is.

There are two ways to clear up this code:

1. Be clear about what the changes are. A better way is to be explicit about what each transformation is, declaring a new variable at each step. This might seem like over-explaining at first, but it makes the function much easier to comprehend.

function get_result(list, seasonal_multiplier) {
	output_per_participant = list.sum / list.length;
	seasonal_output_per_participant = output_per_participant * seasonal_multiplier;
	adjusted_seasonal_output_per_participant = seasonal_output_per_participant + 14;
	
	return adjusted_seasonal_output_per_participant;
}

As a final step, I’d probably rename the function get_adjusted_seasonal_output_per_participant().

2. Combine the math into one line. You can avoid this problem altogether by combining all the steps into one calculation. The reader then has to follow a lot of math at once, but at least the intent of the variable is clear, and one variable has one meaning throughout the function

function list, seasonal_multiplier(input) {
	adjusted_seasonal_output_per_participant = list.sum / list.length *  seasonal_multiplier + 14;
	return adjusted_seasonal_output_per_participant;
}

You might also wonder what that 14 represents. That’s what we call a magic number.

4. Avoid magic numbers

A number used without any context is called a “magic number,” and it makes the code harder to comprehend. Here’s a bit of incomprehensible code:

int yearly_total = monthly_average * 12 + 67.3;

monthly_average * 12 probably refers to the number of months in a year (we deduce from the better named yearly_total).  But we shouldn’t have to deduce. And what the heck is 67.3? It might have made perfect sense to the code’s author, but we’re left to scratch our heads.

A better approach is to assign numbers to well-named variables, then use those in calculations.

int total_at_beginning_of_year = 67.3;
int months_per_year = 12;
int yearly_total = monthly_average * months_per_year + total_at_beginning_of_year;

5. Use few parameters

Functions take inputs (referred to as “parameters” and “arguments”). In most languages, the order of the inputs matters. Let’s say you have a function that takes a fruit, a country, and a year, and tells you the average price for this fruit in that country and year.

function getAverageFruitPrice(fruit, country, year) {
    // get and return the price
}

This seems good so far. To get a price, you can call get_average_fruit_price(“peach”, “Albania”, 2007). But what if we want to narrow the price further by adding an option to only count fruit sold industrially (for example, to juice companies)?

No problem! We can just add a new boolean parameter, function get_average_fruit_price(fruit, country, year, sold_to_industry). But remember, we’ll need to pass in the arguments in the right order, and that’s an easy thing to mess up. We’ll need to check the function definition every time we want to call it! If you mess up and write get_average_fruit_price(“peach”, 2007, “Albania”, false), year is now “Albania”!

A good general rule is to keep your input count to two to three at the most. This way, developers building on top of your code don’t have to constantly check for the correct orders, and are less likely to make mistakes.

There are two ways to improve a function that takes too many inputs:

1. Break it up into smaller functions. Too many parameters could be an indicator that your function is doing too much.
2. Use an options object. If you can pass the parameters in an object (called a hash or a dictionary in some languages), you don’t have to worry about property order. This object is often called options and referred to an “options object.” The downside to this approach is that it’s less explicit about the expected parameters. Here’s how this looks in practice:

function get_average_fruit_price(options) {
    // use options.fruit, options.country, options.year, options.sold_to_industry
}

6. Write good comments

There are a lot of great, thorough rules around code comments, but we’ll focus on the low-hanging fruit…

Use comments sparingly to explain code that’s not otherwise clear. If you can improve the code instead, do that! Here’s an example of a bad comment covering up for unclear code:

 int f = 75; // f is the temperature in Fahrenheit

Let the code speak for itself by renaming f to temp_in_farenheit!

You won’t always have time to improve the hard-to-comprehend code you see. You might also write code you’re not thrilled with due to time constraints. Consider whether the code will need to be “touched” (read or modified) by someone else. Be honest — you don’t want to pass on code that’s hard to maintain!

If the code isn’t likely to be modified by anyone else, you can leave a comment explaining what a particularly challenging bit of code does. Ideally, your function and variable names should already do this. But when the functions contain logic that’s hard to comprehend and the function name doesn’t help, leave a comment!

7. Don’t repeat yourself

The DRY principle stands for “don’t repeat yourself.” If you find yourself using the same code in multiple places, see if you can pull it out into a variable or a function.

string company_description = “Fruit Trucking & Shipping International ships fruit on our modern fleet of trucks and ships, anywhere in the world!”
string company_slogan = “Fruit Trucking & Shipping International: where the best fruits get on the best ships”

string company_name = “Fruit Trucking & Shipping International”
string company_description = company_name + “ ships fruit on our modern fleet of trucks and ships, anywhere in the world!”
string company_slogan = company_name + “: where the best fruit gets on the best ships”

If the company ever changes its name (for example, if it decides to only ship apples…) we can now do this in one place. Conversely, if you don’t follow DRY, you’ll need to find all the places you have duplicated code and remember to make changes in each of them.

Here’s another example: let’s say all your input temperatures are in Farenheit, but you need your output values to be in Celsius. Everytime you work with a temperature, you’ll need to F - 32 * 5/9. If you find out your organization now uses Kelvin , you’ll have to update this to F - 32 * 5/9 + 273.15 every place you reference a temperature.

A better solution is to pull this conversion into a function:

function fahrenheit_to_celsius(temp_in_farenheit) {
   return temp_in_farenheit - 32 * 5/9;
}

Now to switch to Kelvin, you can update the calculation in one place. In this particular example, you could probably just create a new (small, single-purpose) function called fahrenheit_to_kelvin(), and all the calls to fahrenheit_to_celsius() to fahrenheit_to_kelvin(). Now it’s very clear where you’re updating (you don’t have to look for mentions of “temperature” all throughout your code.)

It’s possible to overdo DRY. You don’t want to over-optimize too early, so a good alternative to DRY is WET, or “write everything twice”. If you need to write some code a third time, it’s probably a good idea to try and pull this out into its own function or variable. Over time, you’ll develop judgment for when to reuse the same piece of code over and over (like the example above) and when a few repeated lines are a part of separate processes.

Next steps

Reading about good coding practices is like reading about working out. You have to follow the practices to reap the benefits!

Just because you learned about all the muscles doesn’t mean you need to exercise them all right away. Start small: the next time you code, try to be precise with your names. Then, try to write smaller functions. Then, reference this article to see what else you might have missed!  Think of it like starting your fitness regimen with a few exercises a week.

Remember, we’re writing code for our future colleagues, and, importantly, our future selves. We don’t need to be dogmatic about the rules; we just need to focus on being as clear as we can.

You’ll quickly find that writing clean code involves making tradeoffs. For example, if you’re under time pressure writing a one-off script to parse some data and are confident nobody will see the script again, you can invest less time into writing clear code. But over time, thinking about code maintainability and readability will empower you to solve problems more effectively and will set you apart from many beginner developers..

By implementing some basic software dev principles, you’re leveling up from someone hacking through code to someone maintaining and building software. And that’s something to be proud of!

The post Coding 102: Writing code other people can read appeared first on Stack Overflow Blog.

This lack of security understanding has implications for software development on three layers: the individual developer writing insecure code, the engineering team blindly trusting their dependencies, and the organization thinking that their best bet is to roll their own security controls. In this article, I’ll talk about each of these layers, how a lack of security knowledge affects it, and how individuals and organizations can create software that follows security best practices. 

Hello (insecure) world

From the very first lesson, we are taught to code insecurely and incorrectly. The first thing we do in the famous ‘hello world’ lesson is to put data on the screen: “Hello world!” The second thing is we ask the user for their name: “What is your name?” The user enters their name, we take that data and reflect it back to the screen to say “Hello <insert user’s text here>”. There is no mention of validating the user input to see if it’s potentially malicious, nor is the learner taught to output encode the information, which would disable any potential code the user has entered. 

If we added more to the lesson, validating the input (and rejecting it if it was bad) and then output encoding the data before mirroring it to the screen, then we would be teaching every new coder how to avoid cross-site scripting (XSS). But we don’t teach that as part of the lesson. We show them how to do it insecurely. I suspect this is part of the reason that XSS is so prevalent across the internet: we’ve been drilling it into new coders how to create this vulnerability from the very first day.

Secure coding best practices

If everyone is learning the wrong way from the start, how can we fix this? There are several things, but let’s start with how we build software: the system development life cycle (SDLC).

Some common security activities that can be added to the SDLC include:

• Security requirements: The measures that need to be put in place for the application to be considered secure. You’re building a serverless app? Cool! Here’s a list of security requirements we need you to add to your project’s requirements document to ensure it’s safe enough to put it on the internet.
• Threat modeling: A brainstorming session that includes at least one security professional, a product owner and/or business representative, and a member of the technical team. The goal is to identify as many potential threats to the product as possible, then work to mitigate the ones that seem the most damaging or dangerous. This is often done in the design phase, but it could be done at any point afterwards if you didn’t have time earlier in the SDLC. 
• Code scans: During the coding phase, you could scan your own code with a static application security testing tool (SAST) or a software composition analysis tool (SCA) or throw your app on a dev server and scan it with a DAST tool.
• Break it early: During the testing phase, you could hire a penetration tester to perform thorough automated and manual testing of your entire system.
• Protect code in production: During the release phase, you could set up monitoring, logging, and alerting for your application. You could install it behind a runtime application security protection (RASP) or web application firewall (WAF). 

All the items listed above are just the beginning: you can organize your secure SDLC in any way that works for your organization and teams. You can use some of these or none of these; the key is finding activities and tooling that work for your organization.

We can create a secure SDLC by adding security activities throughout out processes. That said, we have to follow the security steps every time, not just sometimes or when it’s convenient if we want to reliably produce secure software.

If you add one single security step or verification to your SDLC, you will create more secure applications. If you add more than one step, you will create even better applications. Software is always a tradeoff: we don’t want to spend a million dollars on security for an application that will only earn h a couple thousand dollars worth of value. We do want to ensure we are protecting our systems such that they meet the risk tolerance of our organization. In more plain language, we should work with the security team to decide exactly how much security is required, but generally it’s safe to assume that ‘more is more’ when it comes to security.

Zero trust, but not the marketing version

Quite often when designing systems, we create ‘implied trust’; that is, one or more parts of the system don’t verify something they should or could. When someone logs into a system, a security control verifies the user’s identity (via the username and password combination) and then grants them access to the parts they are allowed to access. In programming, we generally call the first part authentication (who are you?) and the second part authorization (should you be here?). If we skip this step, we are creating implied trust. We don’t care who the person is and are assuming that the person is allowed to be there.

Zero trust means never, ever, having any implied trust. Every single possible part of a system is locked down and only opened if it must be. This means closing all ports except the ones you need. It means blocking all connections except the ones you know you need. It means always verifying everything before using it. No trust in anything at all, even the other systems you have as dependencies.

Although zero trust is quite a lot of work to implement, it works. And it works well.

Examples of zero trust that you could implement in programming:

• Validating, sanitizing, and escaping all inputs (in that order)
• Each API, serverless, container, and app is an island; treat them like that. They should not trust each other!
• Authentication and authorization for everyone! Every single integration, even with systems built and maintained by your team, requires authentication and authorization, for every single connection.
• Output encoding, because you never know if someone changed something. Do not trust your user’s input or input that may have been tampered with!

I have never seen an organization implement every single possible variation of zero trust, but that’s okay. Applying zero trust principles to as many systems as possible is good enough. 

Buy, borrow, then build

Building security controls is hard. They are complex in nature to build, they are tested far more often and more aggressively than any other feature (thanks to malicious actors), and there are few public examples to work from due to companies wanting to protect their intellectual property. This puts us in a situation where building our own custom security control ends up being expensive, time consuming, and potentially even dangerous.

With this in mind, security professionals (and software architects, for that matter) often recommend we go in this order when we decide if we will use a pre-existing component or write our own: 

• Buy it if we can afford it with the budget we have. The reason for this is that it’s available immediately, we don’t need to assign resources to create and thus they are free for other work, we don’t need to maintain it ourselves (which is almost always the reason custom software costs more than buying it), and it is very likely to have had significantly more security testing performed against it (big customers demand intensive testing and high security assurance). Although the price tag may seem steep, when you calculate the risk, the wait, and the long-term cost of maintaining the system, it is almost always cheaper to buy than build.
• Borrow it. This can mean open source software, online examples, code from other teams, using the features in your framework, or even having an entire system supplied (this happens in government and nonprofit organizations more often than for-profit/private). Borrowing it is more of an expression than a reality, we mean ‘use code that is open source or other code you can use for free, but that you did not write’. The advantages of this are similar to the ‘buy’ section: the borrowed code is available immediately, it’s free, and we don’t have to maintain it ourselves. That said, if it’s given for free, it might not have had any security testing on it, meaning you should verify that it’s secure before using it if at all possible.  
• If neither of those is an option, we build it. 

This saves our organizations money, time and risk, despite the fact that building our own from scratch is usually way more fun.

With this in mind, whenever possible, use the security features found within your programming framework, such as: encryption, authentication, and session management. They are tried, tested, and true! 

Next, use third-party components (libraries, packages, gems, etc.), as they have (usually) been tested quite thoroughly. Verify third-party code and components via a software composition analysis (SCA) tool and reverify often. 

To summarize the buy/borrow/build philosophy: don’t write your own security features unless you absolutely have to. It’s hard to get right!

In closing, although we may not have been taught it in school, we can start with the steps above to begin creating more secure applications and code. By creating a secure system development lifecycle, avoiding implied trust within the systems we build, and only creating our own security controls when it is absolutely necessary, we will reduce the risk to our organizations in a consistently positive way. The organizations we work for are counting on us to be their first line of defense, and with these new strategies, you should be up to the task!

The post Three layers to secure a software development organization appeared first on Stack Overflow Blog.

We spoke with Dr. Jeannette (Jamie) Garcia, Senior Research Manager of Quantum Applications and Software at IBM Quantum, about their 433 qubit quantum computer and what the real life applications of quantum computing are today. 

The Q&A below has been edited for clarity. If you’d like to watch the full conversation, check out the video of our conversation. 

Ryan Donovan: How did you get into quantum computing?

Jamie Garcia: I am actually a chemist by training—I hold a PhD in chemistry. I came to IBM because I was very interested in some of the material science work that was going on there at the time and started doing some research in that space. I think most experimentalists will tell you if you get a weird outcome from an experiment, one of the first things you need to do is try to figure out why, and that involves a lot of the theory. I was running down the hallway to talk to my computational colleagues to help elucidate what was going on in my flask that I couldn’t actually see.

As a part of that process, I got very interested in computation as a whole and the simulation of nature and trying to use computation towards that end. I realized that there were some real challenges with using classical computers for certain reactions. I would ask my colleagues and they would tell me it was impossible. And I was like, why ?

RD: Can you give an example?

JG: For me, they were surprising examples—small molecules that were really reactive.

You think of radicals, for example, that wreak all sorts of havoc in our bodies, but also turn up in batteries too, which I was studying at the time. The reaction was so high energy and there were so many different things that had to happen with the chemistry that classical computers couldn’t model it even though they were small molecules. It’s just O2 size. 

When I was at Yorktown Heights one day walking down the hallway, I saw one of my colleagues had a poster and it had chemistry on it, which caught my eye. You don’t see that all that often at IBM . It turns out that he was using quantum computers to study a certain property of a molecule.

It stopped me in my tracks, and I realized this is a whole new tool for chemistry. Now we’ve expanded beyond chemistry. We’re looking at all sorts of different things, but that was what got me hooked and interested from the very beginning. 

RD: We’ve talked to a few folks in quantum computing, but I think it’s valuable to kind of get the basics here. What exactly is a qubit?

JG: A qubit is our analog to a classical bit. At IBM we use superconducting qubits. These have to be cooled down to around 15 millikelvin. You may have seen photos of our big dilution refrigerators that cool our qubits down to that level. They’re made out of superconducting materials. 

What you’re doing when you’re programming a qubit is you’re using the materials properties of those superconductors, you’re able to move electrons into different energy states. That basically allows you to program a quantum computer. One of the biggest challenges is keeping them in those states. And I have a feeling we’ll talk about that. 

RD: Especially with your material science background.That seems like that’s a big part of the ball game.

JG: But they’re fundamentally a kind of different beast too, because we’re now using and leveraging quantum mechanics to program the qubits and the quantum computers and be able to perform algorithms on them. So it has a different flavor to it than a classical bit.

In fact, you can use quantum mechanical properties such as superposition and entanglement. Those are new knobs to turn when you’re thinking about algorithms. In certain instances, it can be complementary to classical devices. But it really is a whole new area to explore.

RD: I’ve heard that cubits aren’t exactly stable. You have them super cooled and are trying to keep them in this particular state. To produce one qubit, do you need a lot of redundancy and error correction?

JG:When we’re talking about 433 qubits, it’s all on one chip, right? So when you program them, a lot of times, we leverage two qubit gates where you need to entangle two qubits together.

You set it up and map your circuit onto the qubits in a very specific way in order to get an answer. Now, the stability piece that you’re referring to—qubits are inherently sensitive. We have to cool the qubits that we use down to 15 millikelvin because of exactly what you said.

You’re trying to basically hold the qubit in this state for as long as possible so you can run the calculation that you need to run. Basically, you need to have enough time to perform the gate operations for your circuit. 

Qubits are susceptible to noise. Sometimes we know where that noise comes from and sometimes we don’t. When we think about how we arrange the qubits on the chip, we’re doing it in a way that minimizes noise most of the time. We use what’s called a heavy hex architecture. That limits the crosstalk between qubits to minimize the noise so you’re able to have as long coherence times as possible to run the circuits and do a practical calculation within hours, not in a lifetime. 

We’ve also developed a lot of other techniques to manage the noise. Error correction is something that our teams are working towards and developing out the theory for certain error correction that will include having a fault tolerant device and error rates low enough that we can actually run some of those codes.

But we’re also looking at error mitigation, which leverages classical post-processing methods and can capture the noise regardless of whether we know where it comes from or not, to be able to account for the noise and then correct for it so that we can get out as accurate results as maybe even in an error corrected regime.

There’s active research ongoing and software tools that are being developed so that we can leverage these techniques as they are developed in real time and use them for our applications research and run algorithms and circuits that are interesting to us.

One of the things that we’ve recently released, which you can actually access through Qisket runtime is something called probabilistic error cancellation. What this essentially does is when you run a circuit, it runs the inverse of certain parts of the circuit, and you effectively are able to learn where the noise is that way. Then the post post-processing divides it into smaller circuits and you can pull it all back together and account for the noise.

There are opportunities for machine learning, certainly. We’re thinking very seriously about how AI and quantum intersect. Especially since we just announced our System Two and the plans for that. We’re thinking very carefully about how all these things will play together and where AI can help quantum and where quantum can help AI.

RD: What’s the rough equivalent of 433 qubits to classical computing?

JG: This is a tough question to answer. We think of the qubits in terms of state. If you just do a rough back of the envelope calculation, people will usually say it’s two to the n. So two to the 433 [states] is a lot. Huge. I think two to 275, that’s more than the number of atoms in the universe. So it’s absolutely massive. 

But there’s a lot of nuance that goes into that, especially when we’re talking about actually programming a quantum computer and using it to look at a chemistry problem or a problem in finance or anything like that. In addition to that, you have to take into account the noise that you have present in the system.

So it’s hard to say about what the computing power today is of a device that has 433 qubits. If you project out to where someday we have error rates that are as close to zero as possible, then that’s where you start talking about this two to the n and harnessing the power of the universe. You know, all these things.

That’s the potential that it brings to us in terms of compute. 

RD: That two to the n is what exactly? 

JG: It’s basis states.

You can use the examples of molecules. Water might use somewhere around 14 qubits. If you have 14 qubits, then that’s 10 to the four classical bits, right?

You can calculate it out that way. But again, there’s a lot of nuance here. We need to carefully consider the types of problems that quantum will be good for. It’s not necessarily all the same problems that you can think of classical being good for. That’s my caveat, but it kind of gives you a rough idea.

RD: Some crypto algorithms are trying to be quantum safe, while others like Shor’s Algorithm are uniquely suited for quantum computing. Why is that?

JG: Shor’s is an algorithm that is in that long-term error corrected regime, right? You would need to use error correction for it. A lot of the famous algorithms that you’ve heard of that show exponential speed up with quantum computers, typically what we’re talking about are in that regime. There are some algorithms that are famous for chemistry, like quantum phase estimation.

That said,we’re, we’re doing a lot to bring bring algorithms closer to near term and error mitigation—and maybe even error mitigation combined with error correction—in these early days will allow us to start solving problems that I don’t think we would’ve thought that we would’ve been able to solve before as early as as this. 

Shor’s algorithm definitely leverages quantum devices that have these sort of ancilla qubits. When you think of the back of an envelope calculation for what you would need to be able to run Shor’s algorithm or crack RSA or something like that, you’ll see numbers that are in the millions of qubits. You have to account for that overhead that comes with the error correction. 

The asterisk is we’re doing things earlier than we thought. I think that that’s part of the reason that we’re talking about quantum safe now. We don’t know what the timeline is exactly, but we do have methods to address this that are available today. For example, our zSystems are quantum safe systems already. It’s definitely something to start considering now. If you had asked me the same question like two years ago, I would’ve said that’s so far away.

And now I’m like, Hmm. Start planning now.

RD: What other tasks or applications is quantum computing suited to?

JG: We think about it in three big buckets. The simulation of nature is one of them. That includes not just molecular simulations, but physics falls into this category. Material science falls into this category. You can think of this as being a space that’s interesting because nature is quantum mechanical. So if you are then leveraging a device that is also quantum mechanical—there’s some obvious connection there. In addition to that,there’s been theoretical proofs that show that there should be at least more than polynomial speed up possible with quantum computers with certain problems such as dynamics, energy states, ground states, and things of that nature. 

The second category is generally mathematics and processing data with complex structures. This is where quantum machine learning comes in. We talked about Shor’s and factoring. That fits into this category. There are algorithms that have been shown for quantum machine learning that imply that there should be an exponential speed up possible in certain cases.

We try to focus on these two areas in particular we think hold a lot of promise because they have this greater than polynomial potential associated with them for using a quantum computer. Those are really obvious areas to look at. 

The last category is search and optimization. So Grover’s falls into this category. These are areas that we don’t necessarily have theoretical proofs yet that there could be super polynomial speed up or greater than polynomial or exponential speed up. But we know that it promises probably somewhere around quadratic, maybe more. We’re still researching and looking, so you never know what you’re gonna find. 

There are certain algorithms like amplitude estimation and amplification that we think could act as accelerators for the other two areas that I talked about. Regardless of what kind of speed up, we would expect that it could still help in these other areas as well.

You can imagine it’s almost two to the n number of use cases that map onto those areas and it encompasses a lot of different things. We’re exploring a lot of different areas with partners and coupling it and tying it to things that are really valuable and hard classically.

That’s key, right? If something’s really easy classically, you could argue why look at quantum for it. Something that’s hard classically is where we think that quantum can lend some kind of advantage or some kind of speed up. In the long run, those are the areas that we’re exploring.

RD: Speaking of hypothetical use cases, have you seen the TV show Devs?

JG: No, what was the use case?

RD: Simulating the past and future.

JG: Oh my goodness. Okay…Well, there is prediction, right?

RD: Sure. I mean, simulating nature, right? 

JG: No, it’s not that far.

RD: Okay. Oh, no.

Because you are helping people process quantum jobs, are there any adjustments they need to make for their algorithms or data to be suitable for quantum computing?

JG: It depends on how you want to use quantum computers, right? A lot of our discussions are around—as we’re pointing to the next generation of these quantum-centric supercomputing centers and where you really have classical HPC next to a quantum device—how do you best leverage the workloads between those?

There’s a lot of things that we’ve been thinking about in terms of how you ideally would approach a problem. How would you set it up in such a way that you have the right parts of the problem being addressed classically and then other pieces with a quantum computer.

But the algorithms that we do and the circuits that we run are inherently different from classical ones. Again, it really comes down to how you divvy up the problem, and which pieces you want to put where. At a very high level, that’s what would need to be taken into consideration.

Something to point out here is that quantum computers aren’t big data types of devices. That’s another area that we think that there’s a lot to be done from the classical standpoint. But if you want to look at something that has a high complexity, high interconnectivity, or is by virtue dynamic, those are the kinds of things that the quantum computer handles really well.

If you were to run something on a quantum computer, you want to make sure that it’s the right circuit that’s going into it and the algorithm that you’re using.

RD: Is there anything else you wanted to cover that we didn’t talk about?

JG: In general, thinking about the different use cases and the different areas is really important to do as a field, right? This is a very multidisciplinary area, and we need to have folks that are coming from all points of view. Whether it’s software development, engineering, architects, and even those that are on more of the classical side.

Learning about quantum and bringing that lens has really pushed us forward in a truly unique way for this field. It has to do with the fact that it’s an emerging area. It’s all hands on deck and we’re all kind of learning together. 

The post The nature of simulating nature: Q&A with IBM quantum computing research appeared first on Stack Overflow Blog.

The aforementioned posts on this subject come across as about 80% fair and reasonable criticism, 20% out-of-touch grumbling. Or in other words:

Most developers know better than to say things like “it’s a smartphone OS, how hard can it be?” or “my spreadsheet app in the 90s was 10 kilobytes, how come Factorio is a full gigabyte?” If you weren’t there when it was built, you can’t reliably estimate all the hard knocks and complexity that went into it.

But that doesn’t mean there’s no room for objective criticism. Apps are slower than they used to be. And exponentially larger without a corresponding increase in value. At the very least, there are optimization opportunities in almost any modern app. We could make them faster, probably by orders of magnitude. We could remove code. We could write tiny, purpose-built libraries. We could find new ways to compress assets.

Why don’t we?

Prokopov’s answer is “software engineers aren’t taking pride in their work.” There’s some truth to that. But I strongly believe it’s the natural human state to work hard and make excellent things, and we only fail to do so when something repeatedly stops us. So instead of relying on the myth of laziness to explain slow and buggy software, we should be asking “what widespread forces and incentives are creating an environment where it’s hard for software engineers to do their best work?”

I have a few answers to that.

Speed is a feature, reliability is nothing

Software is envisioned by engineers as networks of interacting components, inputs, and outputs. This model is both accurate and useful. However, it’s not the way software is packaged, marketed, or sold. To businesspeople and customers, software is a list of features.

Take an inventory management app as an example. Its marketing materials will consist of several high-res stock photos, a bold color palette, and statements like the following:

• Tracks inventory across multiple warehouses
• Integrates with Delivery Pro, Supply Chain Plus, and Super Point-of-Sale systems
• Weekly and monthly reporting at multiple levels
• Fine-grained access and security controls
• Instant updates across all terminals
• Runs on Windows, MacOS, and Linux

These are falsifiable statements; either the software does these things or it does not. They can all be proven in a one-hour product demo. And only one deals with speed. The software may in fact be very slow, taking several seconds to respond to a button click, without making the “instant updates” claim a lie.

We can all agree that speed affects a user’s entire experience of an app. It’s an important marker of quality. But it’s difficult to sell. If you spend your time optimizing a core process while your competitor develops a new type of report, you’ll lose eight of your next ten sales over it. If you poll your existing customers about what you should work on next, they’re going to ask for features, not speed—unless the software is so slow it borders on unusable. And god forbid any red-blooded board of directors would allow the company to take a six-month detour from its product roadmap to work on technical debt. The pressure is always on us to build features, features, features.

Programmers want to write fast apps. But the market doesn’t care.

You may notice reliability isn’t on the list at all. How exactly would you say that? “Bug-free?” There’s no way to ensure that, let alone prove it in a product demo. “90% unit test coverage and a full suite of integration tests?” Nobody knows what that means and if you explained it to them, they’d be bored. There’s no way to express reliability in a way customers will both believe and care about. The Agile age has taught them that bugs will inevitably exist and you’ll fix them on an ongoing basis. And since there’s no comprehensive way to measure defects in software (surely if we knew about them, we would have already fixed them?) it’s not a feature that can be compared between products. We can invest time to test, refactor, and improve, but it’s entirely possible no one will notice.

Programmers want to write bug-free apps. But the market doesn’t care.

Disk usage isn’t on the list either, though occasionally it appears in small, low-contrast print below a “Download” button. And of everything here, this one is perhaps least connected with competitiveness or quality in customers’ minds. When was the last time you blamed a developer (as opposed to yourself or your computer) when you ran out of disk space? Or chose between two video games based on download size? Probably never. You can find people who complain about the size of the latest Call of Duty, but the sequels still make a billion dollars the week they come out.

Shrinking an executable or output bundle is thankless work. And it’s often highly technical work, requiring an understanding of not just the app one is building but the hundreds of lower-level libraries it depends on. Furthermore, it’s actively discouraged (“don’t reinvent the wheel”), partially because it’s a minefield. You may not know what a line of code is for, but that doesn’t mean it’s useless. Maybe it’s the difference between a working app and a broken one for the 0.01% of your customers that use Ubuntu on a smartphone. Maybe it’s the one thing keeping the app from crashing to a halt every four years on Leap Day. Even the smallest utility function eventually develops into an artifact of non-obvious institutional knowledge. It’s just not worth messing with.

Some programmers want to write smaller apps. But the benefits aren’t there for the market or for us.

Consumer software is undervalued

It’s not hard to distribute an app. That’s more or less what the Internet is for. But selling an app is like pulling teeth. The same general public who will pay $15 for a sandwich or a movie ticket—and then shrug and move on if they didn’t like it—are overcome by existential doubt if an app they’re interested in costs one (1) dollar. There are only two demographics that are willing to pay for good software: corporations and video gamers. We’ve somehow blundered our way into a world where everyone else expects software to be free.

This expectation has been devastating to the quality of consumer apps. Building an app costs anywhere from 50,000 to half a million dollars. If you can’t get people to pay on the way in, you have to recoup costs some other way. And herein are the biggest causes of bloat and slowness in both web and native applications: user tracking, ads, marketing funnels, affiliate sales, subscription paywalls, counter-counter-measures for all the above, and a hundred even-less-reputable revenue streams. These things are frequently attributed to greed, but more often they’re a result of desperation. Some of the most popular websites on the Internet are just barely scraping by.

It’s hard to overstate the waste and inefficiency of a system like this. You publish a unique, high-quality app for what you believe to be a fair price. It sits at zero downloads, day after day. You rebuild it on a free trial/subscription model. It gets a few hundred downloads but only a handful of users convert to a paid plan, not nearly enough to cover your costs. You put ads in the free version, even though it breaks your UI designer’s heart. You find out that ad views pay out in fractions of a cent. You put in more ads. Users (who, bafflingly, are still using the app for free) complain that there are too many ads. You swap some ads for in-app purchases. Users complain about those, too. You add call-to-action modals to encourage users to pay for the ad-free experience. You find out most of them would sooner delete the app. You add analytics and telemetry so you can figure out how to increase retention. You discover that “retention” and “addiction” might as well be synonyms. The cycle goes on, and before long you no longer have an app; you have a joyless revenue machine that exploits your users’ attention and privacy at every turn. And you’re still not making very much money.

We could avoid all of this if people were willing to pay for apps. But they’re not. So apps are huge and slow and broken instead.

Developers don’t realize the power they have

Lest I be accused of blaming everyone but myself, let’s examine the role of software developers. There has to be something we can do better.

Even in a recession, developers have an extraordinary amount of leverage. We can insist on working with (or not working with) specific technologies. We can hold out for high salaries, benefits, and equity. We can change the culture and work environment of an entire company by exercising even the slightest amount of solidarity. Good programmers are hard to come by. Everyone knows it, and we know they know it.

That’s our power, and we can do more with it.

We should set aside time in every sprint to resolve technical debt. We should procrastinate feature work now and then when there’s an especially promising opportunity to optimize and improve our code. We should persuade our employers to sponsor open-source projects. We should create the expectation that we won’t always be working on the product roadmap; our code and our industry expect more of us.

Most of the time there won’t be any negative consequences. We’re not asking too much. Every other industry has professional standards and requirements that transcend any one job description. Why do we so often act like software development doesn’t?

The only caveat is that the incentives aren’t in our favor. It’s an uphill battle. Some managers won’t be comfortable with us spending time on things they don’t understand. Some salespeople will worry that our software isn’t competitive. Investors may threaten to outsource our work to more pliable developers. It will be a while before customer attitudes and market forces shift. But if changing the state of modern software is a worthy goal, then it’s worth the effort.

Will it get better?

It’s hard to be optimistic about the future of software. Programmers were allowed to build tiny, highly-optimized apps in the 90s because there was no other choice. Their customers had 32 megabytes of RAM and a 200 megahertz single-core processor. If an app wasn’t as lean as possible, it wouldn’t run at all. Today, a two-year-old base-model Macbook Air has 250 times as much memory (not to mention faster memory) and a quad-core processor with several times the speed on any one core. You can get away with a lot more now. And we do. We ship apps that are 90% dead weight. We don’t optimize until someone complains. We package a full web browser installation with apps for sending messages, taking notes, even writing our own code (I’m using one right now).

The last two decades have been dedicated to making software development faster, easier, and more foolproof. And admittedly, we’re creating apps faster than ever, with more features than ever, using less experienced developers than ever. It’s not hard to see the appeal from a business perspective. But we’re paying the price—and so are our customers, the power grid, and the planet.

Things won’t change overnight, probably not even in the next five years. But there are reasons to be hopeful.

The latest wave of web programming languages and technologies (like WebAssembly, ESBuild, SWC, Bun, and Yew) is enabling new levels of speed and reliability, both at compile-time and runtime. Rust, noted for delivering the performance of C and the developer-friendliness of higher-level languages, is gaining popularity on web servers. Lightweight Electron alternatives like Tauri are poised to take over as the web developer’s cross-platform framework of choice. Tree-shaking is something we’ve come to expect from compilers and bundlers.

In terms of the market, several popular video games (like Dead Cells and The Binding of Isaac) have made their way to mobile platforms as paid downloads. There’s still a lot of work to be done, but this is promising headway towards reeducating smartphone users, the world’s largest group of technology consumers, about the cost of software.

If the last 20 years have been about making us more productive—sacrificing efficiency and financial sustainability in the process—perhaps the next 20 will be about tackling our collective technical debt, reclaiming efficiency, and improving economic exchange without losing the productivity that’s made software omnipresent in our lives.

The post Is software getting worse? appeared first on Stack Overflow Blog.