In a 40-hours work week filled with depressing code-related absurdities, I can always hop on here for a quick dose of dopamine and a reminder that things can, and should, be better - and that I'm not crazy for thinking that.
There is another common excuse though, I often hear when people talk about doing performant code that they are doing premature optimization, that you should only care for it when it becomes a real problem and you actually need to rewrite everything. (Even in a high demanding area such as game development)
I wonder what you opinion in this matter is? When is too early to do a performant solution? Is there really such a thing as premature optimization? If so is there a better approach to make our life’s easier so we don’t have to rewrite everything every time our solution is not performant enough?
Completely agree with everything in the video. Personally I've seen quite a bit that a company want each service to be responsible of a single thing, with a database for each one. So to fulfill a simple request I have to go through a call chain of services, each call goes though the network, and to a lot of databases queries. And that just keeps going, adding more and more services for each thing.
Also another one is that to fix the issue above, they put Kafka or something else to allow failures of those services. To the point that a request to do 10 things will be split in 10 Kafka messages (actually what I see is from 10 to 100) to be consumed. And the sole reason to do that, is to make the system more "resilient".
I'm not sure if it is only me, but that sounds really bad to me. Since the probability of an error increases a lot by doing it so complex. And then, you "need" all those machines to be able to handle the load, which is no more than a 1000 request per min.
This video was great Casey! Going through all of those public examples was super interesting.
To add my own performance anecdote here: I used to work for Facebook and there was a huge internal effort to make things more performant. I remember literally multiple efforts to reduce cpu usage in binaries that ran on the entire fleet by as little as 1%. When you multiply that 1% by 5MM servers, the cost savings are pretty crazy.
I even did work to reduce bloat from code generated by their Thrift compiler. 100s of Go binaries went down in size by multiple MBs, and that stuff starts to matter when you’re copying executables around for deployment (sometimes many times a day)
Maybe part of the reason developers tend to make these excuses for not caring about performance is this disconnect between their software and how much it costs to run.
I can tell from personal experience that performance in web frontend can be absolutely critical for customers in at least 2 companies I have worked at. In the second company this happened:
> Webapp is slow and everybody is blaming the frontend framework and talking about rewriting in a new one
> I finally sat down and figured out that the problem was actually a slow memory leak. The javascript heap grew by 70MB or more if you left specific features of the app running for more than 2 hours. And for this app running for 2+ hours was a common usecase. After this happened GC hits increased and everything went badly in general.
> I fixed the problem. Found many other things I could fix in the course of profiling for those things. It took me months.
> Finally merge my changes. Within the week a surprized looking CEO says, "the app has been under load for a while and there are no complaints in the customer-duty channel. Did we really get that much better?" 😂
This was a turn around from a total gloom and doom situation that had been going on for months where it looked like the app will just not scale. And I think it was all because of performance improvements. Although I cannot rule out the the possibility of sales team doing something Im not aware of, I know that literally everyone liked the app much more after I merged those changes.
Another time, in another company there was a form which had multiple tabs and each tab and a bunch of ng-tables. This form was soooo goddamn slow that you would click on a checkbox and get coffee (literally). The product manager who was in charge of that part wanted it to be made faster but nobody thought it could be any faster (it had been that way for a year or more). I said something along the lines of "ill try to make it faster but no guarantees". Turns out the thing was just a monstrosity of useless libraries being used to do very simple things that could be done with no libraries. Removed them all over a weekend, the form became just like any other normal web form. They could not believe it. And I gained a lot of reputation :P
These are just two times I can remember when performance was the thing I worked on in web frontends. And both times I had to do it with the understanding that I am doing it for my own sake, while others talked about a "full rewrite". And then when I had just a little bit of success everyone just relaxed and then the "optimization" effort was stopped right there.
I think the issue is that on the web perf doesn't matter until the app is unusable. Once it starts becoming unusable, it only matters till it becomes usable again. And despite so much tooling being available for performance tuning, I don't know of many people who actually understand how to use them.
On top of everything you have said so far about why "performance aware" programming is important I have a few others (which can be seen as more personal choice)
- Help the environment by allowing your software to run in older hardware instead of blindly forcing your users to update because your software is so badly constructed.
- Caring about the craft in itself, and being proud of what you do for a living, by always trying to deliver the best results with the resources you have at hand.
- Respect your users/client and dont deliver a bad product just because they dont know any better.
- Not assuming writing "clean" code will somehow make your life any easier down the road. I have work in big teams following those programing mantras and the results were always the same. Poor performant code that was horrible to modified because of all the layers of abstraction you needed to learn (and relearn) before touching anything. That also applies to finding and fixing bugs.
- Writting decent code is not slower than writting poorly design code. For any non trivial task, understanding the problem and comming up with a solution is what takes most of your time, unless you type really slow.
Great video. Could you explain however why do you disable comments on all YouTube videos on MollyRocket channel? There seem to be some people, especially ones who are not associated with you, that take disabling comments especially on those types of videos as avoiding discussion.
I bought a subscription to an online accounting software. From week one, I was trying to import the opening balances from the previous software (less than 1k rows with three columns csv), it takes five minutes to render the UI for the rows every time I paste the rows. And I had to do it many times because I was still figuring things out and as a result I wasted hours on it.
Another problem, when I'm typing in a dropdown to select an account, it was doing a request to the server with some kind of cooldown, so I had to stop typing and wait for it to get the list of accounts that matches my search, the previous one was instantaneous because the accounts were downloaded and stored in the client.
Every time I click a button, create something, hit next or click the next page I have to wait for the page to load, sometimes it is half second and sometimes it is 5 seconds. It is ridiculous. The old one is also web but has windows that I can open and move around and there is no page reloading whatsoever, it almost feels like a native app.
Later that week, I bit the bullet, accepted the cost and stayed with the old one.
So, yes! Performance was the most important thing in my case! (even though the old one could be made faster).
By the way, the old software is written by me and I always felt it sucked and I thought switching is going to be amazing. Now, I dedicate small amount of time every week to improve it and I'm never switching again.
In the course will you talk about network architecture, batching requests, how to scale, manage memory, conserve battery life, and all those things you mentioned at the end of the video?
So, the companies mentioned in your email didn't prioritize performance in their initial application versions because, at that time, they:
- Didn't need it
- Had sufficient hardware
- Found it more cost-effective to develop
- Could develop more quickly
- Released features rapidly
Only after years did they decide to refactor proven features for improved performance in production! :)
Discussions about the need for performance typically revolve around balancing development time and convenience against hiring skilled engineers to create fast, high-quality software from the start. This generally pertains to the first version of the software, rather than the 4th or 8th versions, when features developed quickly with a focus on speed-to-market have already generated revenue and a user base that justifies their existence.
Indeed, performance may not be a top priority from the beginning.
Furthermore, it's important to stop criticizing developers who don't prioritize performance and start discussing the realistic impact of creating a performant or non-performant app based on its purpose:
- For games, calculators, or terminals: Ensure the app meets the required frames per second and responsiveness, then release it to the market, since there is NO impact that would depend on of 120fps or 8000 fps you have
- For IDEs: Identify the target audience and average project size, focus on features and accessibility, use any remaining time to optimize for adequate responsiveness, and then release to the market. NO impact if there is 100ms of 500ms of loading the project.
- For servers: This is when performance optimization becomes more important, as it allows users to consume fewer resources and reduce their total cost of ownership. However, for the first version, it's still crucial to focus on features and delivery to test the market. There IS impact due to the multiple instances of your software running, causing direct impact on TCO.
Casey, this message is sorely needed in the programming world. But apparently the rhetoric has surpassed even the realm of "enterprise software" and into video games too. Look at this reply I got 4 hours ago in the comments section of a video going over your debate with Uncle Bob:
"You act as if performance bottlenecks are 99% poorly optimized code. If you’re making some game with realistic graphics for example, the performance hoard will always be the rendering no matter how optimized it is. So if my character movement is 20x slower than it could hypothetically be why would I care? I could heavily optimize all off the gameplay and the resulting performance of the game would barely change. It would be much nicer then to make it as readable and modular as possible"
You'd think that if there were any industry where performance aware programming mattered, it would be the video game industry. But look- the rhetoric has spread enough to where this guy actually uses a video game as an example. Assuming that by "character movement being 20x slower than it could" he if referring to the "performance of the section of the codebase dictating character movement", and not actual character speed through the world (which would have a huge impact on rendering performance), this still seems like a very bad example.
Can you go over some concrete reasons why this rhetoric especially doesn't hold true for video games in particular, over 'enterprise software' like facebook etc.
This problem may end up going away by itself over time. People probably adopted languages like python as doing stuff in them was just way easier than working with C, over-complicated build systems etc (about 12 years ago).
Now there are compiled languages like golang and rust that offer a lot of the same benefits, without most of the waste. And they make it more viable to use lower level control if that's needed.
I'm also looking forward to Jai getting a general release.
I've found a lot of other excuses coming from people who just flat out don't believe they'll ever need to be concerned with performance because their software itself is solving a niche problem, so they're actually just waiting until a better version of their solution is made by someone else.
My father in law works in flooring and the software that he uses to create invoices after he's created a floor plan that pulls in QOH at the store is essentially one of three solutions that exist. His survives literally just by supporting an arcane file format only used by one company, but even then adoption is slowing vs other solutions as these same corporations are trying to migrate AWAY from legacy. Combined with an old windows UI, the slow response time all but guarantees its eventually demise.
Ooh I like this end: "if we do a little bit of work to preserve those performance options things get a lot easier"
This seems to hint at a way of doing development which allows you to prototype and do quick and dirty V1 releases, while at the same time setting yourself up for enhancing performance down the road without a rewrite.
These days I write my personal projects in C, and it can take some time to get things off the ground, would love to know about a way to improve my prototyping time while not giving up on having something that I can optimize later and isn't locked into some other unoptimizable framework!
In a 40-hours work week filled with depressing code-related absurdities, I can always hop on here for a quick dose of dopamine and a reminder that things can, and should, be better - and that I'm not crazy for thinking that.
Loved the video.
There is another common excuse though, I often hear when people talk about doing performant code that they are doing premature optimization, that you should only care for it when it becomes a real problem and you actually need to rewrite everything. (Even in a high demanding area such as game development)
I wonder what you opinion in this matter is? When is too early to do a performant solution? Is there really such a thing as premature optimization? If so is there a better approach to make our life’s easier so we don’t have to rewrite everything every time our solution is not performant enough?
Completely agree with everything in the video. Personally I've seen quite a bit that a company want each service to be responsible of a single thing, with a database for each one. So to fulfill a simple request I have to go through a call chain of services, each call goes though the network, and to a lot of databases queries. And that just keeps going, adding more and more services for each thing.
Also another one is that to fix the issue above, they put Kafka or something else to allow failures of those services. To the point that a request to do 10 things will be split in 10 Kafka messages (actually what I see is from 10 to 100) to be consumed. And the sole reason to do that, is to make the system more "resilient".
I'm not sure if it is only me, but that sounds really bad to me. Since the probability of an error increases a lot by doing it so complex. And then, you "need" all those machines to be able to handle the load, which is no more than a 1000 request per min.
Every project I ever worked during my 12 year programming career that had even a single user, included a lot of work on performance.
Only the last 3 years were in performance oriented domain.
This video was great Casey! Going through all of those public examples was super interesting.
To add my own performance anecdote here: I used to work for Facebook and there was a huge internal effort to make things more performant. I remember literally multiple efforts to reduce cpu usage in binaries that ran on the entire fleet by as little as 1%. When you multiply that 1% by 5MM servers, the cost savings are pretty crazy.
I even did work to reduce bloat from code generated by their Thrift compiler. 100s of Go binaries went down in size by multiple MBs, and that stuff starts to matter when you’re copying executables around for deployment (sometimes many times a day)
Maybe part of the reason developers tend to make these excuses for not caring about performance is this disconnect between their software and how much it costs to run.
I can tell from personal experience that performance in web frontend can be absolutely critical for customers in at least 2 companies I have worked at. In the second company this happened:
> Webapp is slow and everybody is blaming the frontend framework and talking about rewriting in a new one
> I finally sat down and figured out that the problem was actually a slow memory leak. The javascript heap grew by 70MB or more if you left specific features of the app running for more than 2 hours. And for this app running for 2+ hours was a common usecase. After this happened GC hits increased and everything went badly in general.
> I fixed the problem. Found many other things I could fix in the course of profiling for those things. It took me months.
> Finally merge my changes. Within the week a surprized looking CEO says, "the app has been under load for a while and there are no complaints in the customer-duty channel. Did we really get that much better?" 😂
This was a turn around from a total gloom and doom situation that had been going on for months where it looked like the app will just not scale. And I think it was all because of performance improvements. Although I cannot rule out the the possibility of sales team doing something Im not aware of, I know that literally everyone liked the app much more after I merged those changes.
Another time, in another company there was a form which had multiple tabs and each tab and a bunch of ng-tables. This form was soooo goddamn slow that you would click on a checkbox and get coffee (literally). The product manager who was in charge of that part wanted it to be made faster but nobody thought it could be any faster (it had been that way for a year or more). I said something along the lines of "ill try to make it faster but no guarantees". Turns out the thing was just a monstrosity of useless libraries being used to do very simple things that could be done with no libraries. Removed them all over a weekend, the form became just like any other normal web form. They could not believe it. And I gained a lot of reputation :P
These are just two times I can remember when performance was the thing I worked on in web frontends. And both times I had to do it with the understanding that I am doing it for my own sake, while others talked about a "full rewrite". And then when I had just a little bit of success everyone just relaxed and then the "optimization" effort was stopped right there.
I think the issue is that on the web perf doesn't matter until the app is unusable. Once it starts becoming unusable, it only matters till it becomes usable again. And despite so much tooling being available for performance tuning, I don't know of many people who actually understand how to use them.
On top of everything you have said so far about why "performance aware" programming is important I have a few others (which can be seen as more personal choice)
- Help the environment by allowing your software to run in older hardware instead of blindly forcing your users to update because your software is so badly constructed.
- Caring about the craft in itself, and being proud of what you do for a living, by always trying to deliver the best results with the resources you have at hand.
- Respect your users/client and dont deliver a bad product just because they dont know any better.
- Not assuming writing "clean" code will somehow make your life any easier down the road. I have work in big teams following those programing mantras and the results were always the same. Poor performant code that was horrible to modified because of all the layers of abstraction you needed to learn (and relearn) before touching anything. That also applies to finding and fixing bugs.
- Writting decent code is not slower than writting poorly design code. For any non trivial task, understanding the problem and comming up with a solution is what takes most of your time, unless you type really slow.
Great video. Could you explain however why do you disable comments on all YouTube videos on MollyRocket channel? There seem to be some people, especially ones who are not associated with you, that take disabling comments especially on those types of videos as avoiding discussion.
I bought a subscription to an online accounting software. From week one, I was trying to import the opening balances from the previous software (less than 1k rows with three columns csv), it takes five minutes to render the UI for the rows every time I paste the rows. And I had to do it many times because I was still figuring things out and as a result I wasted hours on it.
Another problem, when I'm typing in a dropdown to select an account, it was doing a request to the server with some kind of cooldown, so I had to stop typing and wait for it to get the list of accounts that matches my search, the previous one was instantaneous because the accounts were downloaded and stored in the client.
Every time I click a button, create something, hit next or click the next page I have to wait for the page to load, sometimes it is half second and sometimes it is 5 seconds. It is ridiculous. The old one is also web but has windows that I can open and move around and there is no page reloading whatsoever, it almost feels like a native app.
Later that week, I bit the bullet, accepted the cost and stayed with the old one.
So, yes! Performance was the most important thing in my case! (even though the old one could be made faster).
By the way, the old software is written by me and I always felt it sucked and I thought switching is going to be amazing. Now, I dedicate small amount of time every week to improve it and I'm never switching again.
In the course will you talk about network architecture, batching requests, how to scale, manage memory, conserve battery life, and all those things you mentioned at the end of the video?
I can has new episode? https://media.tenor.com/images/796381f5d09bb121ed947339d3f0c24a/tenor.png
So, the companies mentioned in your email didn't prioritize performance in their initial application versions because, at that time, they:
- Didn't need it
- Had sufficient hardware
- Found it more cost-effective to develop
- Could develop more quickly
- Released features rapidly
Only after years did they decide to refactor proven features for improved performance in production! :)
Discussions about the need for performance typically revolve around balancing development time and convenience against hiring skilled engineers to create fast, high-quality software from the start. This generally pertains to the first version of the software, rather than the 4th or 8th versions, when features developed quickly with a focus on speed-to-market have already generated revenue and a user base that justifies their existence.
Indeed, performance may not be a top priority from the beginning.
Furthermore, it's important to stop criticizing developers who don't prioritize performance and start discussing the realistic impact of creating a performant or non-performant app based on its purpose:
- For games, calculators, or terminals: Ensure the app meets the required frames per second and responsiveness, then release it to the market, since there is NO impact that would depend on of 120fps or 8000 fps you have
- For IDEs: Identify the target audience and average project size, focus on features and accessibility, use any remaining time to optimize for adequate responsiveness, and then release to the market. NO impact if there is 100ms of 500ms of loading the project.
- For servers: This is when performance optimization becomes more important, as it allows users to consume fewer resources and reduce their total cost of ownership. However, for the first version, it's still crucial to focus on features and delivery to test the market. There IS impact due to the multiple instances of your software running, causing direct impact on TCO.
Casey, this message is sorely needed in the programming world. But apparently the rhetoric has surpassed even the realm of "enterprise software" and into video games too. Look at this reply I got 4 hours ago in the comments section of a video going over your debate with Uncle Bob:
"You act as if performance bottlenecks are 99% poorly optimized code. If you’re making some game with realistic graphics for example, the performance hoard will always be the rendering no matter how optimized it is. So if my character movement is 20x slower than it could hypothetically be why would I care? I could heavily optimize all off the gameplay and the resulting performance of the game would barely change. It would be much nicer then to make it as readable and modular as possible"
You'd think that if there were any industry where performance aware programming mattered, it would be the video game industry. But look- the rhetoric has spread enough to where this guy actually uses a video game as an example. Assuming that by "character movement being 20x slower than it could" he if referring to the "performance of the section of the codebase dictating character movement", and not actual character speed through the world (which would have a huge impact on rendering performance), this still seems like a very bad example.
Can you go over some concrete reasons why this rhetoric especially doesn't hold true for video games in particular, over 'enterprise software' like facebook etc.
This problem may end up going away by itself over time. People probably adopted languages like python as doing stuff in them was just way easier than working with C, over-complicated build systems etc (about 12 years ago).
Now there are compiled languages like golang and rust that offer a lot of the same benefits, without most of the waste. And they make it more viable to use lower level control if that's needed.
I'm also looking forward to Jai getting a general release.
I've found a lot of other excuses coming from people who just flat out don't believe they'll ever need to be concerned with performance because their software itself is solving a niche problem, so they're actually just waiting until a better version of their solution is made by someone else.
My father in law works in flooring and the software that he uses to create invoices after he's created a floor plan that pulls in QOH at the store is essentially one of three solutions that exist. His survives literally just by supporting an arcane file format only used by one company, but even then adoption is slowing vs other solutions as these same corporations are trying to migrate AWAY from legacy. Combined with an old windows UI, the slow response time all but guarantees its eventually demise.
Ooh I like this end: "if we do a little bit of work to preserve those performance options things get a lot easier"
This seems to hint at a way of doing development which allows you to prototype and do quick and dirty V1 releases, while at the same time setting yourself up for enhancing performance down the road without a rewrite.
These days I write my personal projects in C, and it can take some time to get things off the ground, would love to know about a way to improve my prototyping time while not giving up on having something that I can optimize later and isn't locked into some other unoptimizable framework!