Designing is hard.
Designing for performance is harder.
Designing for performance under extreme load is an art that requires a village.
In VSAN, I started off with designing a system that would work under 90% of cases and workloads.
Even before that milestone was achieved, the performance team had jumped in and shown me that the initial design was just not good enough for the 10% use case. My first thought was "Who cares? That's really not what I'm targeting".
It turns out that even though the performance folks were "not holding it right", they were doing exactly what all our customers would be doing first: Running a completely fabricated synthetic benchmark... which meant that even though it would be less useful, I would have to cater to it.
The result was me tweaking the design so that it would
1. do-no-harm to the 90% cases
2. address as best as I could the 10% cases.
At the end of this design phase and implementation, the performance had jumped by orders of magnitude for the 10% cases and had improved by a few percentage points for the 90% cases.
In addition, it had reduced "performance jitter".
Even before I had time to bask in the glory of this wave of improvements, I had already been shown the next problem: Performance under load.
This is where simple unit tests, regression tests and regular overload tests wouldn't do.
It required automation of all those tests and one crazy person who'd go stress the system beyond not only what we'd publicly advertise, but also beyond what we'd privately accept as the limits of what we've built.
The way he reported the problem was also funny enough for it to become a meme within my team: He'd start some ridiculously stressful test and go for lunch. On returning, the stress test would have failed (of course) and he'd come with a big smile and say
"I started the test and went for lunch. When I came back it had crashed. I didn't do anything else."
We considered the possibility of not letting him ever go for lunch after the third time. :D
That fellow's efforts led of course to a wave of even more improvements and bug fixes.
The final design was a piece of art. It had beautiful poetry and rough Klingon.
I could never have achieved the final design in the first shot.
Learnings:
Designing for performance is harder.
Designing for performance under extreme load is an art that requires a village.
In VSAN, I started off with designing a system that would work under 90% of cases and workloads.
Even before that milestone was achieved, the performance team had jumped in and shown me that the initial design was just not good enough for the 10% use case. My first thought was "Who cares? That's really not what I'm targeting".
It turns out that even though the performance folks were "not holding it right", they were doing exactly what all our customers would be doing first: Running a completely fabricated synthetic benchmark... which meant that even though it would be less useful, I would have to cater to it.
The result was me tweaking the design so that it would
1. do-no-harm to the 90% cases
2. address as best as I could the 10% cases.
At the end of this design phase and implementation, the performance had jumped by orders of magnitude for the 10% cases and had improved by a few percentage points for the 90% cases.
In addition, it had reduced "performance jitter".
Even before I had time to bask in the glory of this wave of improvements, I had already been shown the next problem: Performance under load.
This is where simple unit tests, regression tests and regular overload tests wouldn't do.
It required automation of all those tests and one crazy person who'd go stress the system beyond not only what we'd publicly advertise, but also beyond what we'd privately accept as the limits of what we've built.
The way he reported the problem was also funny enough for it to become a meme within my team: He'd start some ridiculously stressful test and go for lunch. On returning, the stress test would have failed (of course) and he'd come with a big smile and say
"I started the test and went for lunch. When I came back it had crashed. I didn't do anything else."
We considered the possibility of not letting him ever go for lunch after the third time. :D
That fellow's efforts led of course to a wave of even more improvements and bug fixes.
The final design was a piece of art. It had beautiful poetry and rough Klingon.
I could never have achieved the final design in the first shot.
Learnings:
- There was no way the design could have been made perfect from the first day.
- Even if I had the "perfect design" it would be almost impossible to implement it from scratch.
- There's no way to predict the performance optimizations required without having experienced the problems first. This goes well with the philosophy of "premature optimization is the root of all evil".
- There's no way to do all this alone.
- Unit tests are only the beginning. Performance oriented test teams and automation are mandatory. After that is done, there has to be at least one person who will be the "chaos monkey" to find new ways to break the system.
No comments:
Post a Comment