Evgeny Ivanov
@e10v.me
Product director and former principal data scientist at a services marketplace
Pinned
Evgeny Ivanov
@e10v.me
· Jul 29
I'm excited to introduce tea-tasting, a Python package for the statistical analysis of A/B tests 📈🎉
It features Student's t-test, Bootstrap, variance reduction using CUPED, power analysis, and other statistical methods.
1/5
It features Student's t-test, Bootstrap, variance reduction using CUPED, power analysis, and other statistical methods.
1/5
I was thinking about the labor market congestion problem and came up with a solution that is often used in service marketplaces: pay to apply... 🧵
Prices as model weights: learning from dispersed data | Evgeny Ivanov
Market design to reduce labor market congestion.
e10v.me
October 24, 2025 at 6:33 AM
I was thinking about the labor market congestion problem and came up with a solution that is often used in service marketplaces: pay to apply... 🧵
Another perspective on marshmallow test: market risk and diminishing marginal utility
October 10, 2025 at 7:31 AM
Another perspective on marshmallow test: market risk and diminishing marginal utility
It’s astonishing that Google employs so many multilingual people yet designs products as if users speak only one language
August 31, 2025 at 7:01 AM
It’s astonishing that Google employs so many multilingual people yet designs products as if users speak only one language
Some mentions of the quote "all models are wrong, but some are useful" should be followed by "the map is not the territory"
July 17, 2025 at 4:03 PM
Some mentions of the quote "all models are wrong, but some are useful" should be followed by "the map is not the territory"
We wanted flying cars, instead we got scooter riders honking at our backs and knocking us off our feet
July 2, 2025 at 7:49 AM
We wanted flying cars, instead we got scooter riders honking at our backs and knocking us off our feet
Pre-LLM era take-home assignments: focus on evaluating *answers*; LLM era: focus on evaluating *questions*.
"Here is the problem-solving case: […]. Use an LLM to solve it. Provide a summary of your conversation with the LLM, including the questions you asked and the final solution you obtained."
"Here is the problem-solving case: […]. Use an LLM to solve it. Provide a summary of your conversation with the LLM, including the questions you asked and the final solution you obtained."
May 26, 2025 at 10:03 AM
Pre-LLM era take-home assignments: focus on evaluating *answers*; LLM era: focus on evaluating *questions*.
"Here is the problem-solving case: […]. Use an LLM to solve it. Provide a summary of your conversation with the LLM, including the questions you asked and the final solution you obtained."
"Here is the problem-solving case: […]. Use an LLM to solve it. Provide a summary of your conversation with the LLM, including the questions you asked and the final solution you obtained."
Every relevant cost can ultimately be framed as an opportunity cost. So, in essence, opportunity cost is the only cost that matters.
May 13, 2025 at 1:01 PM
Every relevant cost can ultimately be framed as an opportunity cost. So, in essence, opportunity cost is the only cost that matters.
A ladder of questions in analysis:
1. Which question should we ask to address a decision or a problem?
2. Which model should we choose to answer the question?
3. Which assumptions must hold for the model to be valid, and do they hold?
... 🧵
#DecisionIntelligence #ModelThinking #Logic
1. Which question should we ask to address a decision or a problem?
2. Which model should we choose to answer the question?
3. Which assumptions must hold for the model to be valid, and do they hold?
... 🧵
#DecisionIntelligence #ModelThinking #Logic
May 12, 2025 at 12:35 PM
A ladder of questions in analysis:
1. Which question should we ask to address a decision or a problem?
2. Which model should we choose to answer the question?
3. Which assumptions must hold for the model to be valid, and do they hold?
... 🧵
#DecisionIntelligence #ModelThinking #Logic
1. Which question should we ask to address a decision or a problem?
2. Which model should we choose to answer the question?
3. Which assumptions must hold for the model to be valid, and do they hold?
... 🧵
#DecisionIntelligence #ModelThinking #Logic
Reposted by Evgeny Ivanov
✨ Narwhals now supports PySpark
🎇 If you have a dataframe-agnostic function, you can pass in `pyspark.sql.DataFrame`
📈 Here's a demo, made with @marimo.io
🎁 BONUS feature: combine with SQLFrame, to generate SQL from @pola.rs syntax 🪄
🎇 If you have a dataframe-agnostic function, you can pass in `pyspark.sql.DataFrame`
📈 Here's a demo, made with @marimo.io
🎁 BONUS feature: combine with SQLFrame, to generate SQL from @pola.rs syntax 🪄
May 9, 2025 at 1:41 PM
✨ Narwhals now supports PySpark
🎇 If you have a dataframe-agnostic function, you can pass in `pyspark.sql.DataFrame`
📈 Here's a demo, made with @marimo.io
🎁 BONUS feature: combine with SQLFrame, to generate SQL from @pola.rs syntax 🪄
🎇 If you have a dataframe-agnostic function, you can pass in `pyspark.sql.DataFrame`
📈 Here's a demo, made with @marimo.io
🎁 BONUS feature: combine with SQLFrame, to generate SQL from @pola.rs syntax 🪄
The version 1.0 of tea-tasting, a Python package for the statistical analysis of A/B tests, is now available. Notable improvements:
- Interactive user guides built with @marimo_io notebooks.
- Simulated experiments, including A/A tests.
#abtesting #statistics #datascience
- Interactive user guides built with @marimo_io notebooks.
- Simulated experiments, including A/A tests.
#abtesting #statistics #datascience
May 5, 2025 at 1:42 PM
The version 1.0 of tea-tasting, a Python package for the statistical analysis of A/B tests, is now available. Notable improvements:
- Interactive user guides built with @marimo_io notebooks.
- Simulated experiments, including A/A tests.
#abtesting #statistics #datascience
- Interactive user guides built with @marimo_io notebooks.
- Simulated experiments, including A/A tests.
#abtesting #statistics #datascience
Wow, ChatGPT is really smart now 😜
But better check for yourself: github.com/e10v/tea-tas...
#abtesting #statistics #chatgpt #ai
But better check for yourself: github.com/e10v/tea-tas...
#abtesting #statistics #chatgpt #ai
March 31, 2025 at 9:14 AM
Wow, ChatGPT is really smart now 😜
But better check for yourself: github.com/e10v/tea-tas...
#abtesting #statistics #chatgpt #ai
But better check for yourself: github.com/e10v/tea-tas...
#abtesting #statistics #chatgpt #ai
It's actually Cunningham's Law... Wait... Dammit!
Turnbull's law: "The best way to get the right answer on the Internet is not to ask a question; it's to post the wrong answer." I love it! :D
December 26, 2024 at 3:10 PM
It's actually Cunningham's Law... Wait... Dammit!
The new version of tea-tasting, a Python package for the statistical analysis of A/B tests, is out.
Multiple hypothesis testing correction procedures to control false discovery rate (FDR) and family-wise error rate (FWER) have been added: tea-tasting.e10v.me/multiple-tes...
Multiple hypothesis testing correction procedures to control false discovery rate (FDR) and family-wise error rate (FWER) have been added: tea-tasting.e10v.me/multiple-tes...
Multiple testing - tea-tasting: statistical analysis of A/B tests
A Python package for the statistical analysis of A/B tests
tea-tasting.e10v.me
December 2, 2024 at 1:00 PM
The new version of tea-tasting, a Python package for the statistical analysis of A/B tests, is out.
Multiple hypothesis testing correction procedures to control false discovery rate (FDR) and family-wise error rate (FWER) have been added: tea-tasting.e10v.me/multiple-tes...
Multiple hypothesis testing correction procedures to control false discovery rate (FDR) and family-wise error rate (FWER) have been added: tea-tasting.e10v.me/multiple-tes...
Wow, tea-tasting has just reached 200 stars ⭐📈
It's not thousands, but it still matters to me, given the narrow niche of statistical analysis of A/B tests in Python.
#abtesting #statisics
It's not thousands, but it still matters to me, given the narrow niche of statistical analysis of A/B tests in Python.
#abtesting #statisics
August 21, 2024 at 1:05 PM
Wow, tea-tasting has just reached 200 stars ⭐📈
It's not thousands, but it still matters to me, given the narrow niche of statistical analysis of A/B tests in Python.
#abtesting #statisics
It's not thousands, but it still matters to me, given the narrow niche of statistical analysis of A/B tests in Python.
#abtesting #statisics
Reposted by Evgeny Ivanov
I sorta dug around and slapped together a data people starter pack thing... I'm sure I missed a ton of people but it's hopefully better than nothing?
go.bsky.app/Dyzvuzp
go.bsky.app/Dyzvuzp
August 13, 2024 at 7:32 PM
I sorta dug around and slapped together a data people starter pack thing... I'm sure I missed a ton of people but it's hopefully better than nothing?
go.bsky.app/Dyzvuzp
go.bsky.app/Dyzvuzp
Curious fact about Ronald Fisher, the father of modern statistics and experimental design: he refused to believe that smoking tobacco caused lung cancer 🚬🤯
Fisher argued that correlation does not imply causation and even published a paper in Nature 🙈📊
1/2
#statisics #abtesting #causalinference
Fisher argued that correlation does not imply causation and even published a paper in Nature 🙈📊
1/2
#statisics #abtesting #causalinference
August 12, 2024 at 5:08 AM
Curious fact about Ronald Fisher, the father of modern statistics and experimental design: he refused to believe that smoking tobacco caused lung cancer 🚬🤯
Fisher argued that correlation does not imply causation and even published a paper in Nature 🙈📊
1/2
#statisics #abtesting #causalinference
Fisher argued that correlation does not imply causation and even published a paper in Nature 🙈📊
1/2
#statisics #abtesting #causalinference
I'm excited to introduce tea-tasting, a Python package for the statistical analysis of A/B tests 📈🎉
It features Student's t-test, Bootstrap, variance reduction using CUPED, power analysis, and other statistical methods.
1/5
It features Student's t-test, Bootstrap, variance reduction using CUPED, power analysis, and other statistical methods.
1/5
July 29, 2024 at 6:59 AM
I'm excited to introduce tea-tasting, a Python package for the statistical analysis of A/B tests 📈🎉
It features Student's t-test, Bootstrap, variance reduction using CUPED, power analysis, and other statistical methods.
1/5
It features Student's t-test, Bootstrap, variance reduction using CUPED, power analysis, and other statistical methods.
1/5
I wrote two blog posts about ranking in a services marketplace, where customers can find professionals, and professionals can find customers.
In a two-sided matching platform, both sides have to choose each other for a deal to happen, and that makes the ranking problem harder and kind of unique 📈
In a two-sided matching platform, both sides have to choose each other for a deal to happen, and that makes the ranking problem harder and kind of unique 📈
January 15, 2024 at 1:41 PM
I wrote two blog posts about ranking in a services marketplace, where customers can find professionals, and professionals can find customers.
In a two-sided matching platform, both sides have to choose each other for a deal to happen, and that makes the ranking problem harder and kind of unique 📈
In a two-sided matching platform, both sides have to choose each other for a deal to happen, and that makes the ranking problem harder and kind of unique 📈
A/B testing on the "north star" metric is like gradient descent in reaching business optimum. And the significance level, alpha, is similar to the learning rate. 📈
The lower the alpha:
- The slower the convergence speed.
- The higher the risk of getting stuck in a local optimum.
1/3
The lower the alpha:
- The slower the convergence speed.
- The higher the risk of getting stuck in a local optimum.
1/3
December 26, 2023 at 7:44 AM
A/B testing on the "north star" metric is like gradient descent in reaching business optimum. And the significance level, alpha, is similar to the learning rate. 📈
The lower the alpha:
- The slower the convergence speed.
- The higher the risk of getting stuck in a local optimum.
1/3
The lower the alpha:
- The slower the convergence speed.
- The higher the risk of getting stuck in a local optimum.
1/3
Started working on tea-tasting, a Python package for statistical analysis of A/B tests 📊
Check out my blog post where I explain the motivation for creating this package and the benefits of the RDD approach: e10v.me/tea-tasting-...
Check out my blog post where I explain the motivation for creating this package and the benefits of the RDD approach: e10v.me/tea-tasting-...
Readme Driven Development for an A/B testing analysis package | Evgeny Ivanov
tea-tasting: a Python package for statistical analysis of A/B tests. Adopting the Readme Driven Development approach.
e10v.me
December 11, 2023 at 11:00 AM
Started working on tea-tasting, a Python package for statistical analysis of A/B tests 📊
Check out my blog post where I explain the motivation for creating this package and the benefits of the RDD approach: e10v.me/tea-tasting-...
Check out my blog post where I explain the motivation for creating this package and the benefits of the RDD approach: e10v.me/tea-tasting-...
The first rule of the Dunning-Kruger club... Wait, weren't Dunning and Kruger the founding members? And is there a right way to measure self-overestimation, anyway? 📊📈
Discover more: e10v.me/debunking-du...
Discover more: e10v.me/debunking-du...
December 3, 2023 at 7:41 AM
The first rule of the Dunning-Kruger club... Wait, weren't Dunning and Kruger the founding members? And is there a right way to measure self-overestimation, anyway? 📊📈
Discover more: e10v.me/debunking-du...
Discover more: e10v.me/debunking-du...