h/t to @official-techsupport for finding and help fixing this bug.
When given certain pathological input, `sanitize` would time out
(notably only on posts, rather than comments, perhaps due to the
longer maximum length of input). For example, using as input the
result of:
with open("test.txt", "w") as f:
for i in range(26):
f.write(f":{chr(ord('a') + i)}: ")
f.write('x' * 20_000)
We believe this to be because of some combination of the greedy
quantifiers and the negative lookahead before the match. The regex
was rewritten to (in theory) have much more linear performance.
This bug was discovered when lottery.check_if_end_lottery_task was
failing due to a stack trace thru end_lottery session < badge_grant
< send_repeatable_notifications < sanitize L208. In particular, when
`flask cron` (helpers/cron.py) executes, it does not set g.v, whereas
this code previously assumed that g.v : (None + User) and did not
check for its presence.
These are all minor and uncontroversial enough it just felt gross
making multiple commits.
- Adds marseymummified.
- Changes the new `raise ValueError(...)` in badge_grant to
an `assert`.
- Expands assetcache to a convenient grab bag of JS files.
First, the apparent errors with >= 9 and 99 in the Marseys and
referrals code after the refactor are not actually bugs: they were
bug fixes mixed in with the refactor to fix an off-by-one.
Next, instead of failing silently on the `not user` branch in
badge_grant, we throw a ValueError. This retains the current
behavior where users get 500s to report while also enforcing the
assertion near the edge of the function.
Despite being very fun, this fixes the recently discovered bug where
placing '#' or '!' within the 'pat:' suffix of a patted emoji causes
the enclosing <span> to not be given the proper CSS `display` or
`position`, leading to the hand being sized relative to the comment
bounding box rather than the emoji box.
This should be backward compatible. The only posts it wont fix are
existing ones with the giant hands. Main example being:
https://rdrama.net/h/slackernews/post/76302/
grant_lottery_tickets_to_user incorrectly deposited the full ticket
value into manager account, not just the net value.
Additionally, royalty rate has been set to zero because Chapose won
the first lottery and grifting 8% that could instead go into the
prize pool seems unwarranted given that.
Prior to this comment, the every-1d cron.py command was broken due
to lack of proper stats import. However, while refactoring this, it
was convenient to move other recurring tasks that had been stuffed in
odd places--not least `stats(...)`--into the new cron system. This
entailed a number of refactorings of other things.
1. Move stats(...) from static.py to helpers/stats.py.
2. Move hole inactivity purge task from stats(...) to routes/subs.py.
3. Move bot award timer checks from stats(...) to helpers/awards.py.
4. Unify award timer logic formerly in routes/front.py into the new
helpers/awards.py.
h/t @official-techsupport again for finding another optimization.
We are now cumulatively at about 70% speedup over original.
It remains one of the hottest paths of the codebase in relative
terms, but its absolute performance demands have decreased enough
to buy us potentially substantial time on it.
h/t @official-techsupport for digging into the regex performance and
coming up with one that greatly reduces backtracking. We see an
approximately 2x speedup under typical loads, which proves to be a
major overall savings in performance. Previously, censor_slurs was,
second to ORM DB accesses, by far the most time-consuming function
in the codebase under typical loads. It's still not ideal, but it is
much better.
Future options to improve this critical path further would be:
1) Precompute a slur-replaced HTML, rather than recomputing
each pageload. Storage is cheap.
2) Tokenize the HTML and replace plaintext words using O(1)
exact-match lookups to a dict.
The reddit mentions system contained much duplicated code and was
grafted onto the post thumbnail pipeline to achieve semi-regular
invocation. Instead, we now run it through the new cron system,
and the duplicate code has been refactored out.
Adding an empty __init__.py, the imports-only cli.py, and setting
FLASK_APP in the environment are enough to get the `flask` command
to work. This will enable future changes, including database
migrations.
The proximate reason for the fix is to add a `flask cron` command
to run scheduled tasks within the application from cron. Specifically,
the lottery should be run from cron.
It was observed in prod that the lottery prize as tracked by the DB
had diverged from the amount held in the Lottershe manager account.
This appears to be the result of grant_lottery_tickets_to_user
adding the # of _tickets_ rather than the value of those tickets to
the manager.
- Increment cache version on popover badges.
- Add comments+submission_listing.js to assetcache to support ^.
- Append new words to wordle list.
- Cache bust assorted assets for recent PRs.
In general, we don't do a great job of length validating body_html
fields. Lots of ways to get 500 errors by providing too long of
input. Really ought to find a way to fix it in the classes/comment.py
and classes/submission.py classes. In the interim, the recent gifts
messages change is salient because the notification can 500 out
mid-way through performing coin transactions.
Recommended to find a better way of truncating or safely bubbling
the exception up. Truncating probably not best long-term solution
because it could hypothetically permit strings that would otherwise
be considered unsanitized.
A rare case where users receive 0 lotto tickets from a treasure chest
occurs when they received 10 or 11 coins from a chest pre-conversion
to lotto tickets. Rather than change ticket_count to the ceil of
dividing coins by ticket cost, it seems less distortionary to instead
imperceptibly raise the minimum to avoid this case.
Due to presently hitting perpetual 429s after a mishap with lottery
polling on production, among past events where admins have gotten
rate-limited for doing otherwise normal admin behavior, the
flask_limiter.Limiter now has a request filter to whitelist JL2+.
Despite running on every request, I don't anticipate this undermining
the DoS prevention power of the Limiter.
It is yet unknown whether there are edge cases where running
get_logged_in_user in a different spot in the request pipeline might
e.g. subtly break the logged-in counters. This is not expected at
present, however.
Users now have a toggleable can_gamble setting which disables their
ability to use all chance-based gains on the site: viz. slots,
blackjack, the lottery, and treasure chests.
This only applies on invocation of commands that start gambling
games, so it should cause no bugs when toggled with e.g. active
blackjack games.
This was added for the benefit of users with actual problems with
gambling, be they past addiction or religious conviction. All future
gambling features are humbly requested to respect it.