Go to file
Keyan ca11ac9fb8
backend payment optimism (#1195)
* wip backend optimism

* another inch

* make action state transitions only happen once

* another inch

* almost ready for testing

* use interactive txs

* another inch

* ready for basic testing

* lint fix

* inches

* wip item update

* get item update to work

* donate and downzap

* inchy inch

* fix territory paid actions

* wip usePaidMutation

* usePaidMutation error handling

* PENDING_HELD and HELD transitions, gql paidAction return types

* mostly working pessimism

* make sure invoice field is present in optimisticResponse

* inches

* show optimistic values to current me

* first pass at notifications and payment status reporting

* fix migration to have withdrawal hash

* reverse optimism on payment failure

* Revert "Optimistic updates via pending sats in item context (#1229)"

This reverts commit 93713b33df.

* add onCompleted to usePaidMutation

* onPaid and onPayError for new comments

* use 'IS DISTINCT FROM' for NULL invoiceActionState columns

* make usePaidMutation easier to read

* enhance invoice qr

* prevent actions on unpaid items

* allow navigation to action's invoice

* retry create item

* start edit window after item is paid for

* fix ux of retries from notifications

* refine retries

* fix optimistic downzaps

* remember item updates can't be retried

* store reference to action item in invoice

* remove invoice modal layout shift

* fix destructuring

* fix zap undos

* make sure ItemAct is paid in aggregate queries

* dont toast on long press zap undo

* fix delete and remindme bots

* optimistic poll votes with retries

* fix retry notifications and invoice item context

* fix pessimisitic typo

* item mentions and mention notifications

* dont show payment retry on item popover

* make bios work

* refactor paidAction transitions

* remove stray console.log

* restore docker compose nwc settings

* add new todos

* persist qr modal on post submission + unify item form submission

* fix post edit threshold

* make bounty payments work

* make job posting work

* remove more store procedure usage ... document serialization concerns

* dont use dynamic imports for paid action modules

* inline comment denormalization

* create item starts with median votes

* fix potential of serialization anomalies in zaps

* dont trigger notification indicator on successful paid action invoices

* ignore invoiceId on territory actions and add optimistic concurrency control

* begin docs for paid actions

* better error toasts and fix apollo cache warnings

* small documentation enhancements

* improve paid action docs

* optimistic concurrency control for territory updates

* use satsToMsats and msatsToSats helpers

* explictly type raw query template parameters

* improve consistency of nested relation names

* complete paid action docs

* useEffect for canEdit on payment

* make sure invoiceId is provided when required

* don't return null when expecting array

* remove buy credits

* move verifyPayment to paidAction

* fix comments invoicePaidAt time zone

* close nwc connections once

* grouped logs for paid actions

* stop invoiceWaitUntilPaid if not attempting to pay

* allow actionState to transition directly from HELD to PAID

* make paid mutation wait until pessimistic are fully paid

* change button text when form submits/pays

* pulsing form submit button

* ignore me in notification indicator for territory subscription

* filter unpaid items from more queries

* fix donation stike timing

* fix pending poll vote

* fix recent item notifcation padding

* no default form submitting button text

* don't show paying on submit button on free edits

* fix territory autorenew with fee credits

* reorg readme

* allow jobs to be editted forever

* fix image uploads

* more filter fixes for aggregate views

* finalize paid action invoice expirations

* remove unnecessary async

* keep clientside cache normal/consistent

* add more detail to paid action doc

* improve paid action table

* remove actionType guard

* fix top territories

* typo api/paidAction/README.md

Co-authored-by: ekzyis <ek@stacker.news>

* typo components/use-paid-mutation.js

Co-authored-by: ekzyis <ek@stacker.news>

* Apply suggestions from code review

Co-authored-by: ekzyis <ek@stacker.news>

* encorporate ek feeback

* more ek suggestions

* fix 'cost to post' hover on items

* Apply suggestions from code review

Co-authored-by: ekzyis <ek@stacker.news>

---------

Co-authored-by: ekzyis <ek@stacker.news>
2024-07-01 12:02:29 -05:00
.ebextensions install psql on servers 2024-01-30 18:47:01 -06:00
.github add shellcheck workflow (#1147) 2024-05-11 20:40:44 -05:00
.platform Remove Content-Length header for requests to /api/lnauth (#413) 2023-08-15 12:52:56 -05:00
.vscode remove vscode settings 2022-06-08 16:56:35 -05:00
api backend payment optimism (#1195) 2024-07-01 12:02:29 -05:00
capture Add 200ms delay to scroll in capture worker (#1088) 2024-04-18 10:49:42 -05:00
chat-web sndev matrix chat 2024-04-12 19:15:04 -05:00
components backend payment optimism (#1195) 2024-07-01 12:02:29 -05:00
copilot sndev matrix chat 2024-04-12 19:15:04 -05:00
docker Fix litd healthcheck (#1246) 2024-06-24 20:28:42 -05:00
docs time is a bitch doc 2024-04-02 09:29:56 -05:00
fragments backend payment optimism (#1195) 2024-07-01 12:02:29 -05:00
lib backend payment optimism (#1195) 2024-07-01 12:02:29 -05:00
pages backend payment optimism (#1195) 2024-07-01 12:02:29 -05:00
prisma backend payment optimism (#1195) 2024-07-01 12:02:29 -05:00
public Add security.txt (#912) 2024-03-13 16:04:28 -05:00
scripts add this day in stacker news to newsletter 2024-06-22 15:23:30 -05:00
styles Embed Rumble Video (#1191) 2024-05-28 08:18:32 -05:00
svgs enhance navigation 2024-03-26 18:36:31 -05:00
sw Item mention notifications (#1208) 2024-06-03 12:12:42 -05:00
worker backend payment optimism (#1195) 2024-07-01 12:02:29 -05:00
.env.development update lnbits dev port to not conflict on mac 2024-05-19 15:54:32 -05:00
.env.production Ignore invalid certificate over Tor for CLN autowithdrawals (#1165) 2024-05-09 21:07:47 -05:00
.gitattributes Remove leading dash to opt-in to being treated as vendor code (#942) 2024-03-20 11:13:22 -05:00
.gitignore Add script to use NWC with stacker lnd (#1151) 2024-05-06 15:05:06 -05:00
.npmrc Revert "Revert "shield your eyes; massive, squashed refactor; nextjs/react/react-dom/apollo upgrades"" 2023-07-23 10:08:43 -05:00
Dockerfile add sndev shell script and enhance docker compose local dev 2024-03-13 09:04:09 -05:00
LICENSE add license 2021-06-28 18:28:26 -05:00
Procfile fix module issues when running worker 2023-09-26 20:33:54 -05:00
README.md refine readme a bit 2024-04-14 18:02:13 -05:00
awards.csv Update awards.csv 2024-06-25 16:36:20 -05:00
contributors.txt Force post options open when dirty or on errors (#1129) 2024-05-12 16:17:41 -05:00
docker-compose.yml Fix litd healthcheck (#1246) 2024-06-24 20:28:42 -05:00
jsconfig.json Use module path aliases (#938) 2024-03-19 19:37:31 -05:00
middleware.js Handle Peertube Embeds (#1223) 2024-06-20 11:28:25 -05:00
next.config.js define new env vars in service worker 2024-04-08 17:55:29 -05:00
package-lock.json backend payment optimism (#1195) 2024-07-01 12:02:29 -05:00
package.json backend payment optimism (#1195) 2024-07-01 12:02:29 -05:00
sndev give sndev logs better default params 2024-06-04 13:07:03 -05:00

README.md

Internet Communities with Bitcoin Economies

  • Stacker News makes internet communities that pay you Bitcoin
  • What You See is What We Ship (look ma, I invented an initialism)
  • 100% FOSS
  • We pay bitcoin for PRs, issues, documentation, code reviews and more
  • Next.js, postgres, graphql, and lnd

Getting started

Launch a fully featured SN development environment in a single command.

$ ./sndev start

Go to localhost:3000.


Installation

  • Clone the repo
    • ssh: git clone git@github.com:stackernews/stacker.news.git
    • https: git clone https://github.com/stackernews/stacker.news.git
  • Install docker

Usage

Start the development environment

$ ./sndev start

View all available commands

$ ./sndev help

                            888
                            888
                            888
      .d8888b  88888b.  .d88888  .d88b.  888  888
     88K      888 '88b d88' 888 d8P  Y8b 888  888
     'Y8888b. 888  888 888  888 88888888 Y88  88P
          X88 888  888 Y88b 888 Y8b.      Y8bd8P
      88888P' 888  888  'Y88888  'Y8888    Y88P

manages a docker based stacker news development environment

USAGE
  $ sndev [COMMAND]
  $ sndev help [COMMAND]

COMMANDS
  help            show help

  env:
    start         start env
    stop          stop env
    restart       restart env
    status        status of env
    logs          logs from env
    delete        delete env

  sn:
    login         login as a nym

  lnd:
    fund          pay a bolt11 for funding
    withdraw      create a bolt11 for withdrawal

  cln:
    cln_fund      pay a bolt11 for funding with CLN
    cln_withdraw  create a bolt11 for withdrawal with CLN

  db:
    psql          open psql on db
    prisma        run prisma commands

  dev:
    pr            fetch and checkout a pr
    lint          run linters

  other:
    compose         docker compose passthrough
    sn_lndcli       lncli passthrough on sn_lnd
    stacker_lndcli  lncli passthrough on stacker_lnd
    stacker_clncli  lightning-cli passthrough on stacker_cln

Modifying services

Running specific services

By default all services will be run. If you want to exclude specific services from running, set COMPOSE_PROFILES to use one or more of minimal|images|search|payments|email|capture. To only run mininal services without images, search, or payments:

$ COMPOSE_PROFILES=minimal ./sndev start

Or, as I would recommend:

$ export COMPOSE_PROFILES=minimal
$ ./sndev start

To run with images and payments services:

$ COMPOSE_PROFILES=images,payments ./sndev start

Merging compose files

By default sndev start will merge docker-compose.yml with docker-compose.override.yml. Specify any overrides you want to merge with docker-compose.override.yml.

For example, if you want to replace the db seed with a custom seed file located in docker/db/another.sql, you'd create a docker-compose.override.yml file with the following:

services:
  db:
    volumes:
      - ./docker/db/another.sql:/docker-entrypoint-initdb.d/seed.sql

You can read more about docker compose override files.


Table of Contents


Contributing

We want your help.


We pay bitcoin for contributions

  • pull requests closing existing issues
  • code review
  • issue specification whether for bugs, features, or enhancements
  • discovery of security vulnerabilities
  • discovery of privacy vulnerabilities
  • improvements to development documentation
  • helpfulness

View a current list of granted awards


Just in case

This document in no way legally entitles you to payments for contributions, entitles you to being a contributor, or entitles you to the attention of other contributors. This document lays out the system we can use to determine contribution awards which we generally intend to abide by but importantly we reserve the right to refuse payments or contributions, modify rules and award amounts, make exceptions to rules or reward amounts, and withhold awards for any reason at anytime, even just for the heck of it, at our sole discretion. If you need more certainty than what I've just described, don't participate. We provide awards as an experiment to make FOSS less sucky.


Pull request awards

Rules

  1. PRs closing an issue will be awarded according to the difficulty tag on an issue, e.g. difficulty:easy pays 100k sats.
  2. Issues are occasionally marked with a priority tag which multiplies the award of a PR closing an issue, e.g. an issue marked with priority:high and difficulty:hard awards 2m sats.
  3. An award is reduced by 10% of the award amount for each substantial change requested to the PR on code review, e.g. if two changes are requested on a PR closing an issue tagged with difficulty:hard, 800k sats will be awarded.
    • Reductions are applied before priority multipliers, e.g. a PR closing a priority:high and difficulty:hard issue that's approved after two changes are requested awards 1.6m sats.
    • You are responsible for understanding the issue and requirements before requesting review on a PR.
    • There is no award reduction for asking specific questions on the issue itself or on the PR before requesting review
    • There is no award reduction for asking more general questions in a discussion
  4. A PR must be merged by an SN engineer before a PR receives an award

Due to Rule 3, make sure that you mark your PR as a draft when you create it and it's not ready for review yet.

Difficulty award amounts

tag description award
difficulty:good-first-issue at most a couple lines of code in a couple files and does not require much familiarity with the codebase 20k sats
difficulty:easy at most a couple lines of code in a couple files but does require familiarity with the code base 100k sats
difficulty:medium more code, more places and could require adding columns in the db and some modification chunky db queries 250k sats
difficulty:medium-hard even more code, even more places and requires either more sophisticated logic, more significant db modeling eg adding a table, and/or a deeper study of a something 500k sats
difficulty:hard either a bigger lift than the what's required of medium-hard or very tricky in a particular way that might not require a lot of code but does require a lot of context/troubleshooting/expertise 1m sats

Priority multipliers

tag multiplier
priority:medium 1.5
priority:high 2
priority:urgent 3

Requesting modifications to reward amounts

We try to assign difficulty and priority tags to issues accurately, but we're not perfect. If you believe an issue is mis-tagged, you can request a change to the issue's tags.


Code review awards

Code reviewers will be awarded the amount their code review reduced from the PR author's reward, e.g. two substantial problems/areas of improvement identified in a PR closing a priority:high and difficulty:hard issue awards 400k sats.

Rules

  1. The problem or improvement must be acknowledged as such by SN engineers explicitly
  2. A PR must be merged by an SN engineer before a PR's code reviewers receive an award

Code review approvals are more than welcome, but we can't guarantee awards for them because the work performed to approve a PR is unverifiable.


Issue specification awards

Issue specifiers will be awarded up to 10% of a PR award for issues resulting in a PR being merged by an SN engineer that closes the issue. In addition to being subject to PR award amounts and reductions, specification amounts are awarded on the basis of how much additional help and specification is required by other contributors.

Rules

  1. The issue must directly result in PR being merged by an SN engineer that closes the issue
  2. Issue specification award amounts are based on the final PR award amounts
    • that is, they are subject to PR award code review reductions and priority multipliers
  3. Award amounts will be reduced on the basis of how much additional help and specification is required by other contributors
  4. Issue specifiers who can close their own issues with their own PRs are also eligible for this 10%
    • e.g an issue tagged as difficulty:hard that is both specified and closed by a PR from the same contributor without changes requested awards 1.1m sats

Relative awards

circumstances award
issue doesn't require further help and/or specification from other contributors 10%
issue requires little help and/or specification from other contributors 5%
issue requires more help and/or specification from other contributors than the issue specifier contributed 1%
issue is vague and/or incomplete and must mostly be entirely specified by someone else 0%

For example: a specified issue that's tagged as difficulty:hard, doesn't require additional specification and disambiguation by other contributors, and results in PR being merged without changes requested awards the issue specifier 100k sats.


Responsible disclosure of security or privacy vulnerability awards

Awards for responsible disclosures are assessed on the basis of:

  1. the potential loss resulting from an exploit of the vulnerability
  2. the trivialness of exploiting the vulnerability
  3. the disclosure's detail

Award amounts will be easiest to assess on a case by case basis. Upon confirmation of a vulnerability, we agree to award responsible disclosures at minimum 100k sats and as high as the total potential loss that would result from exploiting the vulnerability.

Rules

  1. Disclosure is responsible and does not increase the likelihood of an exploit.
  2. Disclosure includes steps to reproduce.
  3. Disclosure includes a realistic attack scenario with prerequisites for an attack and expected gains after the exploitation. Disclosures without such scenario, with unrealistic assumptions or without meaningful outcomes will not be eligible for awards.
  4. You must be the first person to responsibly disclose the issue to be eligible for awards.

Development documentation awards

For significant changes to documentation, create an issue before making said changes. In such cases we will award documentation improvements in accordance with issue specification and PR awards.

For changes on the order of something like a typo, we'll award a nominal amount at our discretion.


Helpfulness awards

Like issue specification awards, helping fellow contributors substantially in a well documented manner such that the helped fellow contributes a merged PR is eligible for a one-time relative reward.

circumstances award
substantial and singular source of help 10%
substantial but nonsingular source of help 1-5%
source of relatively trivial help 1%

Contribution extras

We want to make contributing to SN as rewarding as possible, so we offer a few extras to contributors.

Dev chat

We self-host a private chat server for contributors to SN. If you'd like to join, please respond in this discussion.

Triage permissions

We offer triage permissions to contributors after they've made a few contributions. I'll usually add them as I notice people contributing, but if I missed you and you'd like to be added, let me know!

Contributor badges on SN profiles

Contributors can get badges on their SN profiles by opening a pull request adding their SN nym to the contributors.txt file.

What else you got

In the future we plan to offer more, like gratis github copilot subscriptions, reverse tunnels, codespaces, and merch.

If you'd like to see something added, please make a suggestion.


Development Tips


Linting

We use JavaScript Standard Style to enforce code style and correctness. You should run sndev lint before submitting a PR.

If you're using VSCode, you can install the StandardJS VSCode Extension extension to get linting in your editor. We also recommend installing StandardJS code snippets and StandardJS react code snippets for code snippets.


Database migrations

We use prisma for our database migrations. To create a new migration, modify prisma/schema.prisma according to prisma schema reference and apply it with:

./sndev prisma migrate dev

If you want to create a migration without applying it, eg to create a trigger or modify the generated sql before applying, use the --create-only option:

./sndev prisma migrate dev --create-only

Generate the local copy of the prisma ORM client in node_modules after changes. This should only be needed to get Intellisense in your editor locally.

./sndev prisma generate


Connecting to the local database

You can connect to the local database via ./sndev psql. psql is an interactive terminal for working with PostgreSQL.


Running lncli on the local lnd nodes

You can run lncli on the local lnd nodes via ./sndev sn_lncli and ./sndev stacker_lncli. The node for your local SN instance is sn_lnd and the node serving as any external node, like a stacker's node or external wallet, is stacker_lnd.


Testing local auth

You can login to test features like posting, replying, tipping, etc with ./sndev login <nym> which will provide a link to login as an existing nym or a new account for a nonexistent nym. But, it you want to test auth specifically you'll need to configure them in your .env file.

Login with Email

MailHog

  • The app is already prepared to send emails through MailHog so no extra configuration is needed
  • Click "sign up" and enter any email address (remember, it's not going anywhere beyond your workstation)
  • Access MailHog's web UI on http://localhost:8025
  • Click the link (looks like this):
http://localhost:3000/api/auth/callback/email?email=satoshi%40gmail.com&token=110e30a954ce7ca643379d90eb511640733de405f34a31b38eeda8e254d48cd7

Sendgrid

  • Create a Sendgrid account (or other smtp service)
LOGIN_EMAIL_SERVER=smtp://apikey:<sendgrid_api_key>@smtp.sendgrid.net:587
LOGIN_EMAIL_FROM=<sendgrid_email_from>
  • Click "sign up" and enter your email address
  • Check your email
  • Click the link (looks like this):
http://localhost:3000/api/auth/callback/email?email=satoshi%40gmail.com&token=110e30a954ce7ca643379d90eb511640733de405f34a31b38eeda8e254d48cd7

Login with Github

  • Create a new OAuth app in your Github account
    • Set the callback URL to: http://localhost:3000/api/auth/callback/github
  • Update your .env file
GITHUB_ID=<Client ID>
GITHUB_SECRET=<Client secret>
  • Signup and login as above

Login with Lightning

  • Use ngrok to create a HTTPS tunnel to localhost:3000
  • Update LNAUTH_URL in .env with the URL provided by ngrok and add /api/lnauth to it

Enabling web push notifications

To enable Web Push locally, you will need to set the VAPID_* env vars. VAPID_MAILTO needs to be an email address using the mailto: scheme. For NEXT_PUBLIC_VAPID_KEY and VAPID_PRIVKEY, you can run npx web-push generate-vapid-keys.


Internals


Stack

The site is written in javascript (not typescript 😱) using Next.js, a React framework. The backend API is provided via GraphQL. The database is PostgreSQL modeled with Prisma. The job queue is also maintained in PostgreSQL. We use lnd for our lightning node. A customized Bootstrap theme is used for styling.


Services

Currently, SN runs and maintains two significant services and one microservice:

  1. the nextjs web app, found in ./
  2. the worker service, found in ./worker, which runs periodic jobs and jobs sent to it by the web app
  3. a screenshot microservice, found in ./capture, which takes screenshots of SN for social previews

In addition, we run other critical services the above services interact with like lnd, postgres, opensearch, tor, and s3.


Wallet transaction safety

To ensure stackers balances are kept sane, all wallet updates are run in serializable transactions at the database level. Because early versions of prisma had relatively poor support for transactions most wallet touching code is written in plpgsql stored procedures and can be found in the prisma/migrations folder.


Need help?

Open a discussion or issue or email us or request joining the dev chat.


Responsible disclosure

If you found a vulnerability, we would greatly appreciate it if you contact us via security@stacker.news or open a security advisory. Our PGP key can be found here (EBAF 75DA 7279 CB48).


License

MIT