stacker.news/worker/index.js

143 lines
5.3 KiB
JavaScript
Raw Normal View History

Convert worker to ESM (#500) * Convert worker to ESM To use ESM for the worker, I created a package.json file in worker/ with `{ type: "module" }` as its sole content. I then rewrote every import to use ESM syntax. I also tried to set `{ type: "module" }` in the root package.json file to also use ESM in next.config.js. However, this resulted in a weird problem: default imports were now getting imported as objects in this shape: `{ default: <defaultImport> }`. Afaik, this should only be the case if you use "import * as foo from 'bar'" syntax: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import#default_import This is fixed by not using `{ type: "module" }` for some reason. However, then, next.config.js also doesn't support ESM import syntax anymore. The documentation says that if you want to use ESM, you can use next.config.mjs: https://nextjs.org/docs/pages/api-reference/next-config-js But I didn't want to use MJS extension since I don't have any experience with it. For example, not sure if it's good style to mix JS with MJS etc. So I kept the CJS import syntax there. * Ignore worker/ during linting I wasn't able to fix the following errors: /home/runner/work/stacker.news/stacker.news/worker/auction.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/auction.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/earn.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/earn.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/index.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/index.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/nostr.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/nostr.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/ots.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/ots.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/repin.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/repin.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/search.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/search.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/streak.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/streak.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/trust.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/trust.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/views.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/views.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/wallet.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/wallet.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) I tried to tell babel where to find the babel config file (.babelrc), specifying the babel config in worker/package.json under "babel", using babel.config.json etc. to no avail. However, afaict, we don't need babel for the worker since it won't run in a browser. Babel is only used to transpile code to target browsers. But it still would be nice to lint the worker code with standard. But we can figure this out later. * Fix worker imports from lib/ and api/ This fixes the issue that we can't use `{ "type": "module" }` in the root package.json since it breaks the app with this error: app | TypeError: next_auth_providers_credentials__WEBPACK_IMPORTED_MODULE_2__ is not a function app | at eval (webpack-internal:///./pages/api/auth/[...nextauth].js:218:20) app | at process.processTicksAndRejections (node:internal/process/task_queues:95:5) app | LND GRPC connection successful app | - error pages/api/auth/[...nextauth].js (139:2) @ CredentialsProvider app | - error Error [TypeError]: next_auth_providers_credentials__WEBPACK_IMPORTED_MODULE_2__ is not a function app | at eval (webpack-internal:///./pages/api/auth/[...nextauth].js:218:20) { app | digest: undefined app | } app | 137 | app | 138 | const providers = [ app | > 139 | CredentialsProvider({ app | | ^ app | 140 | id: 'lightning', app | 141 | name: 'Lightning', app | 142 | credentials: { app | TypeError: next_auth_providers_credentials__WEBPACK_IMPORTED_MODULE_2__ is not a function app | at eval (webpack-internal:///./pages/api/auth/[...nextauth].js:218:20) app | at process.processTicksAndRejections (node:internal/process/task_queues:95:5) build but we need to tell the worker that the files are MJS, else we get this error: worker | file:///app/worker/wallet.js:3 worker | import { datePivot } from '../lib/time.js' worker | ^^^^^^^^^ worker | SyntaxError: Named export 'datePivot' not found. The requested module '../lib/time.js' is a CommonJS module, which may not support all module.exports as named exports. worker | CommonJS modules can always be imported via the default export, for example using: worker | worker | import pkg from '../lib/time.js'; worker | const { datePivot } = pkg; worker | worker | at ModuleJob._instantiate (node:internal/modules/esm/module_job:124:21) worker | at async ModuleJob.run (node:internal/modules/esm/module_job:190:5) worker | worker | Node.js v18.17.0 worker | worker exited with code 1 * Fix global not defined in browser context * Also ignore api/ and lib/ during linting I did not want to do this but I was not able to fix this error in any other way I tried: /home/ekzyis/programming/stacker.news/api/lnd/index.js:0:0: Parsing error: No Babel config file detected for /home/ekzyis/programming/stacker.news/api/lnd/index.js. Either disable config file checking with requ ireConfigFile: false, or configure Babel so that it can find the config files. (null) Did not want to look deeper into all this standard, eslint, babel configuration stuff ... --------- Co-authored-by: ekzyis <ek@stacker.news> Co-authored-by: Keyan <34140557+huumn@users.noreply.github.com>
2023-09-24 01:19:35 +00:00
import PgBoss from 'pg-boss'
import nextEnv from '@next/env'
import createPrisma from '@/lib/create-prisma.js'
2024-01-07 17:00:24 +00:00
import {
backend payment optimism (#1195) * wip backend optimism * another inch * make action state transitions only happen once * another inch * almost ready for testing * use interactive txs * another inch * ready for basic testing * lint fix * inches * wip item update * get item update to work * donate and downzap * inchy inch * fix territory paid actions * wip usePaidMutation * usePaidMutation error handling * PENDING_HELD and HELD transitions, gql paidAction return types * mostly working pessimism * make sure invoice field is present in optimisticResponse * inches * show optimistic values to current me * first pass at notifications and payment status reporting * fix migration to have withdrawal hash * reverse optimism on payment failure * Revert "Optimistic updates via pending sats in item context (#1229)" This reverts commit 93713b33df9bc3701dc5a692b86a04ff64e8cfb1. * add onCompleted to usePaidMutation * onPaid and onPayError for new comments * use 'IS DISTINCT FROM' for NULL invoiceActionState columns * make usePaidMutation easier to read * enhance invoice qr * prevent actions on unpaid items * allow navigation to action's invoice * retry create item * start edit window after item is paid for * fix ux of retries from notifications * refine retries * fix optimistic downzaps * remember item updates can't be retried * store reference to action item in invoice * remove invoice modal layout shift * fix destructuring * fix zap undos * make sure ItemAct is paid in aggregate queries * dont toast on long press zap undo * fix delete and remindme bots * optimistic poll votes with retries * fix retry notifications and invoice item context * fix pessimisitic typo * item mentions and mention notifications * dont show payment retry on item popover * make bios work * refactor paidAction transitions * remove stray console.log * restore docker compose nwc settings * add new todos * persist qr modal on post submission + unify item form submission * fix post edit threshold * make bounty payments work * make job posting work * remove more store procedure usage ... document serialization concerns * dont use dynamic imports for paid action modules * inline comment denormalization * create item starts with median votes * fix potential of serialization anomalies in zaps * dont trigger notification indicator on successful paid action invoices * ignore invoiceId on territory actions and add optimistic concurrency control * begin docs for paid actions * better error toasts and fix apollo cache warnings * small documentation enhancements * improve paid action docs * optimistic concurrency control for territory updates * use satsToMsats and msatsToSats helpers * explictly type raw query template parameters * improve consistency of nested relation names * complete paid action docs * useEffect for canEdit on payment * make sure invoiceId is provided when required * don't return null when expecting array * remove buy credits * move verifyPayment to paidAction * fix comments invoicePaidAt time zone * close nwc connections once * grouped logs for paid actions * stop invoiceWaitUntilPaid if not attempting to pay * allow actionState to transition directly from HELD to PAID * make paid mutation wait until pessimistic are fully paid * change button text when form submits/pays * pulsing form submit button * ignore me in notification indicator for territory subscription * filter unpaid items from more queries * fix donation stike timing * fix pending poll vote * fix recent item notifcation padding * no default form submitting button text * don't show paying on submit button on free edits * fix territory autorenew with fee credits * reorg readme * allow jobs to be editted forever * fix image uploads * more filter fixes for aggregate views * finalize paid action invoice expirations * remove unnecessary async * keep clientside cache normal/consistent * add more detail to paid action doc * improve paid action table * remove actionType guard * fix top territories * typo api/paidAction/README.md Co-authored-by: ekzyis <ek@stacker.news> * typo components/use-paid-mutation.js Co-authored-by: ekzyis <ek@stacker.news> * Apply suggestions from code review Co-authored-by: ekzyis <ek@stacker.news> * encorporate ek feeback * more ek suggestions * fix 'cost to post' hover on items * Apply suggestions from code review Co-authored-by: ekzyis <ek@stacker.news> --------- Co-authored-by: ekzyis <ek@stacker.news>
2024-07-01 17:02:29 +00:00
autoDropBolt11s, checkInvoice, checkPendingDeposits, checkPendingWithdrawals,
not-custodial zap beta (#1178) * not-custodial zap scaffolding * invoice forward state machine * small refinements to state machine * make wrap invoice work * get state machine working end to end * untested logic layout for paidAction invoice wraps * perform pessimisitic actions before outgoing payment * working end to end * remove unneeded params from wallets/server/createInvoice * fix cltv relative/absolute confusion + cancelling forwards * small refinements * add p2p wrap info to paidAction docs * fallback to SN invoice when wrap fails * fix paidAction retry description * consistent naming scheme for state machine * refinements * have sn pay bounded outbound fee * remove debug logging * reenable lnc permissions checks * don't p2p zap on item forward splits * make createInvoice params json encodeable * direct -> p2p badge on notifications * allow no tls in dev for core lightning * fix autowithdraw to create invoice with msats * fix autowithdraw msats/sats inconsitency * label p2p zaps properly in satistics * add fees to autowithdrawal notifications * add RETRYING as terminal paid action state * Update api/paidAction/README.md Co-authored-by: ekzyis <ek@stacker.news> * Update api/paidAction/README.md Co-authored-by: ekzyis <ek@stacker.news> * Update api/lnd/index.js Co-authored-by: ekzyis <ek@stacker.news> * ek suggestions * add bugetable to nwc card * get paranoid with numbers * better finalize retries and better max timeout height * refine forward failure transitions * more accurate satistics p2p status * make sure paidaction cancel in state machine only * dont drop bolt11s unless status is not null * only allow PENDING_HELD to transition to FORWARDING * add mermaid state machine diagrams to paid action doc * fix cancel transition name * cleanup readme * move forwarding outside of transition * refine testServerConnect and make sure ensureB64 transforms * remove unused params from testServerConnect --------- Co-authored-by: ekzyis <ek@stacker.news> Co-authored-by: k00b <k00b@stacker.news>
2024-08-13 14:48:30 +00:00
checkWithdrawal,
finalizeHodlInvoice, subscribeToWallet
2024-01-07 17:00:24 +00:00
} from './wallet.js'
Convert worker to ESM (#500) * Convert worker to ESM To use ESM for the worker, I created a package.json file in worker/ with `{ type: "module" }` as its sole content. I then rewrote every import to use ESM syntax. I also tried to set `{ type: "module" }` in the root package.json file to also use ESM in next.config.js. However, this resulted in a weird problem: default imports were now getting imported as objects in this shape: `{ default: <defaultImport> }`. Afaik, this should only be the case if you use "import * as foo from 'bar'" syntax: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import#default_import This is fixed by not using `{ type: "module" }` for some reason. However, then, next.config.js also doesn't support ESM import syntax anymore. The documentation says that if you want to use ESM, you can use next.config.mjs: https://nextjs.org/docs/pages/api-reference/next-config-js But I didn't want to use MJS extension since I don't have any experience with it. For example, not sure if it's good style to mix JS with MJS etc. So I kept the CJS import syntax there. * Ignore worker/ during linting I wasn't able to fix the following errors: /home/runner/work/stacker.news/stacker.news/worker/auction.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/auction.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/earn.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/earn.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/index.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/index.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/nostr.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/nostr.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/ots.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/ots.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/repin.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/repin.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/search.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/search.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/streak.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/streak.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/trust.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/trust.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/views.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/views.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/wallet.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/wallet.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) I tried to tell babel where to find the babel config file (.babelrc), specifying the babel config in worker/package.json under "babel", using babel.config.json etc. to no avail. However, afaict, we don't need babel for the worker since it won't run in a browser. Babel is only used to transpile code to target browsers. But it still would be nice to lint the worker code with standard. But we can figure this out later. * Fix worker imports from lib/ and api/ This fixes the issue that we can't use `{ "type": "module" }` in the root package.json since it breaks the app with this error: app | TypeError: next_auth_providers_credentials__WEBPACK_IMPORTED_MODULE_2__ is not a function app | at eval (webpack-internal:///./pages/api/auth/[...nextauth].js:218:20) app | at process.processTicksAndRejections (node:internal/process/task_queues:95:5) app | LND GRPC connection successful app | - error pages/api/auth/[...nextauth].js (139:2) @ CredentialsProvider app | - error Error [TypeError]: next_auth_providers_credentials__WEBPACK_IMPORTED_MODULE_2__ is not a function app | at eval (webpack-internal:///./pages/api/auth/[...nextauth].js:218:20) { app | digest: undefined app | } app | 137 | app | 138 | const providers = [ app | > 139 | CredentialsProvider({ app | | ^ app | 140 | id: 'lightning', app | 141 | name: 'Lightning', app | 142 | credentials: { app | TypeError: next_auth_providers_credentials__WEBPACK_IMPORTED_MODULE_2__ is not a function app | at eval (webpack-internal:///./pages/api/auth/[...nextauth].js:218:20) app | at process.processTicksAndRejections (node:internal/process/task_queues:95:5) build but we need to tell the worker that the files are MJS, else we get this error: worker | file:///app/worker/wallet.js:3 worker | import { datePivot } from '../lib/time.js' worker | ^^^^^^^^^ worker | SyntaxError: Named export 'datePivot' not found. The requested module '../lib/time.js' is a CommonJS module, which may not support all module.exports as named exports. worker | CommonJS modules can always be imported via the default export, for example using: worker | worker | import pkg from '../lib/time.js'; worker | const { datePivot } = pkg; worker | worker | at ModuleJob._instantiate (node:internal/modules/esm/module_job:124:21) worker | at async ModuleJob.run (node:internal/modules/esm/module_job:190:5) worker | worker | Node.js v18.17.0 worker | worker exited with code 1 * Fix global not defined in browser context * Also ignore api/ and lib/ during linting I did not want to do this but I was not able to fix this error in any other way I tried: /home/ekzyis/programming/stacker.news/api/lnd/index.js:0:0: Parsing error: No Babel config file detected for /home/ekzyis/programming/stacker.news/api/lnd/index.js. Either disable config file checking with requ ireConfigFile: false, or configure Babel so that it can find the config files. (null) Did not want to look deeper into all this standard, eslint, babel configuration stuff ... --------- Co-authored-by: ekzyis <ek@stacker.news> Co-authored-by: Keyan <34140557+huumn@users.noreply.github.com>
2023-09-24 01:19:35 +00:00
import { repin } from './repin.js'
import { trust } from './trust.js'
import { auction } from './auction.js'
import { earn } from './earn.js'
import apolloClient from '@apollo/client'
import { indexItem, indexAllItems } from './search.js'
import { timestampItem } from './ots.js'
import { computeStreaks, checkStreak } from './streak.js'
import { nip57 } from './nostr.js'
import fetch from 'cross-fetch'
import { authenticatedLndGrpc } from 'ln-service'
import { views, rankViews } from './views.js'
2023-10-01 23:03:52 +00:00
import { imgproxy } from './imgproxy.js'
import { deleteItem } from './ephemeralItems.js'
Image uploads (#576) * Add icon to add images * Open file explorer to select image * Upload images to S3 on selection * Show uploaded images below text input * Link and remove image * Fetch unsubmitted images from database * Mark S3 images as submitted in imgproxy job * Add margin-top * Mark images as submitted on client after successful mutation * Also delete objects in S3 * Allow items to have multiple uploads linked * Overwrite old avatar * Add fees for presigned URLs * Use Github style upload * removed upfront fees * removed images provider since we no longer need to keep track of unsubmitted images on the client * removed User.images resolver * removed deleteImage mutation * use Github style upload where it shows ![Uploading <filename>...]() first and then replaces that with ![<filename>](<url>) after successful upload * Add Upload.paid boolean column One item can have multiple images linked to it, but an image can also be used in multiple items (many-to-many relation). Since we don't really care to which item an image is linked and vice versa, we just use a boolean column to mark if an image was already paid for. This makes fee calculation easier since no JOINs are required. * Add image fees during item creation/update * we calculate image fees during item creation and update now * function imageFees returns queries which deduct fees from user and mark images as paid + fees * queries need to be run inside same transaction as item creation/update * Allow anons to get presigned URLs * Add comments regarding avatar upload * Use megabytes in error message * Remove unnecessary avatar check during image fees calculation * Show image fees in frontend * Also update image fees on blur This makes sure that the images fees reflect the current state. For example, if an image was removed. We could also add debounced requests. * Show amount of unpaid images in receipt * Fix fees in sats deducted from msats * Fix algebraic order of fees Spam fees must come immediately after the base fee since it multiplies the base fee. * Fix image fees in edit receipt * Fix stale fees shown If we pay for an image and then want to edit the comment, the cache might return stale date; suggesting we didn't pay for the existing image yet. * Add 0 base fee in edit receipt * Remove 's' from 'image fees' in receipts * Remove unnecessary async * Remove 'Uploading <name>...' from text input on error * Support upload of multiple files at once * Add schedule to delete unused images * Fix image fee display in receipts * Use Drag and Drop API for image upload * Remove dragOver style on drop * Increase max upload size to 10MB to allow HQ camera pictures * Fix free upload quota * Fix stale image fees served * Fix bad image fee return statements * Fix multiplication with feesPerImage * Fix NULL returned for size24h, sizeNow * Remove unnecessary text field in query * refactor: Unify <ImageUpload> and <Upload> component * Add avatar cache busting using random query param * Calculate image fee info in postgres function * we now calculate image fee info in a postgres function which is much cleaner * we use this function inside `create_item` and `update_item`: image fees are now deducted in the same transaction as creating/updating the item! * reversed changes in `serializeInvoiceable` * Fix line break in receipt * Update upload limits * Add comment about `e.target.value = null` * Use debounce instead of onBlur to update image fees info * Fix invoice amount * Refactor avatar upload control flow * Update image fees in onChange * Fix rescheduling of other jobs * also update schedule from every minute to every hour * Add image fees in calling context * keep item ids on uploads * Fix incompatible onSubmit signature * Revert "keep item ids on uploads" This reverts commit 4688962abcd54fdc5850109372a7ad054cf9b2e4. * many2many item uploads * pretty subdomain for images * handle upload conditions for profile images and job logos --------- Co-authored-by: ekzyis <ek@ekzyis.com> Co-authored-by: ekzyis <ek@stacker.news>
2023-11-06 20:53:33 +00:00
import { deleteUnusedImages } from './deleteUnusedImages.js'
import { territoryBilling, territoryRevenue } from './territory.js'
2023-12-14 17:30:51 +00:00
import { ofac } from './ofac.js'
import { autoWithdraw } from './autowithdraw.js'
Store hashed and salted email addresses (#1111) * first pass of hashing user emails * use salt * add a salt to .env.development (prod salt needs to be kept a secret) * move `hashEmail` util to a new util module * trigger a one-time job to migrate existing emails via the worker so we can use the salt from an env var * move newsletter signup move newsletter signup to prisma adapter create user with email code path so we can still auto-enroll email accounts without having to persist the email address in plaintext * remove `email` from api key session lookup query * drop user email index before dropping column * restore email column, just null values instead * fix function name * fix salt and hash raw sql statement * update auth methods email type in typedefs from str to bool * remove todo comment * lowercase email before hashing during migration * check for emailHash and email to accommodate migration window update our lookups to check for a matching emailHash, and then a matching email, in that order, to accommodate the case that a user tries to login via email while the migration is running, and their account has not yet been migrated also update sndev to have a command `./sndev email` to launch the mailhog inbox in your browser also update `./sndev login` to hash the generated email address and insert it into the db record * update sndev help * update awards.csv * update the hack in next-auth to re-use the email supplied on input to `getUserByEmail` * consolidate console.error logs * create generic open command --------- Co-authored-by: Keyan <34140557+huumn@users.noreply.github.com> Co-authored-by: keyan <keyan.kousha+huumn@gmail.com>
2024-05-04 23:06:15 +00:00
import { saltAndHashEmails } from './saltAndHashEmails.js'
import { remindUser } from './reminder.js'
not-custodial zap beta (#1178) * not-custodial zap scaffolding * invoice forward state machine * small refinements to state machine * make wrap invoice work * get state machine working end to end * untested logic layout for paidAction invoice wraps * perform pessimisitic actions before outgoing payment * working end to end * remove unneeded params from wallets/server/createInvoice * fix cltv relative/absolute confusion + cancelling forwards * small refinements * add p2p wrap info to paidAction docs * fallback to SN invoice when wrap fails * fix paidAction retry description * consistent naming scheme for state machine * refinements * have sn pay bounded outbound fee * remove debug logging * reenable lnc permissions checks * don't p2p zap on item forward splits * make createInvoice params json encodeable * direct -> p2p badge on notifications * allow no tls in dev for core lightning * fix autowithdraw to create invoice with msats * fix autowithdraw msats/sats inconsitency * label p2p zaps properly in satistics * add fees to autowithdrawal notifications * add RETRYING as terminal paid action state * Update api/paidAction/README.md Co-authored-by: ekzyis <ek@stacker.news> * Update api/paidAction/README.md Co-authored-by: ekzyis <ek@stacker.news> * Update api/lnd/index.js Co-authored-by: ekzyis <ek@stacker.news> * ek suggestions * add bugetable to nwc card * get paranoid with numbers * better finalize retries and better max timeout height * refine forward failure transitions * more accurate satistics p2p status * make sure paidaction cancel in state machine only * dont drop bolt11s unless status is not null * only allow PENDING_HELD to transition to FORWARDING * add mermaid state machine diagrams to paid action doc * fix cancel transition name * cleanup readme * move forwarding outside of transition * refine testServerConnect and make sure ensureB64 transforms * remove unused params from testServerConnect --------- Co-authored-by: ekzyis <ek@stacker.news> Co-authored-by: k00b <k00b@stacker.news>
2024-08-13 14:48:30 +00:00
import {
paidActionPaid, paidActionForwarding, paidActionForwarded,
paidActionFailedForward, paidActionHeld, paidActionFailed,
paidActionCanceling
} from './paidAction.js'
import { thisDay } from './thisDay.js'
import { isServiceEnabled } from '@/lib/sndev.js'
Convert worker to ESM (#500) * Convert worker to ESM To use ESM for the worker, I created a package.json file in worker/ with `{ type: "module" }` as its sole content. I then rewrote every import to use ESM syntax. I also tried to set `{ type: "module" }` in the root package.json file to also use ESM in next.config.js. However, this resulted in a weird problem: default imports were now getting imported as objects in this shape: `{ default: <defaultImport> }`. Afaik, this should only be the case if you use "import * as foo from 'bar'" syntax: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import#default_import This is fixed by not using `{ type: "module" }` for some reason. However, then, next.config.js also doesn't support ESM import syntax anymore. The documentation says that if you want to use ESM, you can use next.config.mjs: https://nextjs.org/docs/pages/api-reference/next-config-js But I didn't want to use MJS extension since I don't have any experience with it. For example, not sure if it's good style to mix JS with MJS etc. So I kept the CJS import syntax there. * Ignore worker/ during linting I wasn't able to fix the following errors: /home/runner/work/stacker.news/stacker.news/worker/auction.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/auction.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/earn.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/earn.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/index.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/index.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/nostr.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/nostr.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/ots.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/ots.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/repin.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/repin.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/search.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/search.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/streak.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/streak.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/trust.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/trust.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/views.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/views.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) /home/runner/work/stacker.news/stacker.news/worker/wallet.js:0:0: Parsing error: No Babel config file detected for /home/runner/work/stacker.news/stacker.news/worker/wallet.js. Either disable config file checking with requireConfigFile: false, or configure Babel so that it can find the config files. (null) I tried to tell babel where to find the babel config file (.babelrc), specifying the babel config in worker/package.json under "babel", using babel.config.json etc. to no avail. However, afaict, we don't need babel for the worker since it won't run in a browser. Babel is only used to transpile code to target browsers. But it still would be nice to lint the worker code with standard. But we can figure this out later. * Fix worker imports from lib/ and api/ This fixes the issue that we can't use `{ "type": "module" }` in the root package.json since it breaks the app with this error: app | TypeError: next_auth_providers_credentials__WEBPACK_IMPORTED_MODULE_2__ is not a function app | at eval (webpack-internal:///./pages/api/auth/[...nextauth].js:218:20) app | at process.processTicksAndRejections (node:internal/process/task_queues:95:5) app | LND GRPC connection successful app | - error pages/api/auth/[...nextauth].js (139:2) @ CredentialsProvider app | - error Error [TypeError]: next_auth_providers_credentials__WEBPACK_IMPORTED_MODULE_2__ is not a function app | at eval (webpack-internal:///./pages/api/auth/[...nextauth].js:218:20) { app | digest: undefined app | } app | 137 | app | 138 | const providers = [ app | > 139 | CredentialsProvider({ app | | ^ app | 140 | id: 'lightning', app | 141 | name: 'Lightning', app | 142 | credentials: { app | TypeError: next_auth_providers_credentials__WEBPACK_IMPORTED_MODULE_2__ is not a function app | at eval (webpack-internal:///./pages/api/auth/[...nextauth].js:218:20) app | at process.processTicksAndRejections (node:internal/process/task_queues:95:5) build but we need to tell the worker that the files are MJS, else we get this error: worker | file:///app/worker/wallet.js:3 worker | import { datePivot } from '../lib/time.js' worker | ^^^^^^^^^ worker | SyntaxError: Named export 'datePivot' not found. The requested module '../lib/time.js' is a CommonJS module, which may not support all module.exports as named exports. worker | CommonJS modules can always be imported via the default export, for example using: worker | worker | import pkg from '../lib/time.js'; worker | const { datePivot } = pkg; worker | worker | at ModuleJob._instantiate (node:internal/modules/esm/module_job:124:21) worker | at async ModuleJob.run (node:internal/modules/esm/module_job:190:5) worker | worker | Node.js v18.17.0 worker | worker exited with code 1 * Fix global not defined in browser context * Also ignore api/ and lib/ during linting I did not want to do this but I was not able to fix this error in any other way I tried: /home/ekzyis/programming/stacker.news/api/lnd/index.js:0:0: Parsing error: No Babel config file detected for /home/ekzyis/programming/stacker.news/api/lnd/index.js. Either disable config file checking with requ ireConfigFile: false, or configure Babel so that it can find the config files. (null) Did not want to look deeper into all this standard, eslint, babel configuration stuff ... --------- Co-authored-by: ekzyis <ek@stacker.news> Co-authored-by: Keyan <34140557+huumn@users.noreply.github.com>
2023-09-24 01:19:35 +00:00
const { loadEnvConfig } = nextEnv
const { ApolloClient, HttpLink, InMemoryCache } = apolloClient
2024-05-08 21:25:01 +00:00
loadEnvConfig('.', process.env.NODE_ENV === 'development')
2023-02-14 22:58:12 +00:00
2022-01-05 20:37:34 +00:00
async function work () {
2022-01-17 17:41:17 +00:00
const boss = new PgBoss(process.env.DATABASE_URL)
const models = createPrisma({
connectionParams: { connection_limit: process.env.DB_WORKER_CONNECTION_LIMIT }
})
2023-12-14 17:30:51 +00:00
2022-01-25 19:34:51 +00:00
const apollo = new ApolloClient({
link: new HttpLink({
uri: `${process.env.SELF_URL}/api/graphql`,
fetch
}),
cache: new InMemoryCache(),
defaultOptions: {
watchQuery: {
2022-11-24 19:22:58 +00:00
fetchPolicy: 'no-cache',
nextFetchPolicy: 'no-cache'
2022-01-25 19:34:51 +00:00
},
query: {
2022-11-24 19:22:58 +00:00
fetchPolicy: 'no-cache',
nextFetchPolicy: 'no-cache'
2022-01-25 19:34:51 +00:00
}
}
})
2023-02-14 22:58:12 +00:00
const { lnd } = authenticatedLndGrpc({
cert: process.env.LND_CERT,
macaroon: process.env.LND_MACAROON,
socket: process.env.LND_SOCKET
})
const args = { boss, models, apollo, lnd }
2022-01-14 17:28:05 +00:00
2022-01-17 17:41:17 +00:00
boss.on('error', error => console.error(error))
2022-01-07 16:32:31 +00:00
2023-11-21 23:32:22 +00:00
function jobWrapper (fn) {
return async function (job) {
console.log(`running ${job.name} with args`, job.data)
try {
await fn({ ...job, ...args })
} catch (error) {
console.error(`error running ${job.name}`, error)
throw error
}
console.log(`finished ${job.name}`)
}
}
2022-01-17 17:41:17 +00:00
await boss.start()
if (isServiceEnabled('payments')) {
await subscribeToWallet(args)
await boss.work('finalizeHodlInvoice', jobWrapper(finalizeHodlInvoice))
await boss.work('checkPendingDeposits', jobWrapper(checkPendingDeposits))
await boss.work('checkPendingWithdrawals', jobWrapper(checkPendingWithdrawals))
await boss.work('autoDropBolt11s', jobWrapper(autoDropBolt11s))
await boss.work('autoWithdraw', jobWrapper(autoWithdraw))
await boss.work('checkInvoice', jobWrapper(checkInvoice))
not-custodial zap beta (#1178) * not-custodial zap scaffolding * invoice forward state machine * small refinements to state machine * make wrap invoice work * get state machine working end to end * untested logic layout for paidAction invoice wraps * perform pessimisitic actions before outgoing payment * working end to end * remove unneeded params from wallets/server/createInvoice * fix cltv relative/absolute confusion + cancelling forwards * small refinements * add p2p wrap info to paidAction docs * fallback to SN invoice when wrap fails * fix paidAction retry description * consistent naming scheme for state machine * refinements * have sn pay bounded outbound fee * remove debug logging * reenable lnc permissions checks * don't p2p zap on item forward splits * make createInvoice params json encodeable * direct -> p2p badge on notifications * allow no tls in dev for core lightning * fix autowithdraw to create invoice with msats * fix autowithdraw msats/sats inconsitency * label p2p zaps properly in satistics * add fees to autowithdrawal notifications * add RETRYING as terminal paid action state * Update api/paidAction/README.md Co-authored-by: ekzyis <ek@stacker.news> * Update api/paidAction/README.md Co-authored-by: ekzyis <ek@stacker.news> * Update api/lnd/index.js Co-authored-by: ekzyis <ek@stacker.news> * ek suggestions * add bugetable to nwc card * get paranoid with numbers * better finalize retries and better max timeout height * refine forward failure transitions * more accurate satistics p2p status * make sure paidaction cancel in state machine only * dont drop bolt11s unless status is not null * only allow PENDING_HELD to transition to FORWARDING * add mermaid state machine diagrams to paid action doc * fix cancel transition name * cleanup readme * move forwarding outside of transition * refine testServerConnect and make sure ensureB64 transforms * remove unused params from testServerConnect --------- Co-authored-by: ekzyis <ek@stacker.news> Co-authored-by: k00b <k00b@stacker.news>
2024-08-13 14:48:30 +00:00
await boss.work('checkWithdrawal', jobWrapper(checkWithdrawal))
// paidAction jobs
await boss.work('paidActionForwarding', jobWrapper(paidActionForwarding))
await boss.work('paidActionForwarded', jobWrapper(paidActionForwarded))
await boss.work('paidActionFailedForward', jobWrapper(paidActionFailedForward))
await boss.work('paidActionHeld', jobWrapper(paidActionHeld))
await boss.work('paidActionCanceling', jobWrapper(paidActionCanceling))
await boss.work('paidActionFailed', jobWrapper(paidActionFailed))
await boss.work('paidActionPaid', jobWrapper(paidActionPaid))
// we renamed these jobs so we leave them so they can "migrate"
await boss.work('holdAction', jobWrapper(paidActionHeld))
await boss.work('settleActionError', jobWrapper(paidActionFailed))
await boss.work('settleAction', jobWrapper(paidActionPaid))
}
if (isServiceEnabled('search')) {
await boss.work('indexItem', jobWrapper(indexItem))
await boss.work('indexAllItems', jobWrapper(indexAllItems))
}
if (isServiceEnabled('images')) {
await boss.work('imgproxy', jobWrapper(imgproxy))
await boss.work('deleteUnusedImages', jobWrapper(deleteUnusedImages))
}
2023-11-21 23:32:22 +00:00
await boss.work('repin-*', jobWrapper(repin))
await boss.work('trust', jobWrapper(trust))
await boss.work('timestampItem', jobWrapper(timestampItem))
await boss.work('auction', jobWrapper(auction))
await boss.work('earn', jobWrapper(earn))
await boss.work('streak', jobWrapper(computeStreaks))
await boss.work('checkStreak', jobWrapper(checkStreak))
await boss.work('nip57', jobWrapper(nip57))
2024-01-19 21:19:26 +00:00
await boss.work('views-*', jobWrapper(views))
2023-11-21 23:32:22 +00:00
await boss.work('rankViews', jobWrapper(rankViews))
await boss.work('deleteItem', jobWrapper(deleteItem))
await boss.work('territoryBilling', jobWrapper(territoryBilling))
await boss.work('territoryRevenue', jobWrapper(territoryRevenue))
2023-12-14 17:30:51 +00:00
await boss.work('ofac', jobWrapper(ofac))
Store hashed and salted email addresses (#1111) * first pass of hashing user emails * use salt * add a salt to .env.development (prod salt needs to be kept a secret) * move `hashEmail` util to a new util module * trigger a one-time job to migrate existing emails via the worker so we can use the salt from an env var * move newsletter signup move newsletter signup to prisma adapter create user with email code path so we can still auto-enroll email accounts without having to persist the email address in plaintext * remove `email` from api key session lookup query * drop user email index before dropping column * restore email column, just null values instead * fix function name * fix salt and hash raw sql statement * update auth methods email type in typedefs from str to bool * remove todo comment * lowercase email before hashing during migration * check for emailHash and email to accommodate migration window update our lookups to check for a matching emailHash, and then a matching email, in that order, to accommodate the case that a user tries to login via email while the migration is running, and their account has not yet been migrated also update sndev to have a command `./sndev email` to launch the mailhog inbox in your browser also update `./sndev login` to hash the generated email address and insert it into the db record * update sndev help * update awards.csv * update the hack in next-auth to re-use the email supplied on input to `getUserByEmail` * consolidate console.error logs * create generic open command --------- Co-authored-by: Keyan <34140557+huumn@users.noreply.github.com> Co-authored-by: keyan <keyan.kousha+huumn@gmail.com>
2024-05-04 23:06:15 +00:00
await boss.work('saltAndHashEmails', jobWrapper(saltAndHashEmails))
await boss.work('reminder', jobWrapper(remindUser))
await boss.work('thisDay', jobWrapper(thisDay))
2022-01-05 20:37:34 +00:00
2022-01-17 17:41:17 +00:00
console.log('working jobs')
2022-01-05 20:37:34 +00:00
}
work()