stacker.news/worker/deleteUnusedImages.js

29 lines
1.1 KiB
JavaScript
Raw Normal View History

Image uploads (#576) * Add icon to add images * Open file explorer to select image * Upload images to S3 on selection * Show uploaded images below text input * Link and remove image * Fetch unsubmitted images from database * Mark S3 images as submitted in imgproxy job * Add margin-top * Mark images as submitted on client after successful mutation * Also delete objects in S3 * Allow items to have multiple uploads linked * Overwrite old avatar * Add fees for presigned URLs * Use Github style upload * removed upfront fees * removed images provider since we no longer need to keep track of unsubmitted images on the client * removed User.images resolver * removed deleteImage mutation * use Github style upload where it shows ![Uploading <filename>...]() first and then replaces that with ![<filename>](<url>) after successful upload * Add Upload.paid boolean column One item can have multiple images linked to it, but an image can also be used in multiple items (many-to-many relation). Since we don't really care to which item an image is linked and vice versa, we just use a boolean column to mark if an image was already paid for. This makes fee calculation easier since no JOINs are required. * Add image fees during item creation/update * we calculate image fees during item creation and update now * function imageFees returns queries which deduct fees from user and mark images as paid + fees * queries need to be run inside same transaction as item creation/update * Allow anons to get presigned URLs * Add comments regarding avatar upload * Use megabytes in error message * Remove unnecessary avatar check during image fees calculation * Show image fees in frontend * Also update image fees on blur This makes sure that the images fees reflect the current state. For example, if an image was removed. We could also add debounced requests. * Show amount of unpaid images in receipt * Fix fees in sats deducted from msats * Fix algebraic order of fees Spam fees must come immediately after the base fee since it multiplies the base fee. * Fix image fees in edit receipt * Fix stale fees shown If we pay for an image and then want to edit the comment, the cache might return stale date; suggesting we didn't pay for the existing image yet. * Add 0 base fee in edit receipt * Remove 's' from 'image fees' in receipts * Remove unnecessary async * Remove 'Uploading <name>...' from text input on error * Support upload of multiple files at once * Add schedule to delete unused images * Fix image fee display in receipts * Use Drag and Drop API for image upload * Remove dragOver style on drop * Increase max upload size to 10MB to allow HQ camera pictures * Fix free upload quota * Fix stale image fees served * Fix bad image fee return statements * Fix multiplication with feesPerImage * Fix NULL returned for size24h, sizeNow * Remove unnecessary text field in query * refactor: Unify <ImageUpload> and <Upload> component * Add avatar cache busting using random query param * Calculate image fee info in postgres function * we now calculate image fee info in a postgres function which is much cleaner * we use this function inside `create_item` and `update_item`: image fees are now deducted in the same transaction as creating/updating the item! * reversed changes in `serializeInvoiceable` * Fix line break in receipt * Update upload limits * Add comment about `e.target.value = null` * Use debounce instead of onBlur to update image fees info * Fix invoice amount * Refactor avatar upload control flow * Update image fees in onChange * Fix rescheduling of other jobs * also update schedule from every minute to every hour * Add image fees in calling context * keep item ids on uploads * Fix incompatible onSubmit signature * Revert "keep item ids on uploads" This reverts commit 4688962abcd54fdc5850109372a7ad054cf9b2e4. * many2many item uploads * pretty subdomain for images * handle upload conditions for profile images and job logos --------- Co-authored-by: ekzyis <ek@ekzyis.com> Co-authored-by: ekzyis <ek@stacker.news>
2023-11-06 20:53:33 +00:00
import { deleteObjects } from '../api/s3'
import { ANON_USER_ID } from '../lib/constants'
Image uploads (#576) * Add icon to add images * Open file explorer to select image * Upload images to S3 on selection * Show uploaded images below text input * Link and remove image * Fetch unsubmitted images from database * Mark S3 images as submitted in imgproxy job * Add margin-top * Mark images as submitted on client after successful mutation * Also delete objects in S3 * Allow items to have multiple uploads linked * Overwrite old avatar * Add fees for presigned URLs * Use Github style upload * removed upfront fees * removed images provider since we no longer need to keep track of unsubmitted images on the client * removed User.images resolver * removed deleteImage mutation * use Github style upload where it shows ![Uploading <filename>...]() first and then replaces that with ![<filename>](<url>) after successful upload * Add Upload.paid boolean column One item can have multiple images linked to it, but an image can also be used in multiple items (many-to-many relation). Since we don't really care to which item an image is linked and vice versa, we just use a boolean column to mark if an image was already paid for. This makes fee calculation easier since no JOINs are required. * Add image fees during item creation/update * we calculate image fees during item creation and update now * function imageFees returns queries which deduct fees from user and mark images as paid + fees * queries need to be run inside same transaction as item creation/update * Allow anons to get presigned URLs * Add comments regarding avatar upload * Use megabytes in error message * Remove unnecessary avatar check during image fees calculation * Show image fees in frontend * Also update image fees on blur This makes sure that the images fees reflect the current state. For example, if an image was removed. We could also add debounced requests. * Show amount of unpaid images in receipt * Fix fees in sats deducted from msats * Fix algebraic order of fees Spam fees must come immediately after the base fee since it multiplies the base fee. * Fix image fees in edit receipt * Fix stale fees shown If we pay for an image and then want to edit the comment, the cache might return stale date; suggesting we didn't pay for the existing image yet. * Add 0 base fee in edit receipt * Remove 's' from 'image fees' in receipts * Remove unnecessary async * Remove 'Uploading <name>...' from text input on error * Support upload of multiple files at once * Add schedule to delete unused images * Fix image fee display in receipts * Use Drag and Drop API for image upload * Remove dragOver style on drop * Increase max upload size to 10MB to allow HQ camera pictures * Fix free upload quota * Fix stale image fees served * Fix bad image fee return statements * Fix multiplication with feesPerImage * Fix NULL returned for size24h, sizeNow * Remove unnecessary text field in query * refactor: Unify <ImageUpload> and <Upload> component * Add avatar cache busting using random query param * Calculate image fee info in postgres function * we now calculate image fee info in a postgres function which is much cleaner * we use this function inside `create_item` and `update_item`: image fees are now deducted in the same transaction as creating/updating the item! * reversed changes in `serializeInvoiceable` * Fix line break in receipt * Update upload limits * Add comment about `e.target.value = null` * Use debounce instead of onBlur to update image fees info * Fix invoice amount * Refactor avatar upload control flow * Update image fees in onChange * Fix rescheduling of other jobs * also update schedule from every minute to every hour * Add image fees in calling context * keep item ids on uploads * Fix incompatible onSubmit signature * Revert "keep item ids on uploads" This reverts commit 4688962abcd54fdc5850109372a7ad054cf9b2e4. * many2many item uploads * pretty subdomain for images * handle upload conditions for profile images and job logos --------- Co-authored-by: ekzyis <ek@ekzyis.com> Co-authored-by: ekzyis <ek@stacker.news>
2023-11-06 20:53:33 +00:00
2023-11-21 23:32:22 +00:00
export async function deleteUnusedImages ({ models }) {
// delete all images in database and S3 which weren't paid in the last 24 hours
const unpaidImages = await models.$queryRaw`
SELECT id
FROM "Upload"
WHERE (paid = 'f'
OR (
-- for non-textarea images, they are free and paid is null
paid IS NULL
-- if the image is not used by a user or item (eg jobs), delete it
AND NOT EXISTS (SELECT * FROM users WHERE "photoId" = "Upload".id)
AND NOT EXISTS (SELECT * FROM "Item" WHERE "uploadId" = "Upload".id)
))
AND created_at < date_trunc('hour', now() - CASE WHEN "userId" = ${ANON_USER_ID} THEN interval '1 hour' ELSE interval '24 hours' END)`
Image uploads (#576) * Add icon to add images * Open file explorer to select image * Upload images to S3 on selection * Show uploaded images below text input * Link and remove image * Fetch unsubmitted images from database * Mark S3 images as submitted in imgproxy job * Add margin-top * Mark images as submitted on client after successful mutation * Also delete objects in S3 * Allow items to have multiple uploads linked * Overwrite old avatar * Add fees for presigned URLs * Use Github style upload * removed upfront fees * removed images provider since we no longer need to keep track of unsubmitted images on the client * removed User.images resolver * removed deleteImage mutation * use Github style upload where it shows ![Uploading <filename>...]() first and then replaces that with ![<filename>](<url>) after successful upload * Add Upload.paid boolean column One item can have multiple images linked to it, but an image can also be used in multiple items (many-to-many relation). Since we don't really care to which item an image is linked and vice versa, we just use a boolean column to mark if an image was already paid for. This makes fee calculation easier since no JOINs are required. * Add image fees during item creation/update * we calculate image fees during item creation and update now * function imageFees returns queries which deduct fees from user and mark images as paid + fees * queries need to be run inside same transaction as item creation/update * Allow anons to get presigned URLs * Add comments regarding avatar upload * Use megabytes in error message * Remove unnecessary avatar check during image fees calculation * Show image fees in frontend * Also update image fees on blur This makes sure that the images fees reflect the current state. For example, if an image was removed. We could also add debounced requests. * Show amount of unpaid images in receipt * Fix fees in sats deducted from msats * Fix algebraic order of fees Spam fees must come immediately after the base fee since it multiplies the base fee. * Fix image fees in edit receipt * Fix stale fees shown If we pay for an image and then want to edit the comment, the cache might return stale date; suggesting we didn't pay for the existing image yet. * Add 0 base fee in edit receipt * Remove 's' from 'image fees' in receipts * Remove unnecessary async * Remove 'Uploading <name>...' from text input on error * Support upload of multiple files at once * Add schedule to delete unused images * Fix image fee display in receipts * Use Drag and Drop API for image upload * Remove dragOver style on drop * Increase max upload size to 10MB to allow HQ camera pictures * Fix free upload quota * Fix stale image fees served * Fix bad image fee return statements * Fix multiplication with feesPerImage * Fix NULL returned for size24h, sizeNow * Remove unnecessary text field in query * refactor: Unify <ImageUpload> and <Upload> component * Add avatar cache busting using random query param * Calculate image fee info in postgres function * we now calculate image fee info in a postgres function which is much cleaner * we use this function inside `create_item` and `update_item`: image fees are now deducted in the same transaction as creating/updating the item! * reversed changes in `serializeInvoiceable` * Fix line break in receipt * Update upload limits * Add comment about `e.target.value = null` * Use debounce instead of onBlur to update image fees info * Fix invoice amount * Refactor avatar upload control flow * Update image fees in onChange * Fix rescheduling of other jobs * also update schedule from every minute to every hour * Add image fees in calling context * keep item ids on uploads * Fix incompatible onSubmit signature * Revert "keep item ids on uploads" This reverts commit 4688962abcd54fdc5850109372a7ad054cf9b2e4. * many2many item uploads * pretty subdomain for images * handle upload conditions for profile images and job logos --------- Co-authored-by: ekzyis <ek@ekzyis.com> Co-authored-by: ekzyis <ek@stacker.news>
2023-11-06 20:53:33 +00:00
2023-11-21 23:32:22 +00:00
const s3Keys = unpaidImages.map(({ id }) => id)
2024-01-28 18:29:56 +00:00
if (s3Keys.length === 0) {
console.log('no images to delete.')
return
}
2023-11-21 23:32:22 +00:00
console.log('deleting images:', s3Keys)
2024-01-28 18:29:56 +00:00
const deleted = await deleteObjects(s3Keys)
console.log('deleted images:', deleted)
await models.upload.deleteMany({ where: { id: { in: deleted } } })
Image uploads (#576) * Add icon to add images * Open file explorer to select image * Upload images to S3 on selection * Show uploaded images below text input * Link and remove image * Fetch unsubmitted images from database * Mark S3 images as submitted in imgproxy job * Add margin-top * Mark images as submitted on client after successful mutation * Also delete objects in S3 * Allow items to have multiple uploads linked * Overwrite old avatar * Add fees for presigned URLs * Use Github style upload * removed upfront fees * removed images provider since we no longer need to keep track of unsubmitted images on the client * removed User.images resolver * removed deleteImage mutation * use Github style upload where it shows ![Uploading <filename>...]() first and then replaces that with ![<filename>](<url>) after successful upload * Add Upload.paid boolean column One item can have multiple images linked to it, but an image can also be used in multiple items (many-to-many relation). Since we don't really care to which item an image is linked and vice versa, we just use a boolean column to mark if an image was already paid for. This makes fee calculation easier since no JOINs are required. * Add image fees during item creation/update * we calculate image fees during item creation and update now * function imageFees returns queries which deduct fees from user and mark images as paid + fees * queries need to be run inside same transaction as item creation/update * Allow anons to get presigned URLs * Add comments regarding avatar upload * Use megabytes in error message * Remove unnecessary avatar check during image fees calculation * Show image fees in frontend * Also update image fees on blur This makes sure that the images fees reflect the current state. For example, if an image was removed. We could also add debounced requests. * Show amount of unpaid images in receipt * Fix fees in sats deducted from msats * Fix algebraic order of fees Spam fees must come immediately after the base fee since it multiplies the base fee. * Fix image fees in edit receipt * Fix stale fees shown If we pay for an image and then want to edit the comment, the cache might return stale date; suggesting we didn't pay for the existing image yet. * Add 0 base fee in edit receipt * Remove 's' from 'image fees' in receipts * Remove unnecessary async * Remove 'Uploading <name>...' from text input on error * Support upload of multiple files at once * Add schedule to delete unused images * Fix image fee display in receipts * Use Drag and Drop API for image upload * Remove dragOver style on drop * Increase max upload size to 10MB to allow HQ camera pictures * Fix free upload quota * Fix stale image fees served * Fix bad image fee return statements * Fix multiplication with feesPerImage * Fix NULL returned for size24h, sizeNow * Remove unnecessary text field in query * refactor: Unify <ImageUpload> and <Upload> component * Add avatar cache busting using random query param * Calculate image fee info in postgres function * we now calculate image fee info in a postgres function which is much cleaner * we use this function inside `create_item` and `update_item`: image fees are now deducted in the same transaction as creating/updating the item! * reversed changes in `serializeInvoiceable` * Fix line break in receipt * Update upload limits * Add comment about `e.target.value = null` * Use debounce instead of onBlur to update image fees info * Fix invoice amount * Refactor avatar upload control flow * Update image fees in onChange * Fix rescheduling of other jobs * also update schedule from every minute to every hour * Add image fees in calling context * keep item ids on uploads * Fix incompatible onSubmit signature * Revert "keep item ids on uploads" This reverts commit 4688962abcd54fdc5850109372a7ad054cf9b2e4. * many2many item uploads * pretty subdomain for images * handle upload conditions for profile images and job logos --------- Co-authored-by: ekzyis <ek@ekzyis.com> Co-authored-by: ekzyis <ek@stacker.news>
2023-11-06 20:53:33 +00:00
}