I had a good week doing one thing. The Client API now supports the main duplicates commands, fetching current file relationships and the main filter workflows, and then setting basic file relationships back again. This is a complicated feature for users who play around with the latest Client API tech, so tomorrow's release is for those advanced users only!

The release should be as normal tomorrow.

Is there a way to show the total duration of all selected video files? It shows the total file size (rounded), but I can't find an option to show the total duration.

Is there some way to apply to all selected images tags like page:1, page:2, etc in same order as it visible now?

>>1442
I had this exact same thought over my vacation when I was looking at some vidya captures I had made. I'll add this!

>>1443
Not yet, but I would like to add this. I've never really been happy with how hydrus handles 'page:1' style tags, but we'll never get better at it as long as it is difficult to easily add them to files after they are imported.

I had a great week back at work. I cleared a variety of small jobs and spent some extra time on the duplicates system. Several duplicates search bugs are fixed, and you can now set a potential pairs search for 'one file is in search A, the other is in search B'.

The release should be as normal tomorrow.

I had a good week mostly fixing bugs, including the recent annoying-but-harmless shutdown bug some users had and the macOS app boot bug. There's also some UI quality of life.

The release should be as normal tomorrow.

>>1425

Thanks for your continuing work on the project man, much appreciated. I have some feedback/wishes:

1. 4chan thread watcher via Tor gives 403 Cloudflare error - simple downloader works. Is there something one can do about this beyond switching circuits/proxy/clearnet? I mentioned this before and the browser header shouldn't make a difference here, as downloads as such work, just not the watcher.

2. 4chan simple downloader fails at .webms in same setup, these have to be manually added.

3. Add a second/third bar for tabs, so that the titles are not squeezed when having many tabs open.

4. PDF/ebook compatbility to use Hydrus for more informative tasks (this might get you a lot of support from librarian types).

5. Grouping of tags - as with the namespace/creator group by function, have colored bars or something that groups all items belonging to that category - or some similar function to have a "folder like" grouping in the main window. Tags are awesome for precise searches/classifications, but sometimes that "all in one" feeling that folders offer might be useful; I may have overlooked something though.

As always, I really love the work you do and you simplify a lot of time intensive stuff. I'd love to have a setup working that scrapes threads automatically by topic, I was thinking of a hydrus + something scripted setup that identifies threads by title/keyword and then automatically adds them to the watcher. This would save so much time and erase blind spots when covering news topics etc. Any ideas as to the feasibility?

>>1426
Thanks, I'm glad you like it!

1) The watcher hits up the 4chan API (in this form https://a.4cdn.org/m/thread/16086187.json), whereas the simple downloader hits the main html. My guess is that's what CF is blocking. If you had to set up headers or User-Agent to get the html to work, might want to replicate it for a.4cdn.org. Hydrus Companion may help here to copy cookies, not sure. If CF just blocks API hits from TOR, you may be out of luck.

2) Yeah, I bet the simple downloader looks for img tags or something, whereas webms are under 'video' or an iframe or something. Simple downloader can't do two things at once very well, I'm afraid! If you are feeling brave, you could try editing the simple parsing formula to try to get both img and (video), but I'm not sure if it is that clever.

3) Check out the 'page of pages'. Hit F9->special->page of pages. That'll let you nest your page tabs.

4) Thanks. Yeah, I'd like some basic support for more text formats, and 'any file' support, which will take some extra work for some internal technical reasons. But I'd like it. However, I do not think I can support nice text reading any time soon. I'm terrible at drawing lines, but I know I'll never have the time to make something anywhere near as good as Calibre, so the support will stay as an 'open externally' button, like PDFs now, for a long time.

5) Yeah, I absolutely want [+] expand/tree format for the tag list. That list is my own custom widget, so I have to code everything in it. But this is something I really want for myself too, and options for things like 'start with title tags collapsed'. And custom namespace sorting so you can put creator up top, that sort of stuff.

Having auto-scraping has been a long-time thought. I'm hesitant to make too many automatic searchers, since I like human eyeballs behind big decisions. I know some users, however, who have had big success writing their own scripts that fetch the current thread catalog, scan it for threads they want, and then pass them on to Hydrus using the Client API. If you are ok with scripting, you might like to play around with it: https://hydrusnetwork.github.io/hydrus/client_api.html

As always, I'm short on work time. When I eventually roll some of 4 + 5 out, let me know how it goes for you, and if you have any other problems or ideas.

>>1428
1) Will try that out, once I find the time. Thanks for the answer!

2) I'll experiment on that as well and will report back with how it goes.

3) Nice, awesome!

4) I'd be enough to merely have the works indexed inside Hydrus. You can always CTRL+E to open in your preferred reader/program with all the capabilities you need on the media itself. It's just so you can collect and organize the files, which Hydrus really blows anything else I know out of the water. There's some academic software that lets you organize your sources and stuff, but it's more of a list thing. Since you can add tons of metadata... well let's just say Hydrus and zlib would go hand in hand ;)

5) Yeah, collapsible tag trees would make stuff more easily readable. I know you can organize by custom namespaces and that is already really powerful, but it'd be cool to have tag:one tag:two tag:three below each other in the tag list and in the folder list as well, in "blocks" so to speak.

I feel you on automation. One misstep and you practically DDoS a site or flood your hard drive. I also didn't listen to your warning in the documentation haha. "and then THEY DO SO only to work through thousands of files", I felt that so hard and it brought a smile to my face. But once you have figured out what you want to parse daily etc. you then really want the automation for these specifica.

I really do appreciate your time man. Your project is one I always look into the documentation again and check the forums here. That's a rare exception for a really solid project. If you go the academic/library route, you might catch some licensing stuff to capitalize on your outstanding work, like for official use and stuff.

birthday

The first non-experimental beta of hydrus was released on December 14th, 2011. We are now going on eleven years.

I had a challenging 2022. Several IRL problems appeared from nowhere, but I decided to keep on trudging, and we got some work done regardless. I'm satisfied with the result. We started the year with the run up to multiple local file services, a long-planned expansion to the sheer shape of the database, and with that, beyond the new file services, came an array of new database search tech, from the ability to search deleted files and search unions of different file domains to enhanced cancel tech for snappier UI and more accurate tag counts and the nice user-made file history chart. We also saw new time search tech, with nicer UI, the ability to search the last view time of a file, the recording of archive times, and the broad 'modified time' parsing expanson, which combines post times from many sources to give you a more precise 'source time' of the things you download. Thanks to user help, we also figured out the new help, which looks great, is easily searchable, and is simple to edit. We hammered out more stable versions of mpv and other spicy libraries for the different builds and user environments. The UI had numerous quality of life improvements, from dejanked layouts and sizing calculations to fixed custom widgets and overhauled hover windows to cleaner selection logic and quick-action buttons for easy pastes and mass-application actions. File notes are finally parsable and now display on the media viewer. Sidecar importing and exporting became more powerful. EXIF and other human-readable file metadata is now parsed and viewable. The serverside janitor workflow finally received some improvement. The Client API grew to support autocomplete tag searching, note editing, cleverer file reference and file delete, nicer thumbnail fetching, better error/disconnect handling, tag display type support, more hash fetchings and lookups, and the new timestamps and file metadata statuses like EXIF. And the program became significantly easier for any user to run from source.

There is still no end of work to do, and I am thankfully able to continue, so I plan to keep going into 2023. I deeply appreciate the feedback, help, and support over the years. Thank you!

If you would like to further support my work and are in a position to do so, my simple no-reward Patreon is here: https://www.patreon.com/hydrus_dev

Congrats on turning 11, dev. Hope you're doing well.
I've been using Hydrus off and on since 2015, but I reimported a few times early on so my oldest files are from 6 years ago. I've been going through some of them, finally getting around to sorting a bunch of stuff I scraped from tumblr back then.
I just transferred my "active" copy of Hydrus to a new drive, figured the original one might be getting worn out, best to keep it as the backup. In the process, it seems like it forgot to keep sound off, that or sound in-client is just a recent addition.
Is there a way I can toggle sound on and off globally for the program?

I had a good week. I fixed some bugs, added some new import and tag search options for advanced users, and made thumbnails look crisp at high UI scale.

The release should be as normal tomorrow.

>>1437
Thanks. Yeah, I am doing good overall. Looking forward to my vacation next week though!

We added 'mpv' support I think two or three years ago--that's where the sound is coming from. You should notice video plays a lot smoother too. Any time you are looking at a noisy video, move your mouse to the bottom of the video frame so the scanbar pops up. On the right should be a 'speaker' symbol. Click it and you'll get global mute. There's another speaker symbol in a popup that just mutes preview/media viewer.

There's some shortcuts for this too, under file->shortcuts->global.

>>1438
Thanks!

I had a good week. I focused on improving file notes, making it easier to copy them from one file to another and adding them to the duplicate merge settings. I also made taglist drag-selection feel nicer.

The release should be as normal tomorrow.

>>1429
Unfortunately this is not supported yet. I'd love it though, and other users. It will take a lot of work, so I cannot predict when it will happen.

>>1430
Probably this is naive and I miss something, but can I do it via client.master DB editing?
For example id of artist namespace is 1 and creator is 2. Whats happen if in table tags i replace all 1 in namespace_id column by 2?

>>1431
These are my local tags and I want to adjust them to the hydrus standard. So it will be fully ok if there is no sibling for it or something else. Simple replace, that's all that I want.

>>1431
>>1432
Unfortunately I would say it is not reasonably possible. If you want to poke around client.master.db, feel free, but you must make a backup beforehand.

The stuff in client.master.db is replicated in client.caches.db, and the 'master' mappings records are in client.mappings.db, which are again replicated in the caches db. You'd likely have to change ids or definitions in multiple locations to make this work without errors, and all your tag counts might need to be regenerated since we are talking about a merge rather than a rename here. With few exceptions, the database just isn't simple enough to edit a namespace by hand.

>>1433
Oh, I see, better I do not touch it. Thank you for your help.

everything is broke again
venv won't start because there is no activate.bat
why do you hate me so much, why do you make me go through all this pain and torture? just because i won't submit to windows10?

I had a great week fixing some bugs, adding a hash lookup to the Client API, and attaching a Tag Filter to the PTR, which will make it easier to clear out some less useful and misparsed tags right as they are uploaded, no petition needed.

The release should be as normal tomorrow.

>>1420
Sorry to hear your trouble. Can you talk more about what happened? Did you change the venv, or could an anti-virus or other program have interfered with it? If the activate.bat has disappeared, try running the setup-venv.bat again--that'll clear out your old venv folder and reinstall everything.

Since you have had problems with this, could you tell me what has been most difficult to get going? Has my 'running from source' help been confusing at any point? Did the setup bat fail? How did things break before, and what was the solution? How could I make it easier? I have been adjusting the help and the scripts in recent weeks to be more clear on each step, so I regret it hasn't worked for you.

>>1421
I managed to make it work by building from scratch in a new folder.
Before I had database failure, due to my mistake of not backing it up properly.
Venv failed before until I found working mix of qt5, old python, new mpv.
It's painful because I don't even know the difference and simply seeking blindly for something that will run somehow.

>>1423
Sorry again for the problems. I am glad you are working now. Which old version of python worked for you? 3.8 or earlier? To make it less of a 'blind' situation in future, how could I improve my help? Are there specific version recommendations I could make for people on Windows 7?

You can back up the venv folder, btw, so now you have something that works, you can save it all--it is completely portable.

I will try and ensure that hydrus keeps working for Qt5 and older python as long as I reasonably can, so let me know if a future update breaks something.

I had an ok week fixing some bugs, expanding the Client API a little, and improving some server admin tools.

The release should be as normal tomorrow.

icc profile and pixel hash

This is mostly database prep for future duplicate system expansions.

The client database now records whether still images have an ICC profile, and it also saves data for 'these images are exact pixel duplicates' decisions. On update, all your existing files will be queued for scans to fill in this data in the background. Anything with an ICC profile will also regenerate its thumbnail. You don't have to do anything, this will all happen automatically over the coming week(s).

In time, you'll be able to search for images with ICC profiles with the new 'system:has icc profile' search predicate. This predicate is weird and advanced, so I think I'll hide it away soon under an umbrella for advanced stuff.

The 'exact pixel duplicate' data will be useful in the near future, when I expand the duplicate system to find (and optionally automatically merge) certain pairs that are perfect visual dupes.

full list

- misc:
- fixed a recent bug in wildcard search where 't*g' unnamespaced wildcards were not returning namespace results
- sped up multi-predicate wildcard searches in some unusual file domains
- the file maintenance manager is now more polite about how it works. no matter its speed settings, it now pauses regularly to check on and wait until the main UI is feeling good to start new work. this should relieve heavier clients on older machines who will get a bunch of new work to do this week
- 'all local files' in _review services_ now states how many files are awaiting physical deletion in the new deferred delete queue. this live updates when the values change but should be 0 pretty much all the time

- 'has icc profile' now cached in database:
- the client database now keeps track of which image files have an icc profile. this data is added on file import
- a new file maintenance task can generate it retroactively, and if a file is discovered to have an icc profile, it will be scheduled for a thumbnail regeneration too
- a new system predicate, 'system:has icc profile', can now search this data. this system pred is weird, so I expect in future it will get tucked into an umbrella system pred for advanced/rare stuff
- on update, all your existing image files are scheduled for the maintenance task. your 'has icc profile' will populate over time, and thumbnails will correct themselves
- .
- pixel hash now cached in database:
- the client database now keeps track of image 'pixel hashes', which are fast unique identifiers that aggregate all that image's pixels. if two images have the same pixel hash, they are pixel duplicates. this data is added on file import
- a new file maintenance task can generate it retroactively
- on update, all your existing image files are scheduled for the maintenance task. it'll work lightly in the background in prep for future duplicate file system improvements
- .
- boring search code cleanup:
- fixed a bug where the tag search cache could lose sibling/parent-chained values when their count was zeroed, even though they should always exist in a domain's lookup
- fixed up some repository reset code that was regenerating surplus tag search data
- with the new deleted files domain, simplified the new complex domain search pattern
- converted basic tag search code to support multiple location domains
- cleaned up some search cache and general table creation code to handle legacy orphan tables without error

next week

I'll take Christmas week off, so I only have two more proper weeks in the year. I would like to have basic pixel duplicate search working before then. Just a dropdown on the duplicates page for 'pair must/must not be pixel dupes' or similar. So, I will work on that and see if we can aim for a 'clean' release for end of year.

birthday

My todo list reminded me yesterday that I put out the first non-experimental beta of hydrus on December 14th, 2011. This is the rough 'start date' of the project and its birthday now. It will be ten years this week, which is pretty crazy.

Like a lot of people, 2021 was an odd year for me. I changed some lifestyle stuff, dropping some unhealthy habits, and also had some unexpected stress. After looking back though, I am overall happy with my work. Although I completed fewer big new projects than I hoped, and at times I felt bogged down in rewrites and fixes, the general performance of the client grew significantly this year. As well as a variety of new tag search and display options, the sibling and parent system was completely overhauled on several fronts, with the improved virtualised storage in the database and asynchronous real-time application calculation, and with that the autocomplete search finally supported 'perfect' sibling+parent adjusted tag counts in very fast time. Years-old sibling and parent application bugs were finally drilled down to and fixed. The tag lists across the program gained better sibling and parent display and commands. Wildcard searches became much faster too, and all sorts of tag and file search improved on smaller domains, sometimes by a factor of a thousand, even when a client had the whole PTR lurking in the background. We also moved to automatic repository account creation and improved serverside privacy, tags became easier to sort, file and database maintenance gained multiple new commands that saved a ton of time and inconvenience, the database learned to repair much of itself, system predicates became parseable in the Client API and editable in main UI, we moved from my duct-taped dev machines to github-built releases, the image renderer moved to a tiled system that allowed very fast zoom, sessions could grow much larger without CPU death and could save to disk with a fraction of their old write I/O, and a whole ton of other little fixes and quality of life improvements to every system.

I get a lot out of working on hydrus, and I hope to continue just like this. I appreciate everyone's feedback and support over the years. Thank you!

If you would like to further support my work and are in a position to do so, my simple no-reward Patreon is here: https://www.patreon.com/hydrus_dev

I had a great week. I managed to improve the duplicate filter search more than I thought, adding the ability to filter based on pixel duplicates and also pair similarity, and then I was able to rework the video scanbar so it sits inside the video frame and autohides based on mouse position.

The release should be as normal tomorrow.

dfcgvbhjnk

misc highlights

If you are interested in changing page drag and drop behaviour or regularly have overfull page tab bars, check the new checkboxes in options->gui pages.

If you are on Windows and have the default options->style, booting the client with your Windows 'app darkmode' turned on should magically draw most of the client stuff in the correct dark colours! Switching between light and dark while the program is running seems buggy, but this is a step forward. My fingers are crossed that future Qt versions improve this feature, including for multiplatform.

Thanks to a user, the twitter downloader is fixed. The twitter team changed a tiny thing a few days ago. Not sure if it is to do with Elon or not; we'll see if they make more significant changes in future.

I fixed a crazy bug in the options when you edit a colour but find simply moving the mouse over the 'colour gradient' rectangle would act like a drag, constantly selecting. This is due to a Qt bug, but I patched it on our side. It happens if you have certain styles set under options->style, and the price of fixing the bug is I have to add a couple seconds of lag to booting and exiting a colour picker dialog. If you need to change a lot of colours, then set your style to default for a bit, where there is no lag.

next week

I pushed it hard recently, and I am due a cleanup week, so I am going to take it easy and just do some refactoring and simple fixes.

I had a great week working on a variety of small cleanup, fixes, client api, sidecars, and UI quality of life.

The release should be as normal tomorrow.

- importer/exporter sidecar expansion (boring stuff):
- split the importer/exporter objects into separate importers and exporters. existing router objects will update and split their internal objects safely
- all objects in this system can now describe themselves
- all import/export nodes now produce appropriate example texts for string processing and parsing UI test panels
- Filename Tagging Options objects no longer track neighbouring .txt file importing, and their UI removes it too. Import Folders will suck their old data on update and convert to metadata routers
- wrote a json sidecar importer that takes a parsing formula
- wrote a json sidecar exporter that takes a list of dictionary names to export to. it will edit an existing file
- wrote some ui panels to edit single file metadata migration routers
- wrote some ui panels to edit single file metadata migration importers
- wrote some ui panels to edit single file metadata migration exporters
- updated edit export folder panel to use the new UI. it was already using a full static version of the system behind the scenes; now this is exposed and editable
- updated the manual file export panel to use the new UI. it was using a half version of the system before--now the default options are updated to the new router object and you can create multiple exports
- updated import folders to use the new UI. the filename tagging options no longer handles .txt, it is now on a separate button on the import folder
- updated manual file imports to use the new UI. the 'add tags before import' window now has a 'sidecars' page tab, which lets you edit metadata routers. it updates a path preview list live with what it expects to parse
- a full suite of new unit tests now checks the router, the four import nodes, and the four export nodes thoroughly
- renamed ClientExportingMetadata to ClientMetadataMigration and moved to the metadata module. refactored the importers, exporters, and shared methods to their own files in the same module
- created a gui.metadata module for the new router and metadata import/export widgets and panels

I was ill this week and am short on work time. I will spend tomorrow doing some more normal work instead of the release.

505 should be on the 9th of November. Thanks everyone!

----

I did just commit changes for users who run from source. I updated the easy-setup scripts and wrote variants for Linux and macOS. My test options are limited, so if you are an advanced Linux or macOS user, please do test these new scripts and let me know where things break or are otherwise confusing. The updated help walks through it all: https://hydrusnetwork.github.io/hydrus/running_from_source.html

I have thumbnails hidden by default on new pages, I have to do "pages - management & previevs - show" every time. Any way to keep them shown?

Are you on Qt5? You just see a widely stretched left-side panel, and the thumbnail panel is 0 pixels wide on the right? I am going to investigate a bug where in various situations, the thumbnail panel is disappearing in Qt5. Please let me know if I fix your issue in v505.

I had a great couple of weeks working on a variety of features. I fixed a heap of bugs, made it possible to search for files that have EXIF and other human-readable file metadata, and updated and translated my easy setup scripts, so it is now easy for any user on any platform to run the program from source.

The release should be as normal tomorrow.