Nerve Center

Version 336




os x






Last week, v335, was an important update. If you are updating from v334 or earlier, please go read that release post for your special one-time update instructions and update to v335 before you try this week. Once you have updated to v335, updating to anything newer is back to normal.

I had a good week. A bunch of py3-related bugs are fixed, and I improved some heavy-load throttling.

py3 fixes

Overall, last week's update went well. CPU and memory use and overall jank is down for all users, and afaik there weren't any permanent disasters in the update. This was a big relief for me.

That said, as expected, several unusual bugs related to py3's new unicode handling slipped through. Thank you for the reports. A common one was related to windows logging, another was in pixiv searching with kanji tags, another with videos with unicode metadata. I believe I have cleaned most if not all of it up. There may be a couple more out there, so please let me know if you get anything new. Otherwise, please resume any broken subs or retry any broken webm imports.

I also fixed the numerical tag ui, which was coercing all clicks to to zero or max stars, and which I messed up by accident!

better heavy workflow

The client now has a strict thread-level throttle on the number of importers that can work at once. It seperates the different sub-parts into their own max limits, but the overall total limit is about 20-40. This is in an effort to stop ui hang after accidental heavy import situations such as pasting 100+ queries into a gallery downloader. This is a 1.0 throttle and doesn't have any ui to report 'waiting on a slot' or anything. I don't expect 99% of users will even notice it, but if it affects you seriously, please let me know.

In a related note, for advanced users, gallery downloaders now have the ability to 'bundle' mass-pastes into one downloader, queueing up all the different gallery queries sequentially in the same object (rather than in many in parallel, which is what previously cluttered the ui and boshed the CPU). If you sometimes paste a bunch of single-file md5-queries at once, please check this new check menu item under the gallery downloader's cog menu.

full list

- ´╗┐fixed an issue where the numerical rating control was coercing all clicks to either the minimum or maximum allowable rating (e.g. 3/5 stars wasn't working)

- fixed some text file and process i/o, which was handling some unicode decoding/encoding incorrectly. it now mandates utf-8 in all cases

- fixed a referral url encoding problem that was stopping pixiv from downloading when the gallery page url had kana/kanji characters (from a search term)

- fixed a str vs bytes issue when loading the filename tagging panel

- fixed the delete button on the filename tagging quick namespaces panel (the edit and delete buttons are also now 'live' and will disable when nothing is selected)

- improved some json dump deserialisation code

- fixed a data-sorting issue that would appear with certain parsers in the edit parsers panel

- improved video metadata parsing, fixing an issue when the video has a 'title' row containing inconvenient data

- fixed some hex character processing for system preds

- added an advanced check item to the gallery downloader cog icon menu that will 'bundle' multiple query-pastes to the same single gallery downloader (this is helpful if you are pasting a whole bunch of md5 queries in one go and would rather one downloader work through them sequentially than 50+ separate ones blat your CPU simultaneously)

- the different kinds of importer worker threads now have several limits on the max number that can be working at once, to stop accidental ui overload when a hundred or more are in memory and all want to work at once (like after a big paste event or resuming after computer sleep). during periods of heavy import activity, the importers will now naturally space themselves out to smooth out the spike. the limits are hardcoded for now, let me know if it noticeably bottlenecks your situation

- made some menubar update code a bit less complicated and reduced how often it'll spam during heavy update

- the 'what to do?' buttons that appear in manage tags sometimes on a tag action got a simplification pass and are now on the new dialog system

- simplified my new dialog code significantly, clearing out redundant code and classes and pushing all okable/cancellable/vetoable closing checks through one single method

- wrote some new help.txt in the db dir about hanging startups

next week

I still have a bunch of small stuff going on, but I'd like to start on the Client API, the next 'big thing', as well. I expect this will involve several weeks of ui and clientside server prep before anything exciting happens.

Release Tomorrow!

I had a good week. I fixed several more bugs, including some more unicode issues and shutdown scheduling, and the big shutdown memory explosion some users were seeing, and added a couple of new small features. The client api work started well but is not ready for use yet.

The release should be as normal tomorrow.

Bugs Thread

Gonna start another thread because >>173 is becoming too big for most people.

In the tag manager, whenever multiple tags are selected for multiple files with different tags, if you press the del key, the program will still ask you if you want to delete them all or put them all in the selected files. This does not make sense, if I'm pressing the del key it's obvious that I want them gone.

Version 337




os x






The recent v335 was an important update. If you are updating from v334 or earlier, please go read that release post for your special one-time update instructions and update to v335 before you try this week. Once you have updated to v335, the process for updating to anything newer is back to normal.

I had a good week. A bunch more bugs are fixed, and I added some little things as well.

all misc this week

I fixed a variety of bugs this week. A unicode log printing bug still hanging over from the py3 update, some unusual webms (mostly ones that tend to be a single frame with music), and some network download file size checking.

I also believe I have completely fixed the shutdown memory explosion some users were seeing. I managed to recreate the bug and figured out some bad shutdown scheduling that was causing an event processing routine to go crazy. I cleaned a bunch of other shutdown code as well, including a couple instances of subscription processing not stopping promptly on a client shutdown. Overall shutdown (particularly the last moments just before the splash screen closes) should be a bit smoother for all users.

I also added some little new things. The duplicate system now merges/copies pending tags as well as 'current' tags (and the advanced service-wide content update routine does as well). The 'lain' image on the downloader easy-import page is now clickable to launch a file selection dialog. And the manual export files dialog has a new experimental checkbox for advanced users that tries out exporting symlinks (which requires running as admin mode in Windows, so I don't really recommend it for that). If you try this symlink export, let me know how it goes!

full list

- ´╗┐fixed another couple of unicode encoding problems with the logging and profiling code

- the logger now sticks a unicode BOM at the top of new log files to help text readers guess the utf-8 encoding

- fixed musical webm import when the video stream has no stated duration but the audio stream does and typically stretches out a 'single frame' video

- fixed some 'max size' download file size testing

- 'waiting on bandwidth' statements on network job controls now show the specific network context (like 'web domain:') they are waiting on (you might need to hover over to see the tooltip for this)

- the downloader easy-importer lain image is now clickable to launch a file selection dialog

- if you are in advanced mode, the manual file export dialog now lets you export symlinks with a new checkbox. this is experimental, so if you are interested, give it a brief test and let me know how it works for you

- duplicate content merging now applies to pending as well as current tags

- the duplicate filter now counts pending tags when saying which file has more tags

- advanced content updates now _copy_ both current and pending tags. the other actions now state what they do more clearly

- stopped printing long server error text on 304 and 404, where actual response content is uninteresting

- removed wx import that accidentally came into server space due to 2to3 check

- improved the path-fixing code that helps environment construction when launching external programs from non-windows frozen builds

- fixed a critical pubsub processing bug that kicked in at a certain stage of client shutdown. this should stop the post-shutdown-processing memory explosion certain users were seeing and should stop any ui jank in the last 0.2s of the program for everyone else

- improved some other shutdown memory cleanup that was sometimes leading to double-log-printing of exit statements

- did a full pass over the daemon scheduling code. it now reacts more responsively to various shut down situations

- reduced db disk cache aggression significantly and added more memory maintenance to the cache population process

- fixed an issue where subscriptions were not promptly responding to shutdown events

- fixed an issue where some delayed network jobs (e.g. while all network traffic is paused) were also not responding to shutdown events

- added a 'pubsub report mode' debug mode for simpler pubsub review

- the db is now less redundantly spammy on certain behind the scenes update notifications

- wrote a first version of the client api manager and permissions handling objects

- misc fixes, cleaned some shutdown code

next week

I started the Client API for real this week. The first 'manager' component is basically done, and it wasn't as complicated as I feared. I'll get started on some ui code for service review and authorisation management this coming week and otherwise just do some more small work to keep catching up on things.

Q&A Thread: For simple questions that don't need their own thread

Here you can ask questions so that the board is not clogged with small threads.

>>6021 has reached its bump limit, so I made a new thread.


Drag and drop windows with tag rules. Show two windows side by side and one window can be programmed with the rule "ADD tag foo" and the other one has the rule "REMOVE tag foo, ADD tag bar" and you can drag and drop files to them.

Deriving tags from regex of other tags/namespace tags. A file has the tag "filename:big_ugly_name" and we could regex that namespace for another tag.

Tag sets with hotkeys: save a set of tags under a hotkey so it's quick to add them to a file while filtering

Opaque window behind tag list in the corner so it doesn't get hidden by picture background

Option to default certain mime types to be excluded from slideshow and only open externally, will help with videos with odd codecs that don't preview in the slideshow correctly

Option to specify hamming distance in "find similar images", you can't change the option once it's in the filter window and you have to enter the hash manually in the "system:similar to" option