Diko said:
EdelweissPirate said:
The association between Lightroom's database and its performance issues exists only in your imagination. It's a mistake to claim that SQLite is an a priori cause of Lightroom's performance issues.
Do me a favour and go try putting 20 healing spots on an image. Do that for the next 10 out of 50. Let me know if it gets sluggish when you apply to the next image. Also walk around the images.
And if you ask me what that has to do with the SQLite I think our debate would end here ;-)
Umm, okay. Would the debate end because you don't understand what would be in the database in such a case?
This is exactly the sort of thing that is
not exercising the database (unless Adobe's engineers are grossly incompetent, but if you believe that then why are you even giving their products a second glance?)
There are several databases in Lightroom:
[list type=decimal]
[*]The catalog, which stores references to all the images on disk, as well as a cache of the modification instructions for those images.
[*]The "Previews" (1:1 and Standard) cache, which caches rendered 1:1 and "standard" size renders of the final images
[*]The "Smart Previews" cache, which caches the original raw data compressed so that basic changes can be made in the UI before the original RAW file is pulled up from disk and if the original RAW file is missing
[/list]
There might be more in some circumstances, but those are the biggies. Also note that almost everything above is categorized as "cache". That means, the actual source of record for that information is elsewhere - for 1:1 previews, that is the original RAW file plus the list of instructions applied to it as described in the sidecar file. These databases' whole existence is because
they fixed performance problems in the non-DB-based early versions of Lightroom.
I'm assuming you are trying to say that the first database (the catalog) is the issue here, as it is what contains the stack of changes per image pulled from disk (where those healing brush source and destinations are, and the geometry of the spots). That is a fairly small amount of data to store (the mask of the healing image, which should be an 8-bit-per-pixel RLE-compressed bitmap unless, again, Adobe's engineers are wholly incompetent, is the biggest bit), and it is stored alongside all the other Develop instructions stack for each image.
Let's look at what happens when you go from image to image in the Develop module:
[list type=decimal]
[*]The Smart Preview is pulled up, if in cache, which is by all accounts instantaneous
[*]The RAW file is pulled into memory if available, replacing the Smart Preview. This may take some time as it is a disk access.
[*]The list of changes is pulled out of the Catalog database
[*]Each change is applied to the image in turn, using the rendering engine appropriate for the change (ex, rotator and cropper, spot removal, exposure, USM for sharpening, etc).
[*]The rendered image is displayed on screen
[/list]
If the third step above were a problem, it would be a problem no matter what is in that list. For instance, rotate all your images slightly, adjust the exposure, add a contrast curve, etc. From a database perspective, 99% of the cost of step 3 is the lookup - finding the row in the database - while the rest of the cost is pulling the data out. I'm not sure about the specific RDBMS table structure in Lightroom's database, but assuming the developers know what they are doing, there are only a few possibilities that really make sense. I'd guess that likely the list of changes to apply is kept in a child table related to the parent "Photo" table.
But, we don't see such a drag just by doing "any" set of changes to the images. We need to do computationally-intense (relatively) changes to result in a measurable slowdown.
This is exactly what we would expect if #4 above is the bottleneck, because that is the first point in the whole process where an image that has ten crop/rotates and ten exposure adjustments looks different from one which has twenty healing brush adjustments.
Now I read carefully the posts and tend to believe that most people here don't realize that there IS a scenario in which all performance issues are not related. Not caused by one and the same bottleneck.
Also all more knowledgeable IT guys that have posted seem to omit the possibility that everything is the cause.
- SQLlite
-LUA (I forgot to rant about it in my previous posts)
- Cocoa and Silver
Each of the above has its own merits and setbacks. If I recall correctly the memory leaks issues began after the transition to LUA, which was for better API for users to create custom plugins (only presumption) and also as they stated in the mentioned presentation to better get core API functions calls easily . But as noted - it is metalanguage. It needs an interpreter (virtual machine) to run. It is more like a script language than a true language.
I've been involved with enough refactors and language rewrites to know that the language chosen can make a big difference in performance, but "interpreted" languages are not necessarily worse for a task. If you are dealing with evolving algorithms, in fact, moving to a higher-level, "less efficient" language will often allow for significant performance boosts at the algorithm level which would have been impractical with the "more efficient" language. We have a school scheduling engine which I've rewritten a few times now, and as one example moving from C++ to Java with the original algorithm intact cost us about 20% performance (this was back in Java 4 days, without the great JIT compilers we enjoy now), but allowed us to put a much more elegant algorithm in place which yielded gains at the 10,000% level and in some cases better (a build on track for 15 years to complete in the old codebase completing in 200 milliseconds).
Now, I can't speak to Adobe's use of Lua. But, it can be compiled all the way to machine code, and Lua can be run in a JIT-supporting bytecode interpreter, although the quality of the Lua JITs might not be anywhere near those found in .Net or Java VMs. I also don't know how "deep" Adobe's use of Lua is, if it is just at the UI and API levels or if it actually runs any of the image manipulation algorithms. I would not expect it to be used in the latter case, generally.
As for any of these being the primary bottleneck in the "moving from image to image" performance problem, I'd say Lua is much more likely to be the problem than SQLLite. Not sure what Cocoa has to do with it (this is a problem on Windows too, right?) or what you mean by "Silver" (Silverlight? Wouldn't Adobe more likely have Flash in there if anything?)
Each one of us has experienced one or another issue. Adobe should aim at updating its core engines to include more than SSE2 (2001).to something more advanced like AVX2 (2013), which was 4 years ago, which means that if you edit photos on so old computer you need an upgrade or at least don't need that much professional software.
Agreed that the use of SSE2 only (and none of the expanded newer instruction sets) is, if true, definitely going to be a performance problem, and will show up exactly where we see it: applying adjustments is just plain slow. Adobe is likely using a modular architecture which would allow them to compile key image-processing modules with several levels (SSE2 at one end and the latest at the other end, perhaps nothing-newer-than-five-years in the middle) and pull in the dylib/DLL appropriate to the host machine's architecture at runtime. This is well-trodden territory, and would not require Adobe to "write off" even customers on the oldest hardware (although at a cost of potentially having to code these low-level optimizations three times instead of once). But, as you point out later, it seems like "SSE2 only" is another one of those myths, a tidy little story people tell themselves to explain why Lightroom's performance sucks in several situations.
IMHO, more likely the issue is not with the database, or with the use of Lua, or even with the raw performance of the image manipulation steps (because I suspect they are tightly tuned to do what they do without having quality issues). More likely the root issue is that Lightroom constantly
throws its work away instead of storing it or caching it. At the same time, it sits nearly 100% idle much of the time, wasting potential CPU cycles.
There is absolutely NO reason why if I apply a bunch of changes to one image in Develop (which it renders on screen), click on another image in the filmstrip, then click back, that there should be ANY delay in showing me the fully rendered image with all its changes.
It just did the full render. But, it threw everything away when I went to look at another image in the filmstrip for reference.
There is also absolutely NO reason why if I am flipping through images in the Library or Develop module and pause for a second on one image, I shouldn't be able to flip to the next two or three without any rendering delay. Instead, while I am looking at image 27/300, Lightroom is twiddling its thumbs and dreaming of unicorns. It should be anticipating my next move: the guy just went from image 1 to 2, then 2 to 3, etc, then 26 to 27; they are likely going to want 28, 29, and 30 next).
The problem with the "the issue is the database" myth is that to do either of the above
real, actual improvements to Lightroom performance, means putting
more cached processed data into a cache database (which, ultimately, is pretty much what all of the Lightroom databases are, other than the specific catalog information). And, to do that, Adobe needs to improve its underlying database management procedures (there is no reason why Lightroom can't automatically detect changes on disk without us having to tell it to rebuild the cache via "Synchronize Folder", and there is no reason why we shouldn't have significantly better control over how long 1:1 previews and the like hang around in the caches, even to the point of being able to remove a particular image from caches). The issue isn't "too much database", it is "too much calculated on-the-fly / too little database".