Why does JCB change two column sizes of the #__assets table? #616
Labels
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: joomla/Component-Builder#616
Loading…
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Steps to reproduce the issue
Install component builder
Expected result
Component Installed
Actual result
The core database table #__assets has its structure changed
https://github.com/vdm-io/Joomla-Component-Builder/blob/staging/admin/sql/install.mysql.utf8.sql#L2434-L2441
Additional comments
This is wrong in so many ways - I am speechless
We both are speechless... so lets talk.
Resolving this issue five years ago was not possible, and today it is not that simple, or straightforward. I was simply at the time unwilling to wait for Joomla, no one took me seriously when I asked for this change (not that I blame them..), but today I know the current team better, and probably we can fix this is some way or another.
Simple fact is JCB can build large permission structures per/component that are to large for the current column size, take this one that is public for example:
https://github.com/Llewellynvdm/Joomla-Cost-Benefit-Projection/blob/master/admin/access.xml
So when you save that permissions structure, the JSON gets truncated and the permissions break. Yet, that is still a small one... I have a number of projects where that is far larger. SO yes, I know the permission structure itself is the pain... and enlarging the column does not fix the problem. But for me and many other it did allow us the freedom we need, and we will probably continue this way until Joomla improves this area.
That is the power of JCB every time you compile a component, it gets written from the ground-up and so introducing changes, and fixes are much faster, and easier to navigate. Fix JCB and you fix 5000+ extensions, of which 200 is just my own.
I think very few have realized what JCB has done, and is doing...
Giving a notice that we changed this table could be an improvement, but removing the change on uninstall of the component could cause issues, since there could be more then one component installed that needs this larger column, and reverting the change when the one component is uninstalled will cause the other to break.
Now I know the point of breaking the component up in smaller components is the current best practice, but I do not what to add that limitation forcefully to JCB, but I do give notice of this when tables go over 50 fields. We could add more notices... yet why not just enlarge the column, so I will go back to Joomla and see if they are willing.
I am going to try and answer this in multiple parts.
1. Dont hack the core
The change you are making to core files should never be done. No extension should ever touch any of the core files ever. And yes changing the structure of a core database table is hacking the core
By hacking the core you have done a disservice to your users and their users. Joomla may contain updates the core database tables and you have just broken that and potentially broken their sites during an update. In the recent past we have had updates to address utf8 and with joomla 3.10 and 4 to address null values. These updates have of course been written on the assumption of a known database structure.
Yay you just broke it. Guess who will get the blame and who will have their reputation damaged - it won't be you.
2. Don't hide the hacks
Perhaps its just my inquisitive mind but I always look at changelogs and that's where I saw a hint at this hack. There is no mention of it anywhere in the documentation or installation notes.
3. Clean up after yourself
When I uninstall an extension I expect it to remove everything it did when it was installed. Some extensions might leave any data tables but they will often inform you of that. You have an uninstall sql but it does not revert the hack to the database. It will stay there forever or until a joomla update breaks the site.
Yay you just broke it. Guess who will get the blame and who will have their reputation damaged - it won't be you.
4. Performance
There is a massive performance difference between a varchar (512) and a mediumtext ref High Performance Mysql. So now not only do you not clean up after yourself but you are degrading the performance of every joomla web site that ever tried your code. To be clear the change has impact on the entire web site not just your code.
Yay you just broke it. Guess who will get the blame and who will have their reputation damaged - it won't be you.
5. Code
It doesnt matter if this is your 1st or 90th component. This is wrong and you should change it. Having an ACL rules table with 1000 options is crazy crazy crazy (see image below)
6. You don't need it
Now I get to wondering why you even need the change. The column in the database only stores the acl rules that are changed - it doesn't save every single one of the 1000 rules it just saves the changes you just made.
So yes it is a big deal. I really don't care what people do in their own code. I care what people do that impacts the joomla code. This hack of yours has a serious impact on any joomla site it comes into contact with. Like all infections, the hack needs to be eliminated as a priority.
Llewellyn moved the image so the thread is shorter to read, since yes it is a very large image.... 👍
https://imgur.com/QFLgnG0
Thank you for all the valuable information you shared... very helpful.
Hope you understand that I stand by my change to the asset table that it is needful at times, as a temporary fix 5 years ago with the hope that it will serve as a simple work around until Joomla improves this area, which they must do and hopefully will soon.
Even though I don't agree with all you said... and somethings I will just ignore.
I have seen a solution that could possibly be the only amicable path forward where we remove the automatic change, and let you the developer choose which path you want to take, and it seems like there are three paths. I will share more here soon...
I am working on moving the filters above the list views at the moment, and when that is done, I will take this next.
Disappointed that you changed the title to try and hide the severity of the issue
To anyone who comes this way, we have added the uninstall revers code to the default JCB path, and we will soon add more GUI options to control this, I will continue to link this ticket to any related changes.
The conclusion
Okay, so I finally had time to research this issue more and the Joomla column is now varchar(5120)
The var char size limit is based on the row limit which is about 65535 var chars. This basically means that Joomla is not close to the limit yet with the varchar(5120), yet I was told that Joomla does not plan to enlarge this any time soon.
So why do they insist to use the varchar instead of text, or medium text like JCB does? (which is no big deal at all) Basically they say since varchar is stored in the row array and not at another location in memory, so it has a fractional faster access. So how much faster is this access? Well in this old article there seems to be very little difference whatsoever. Honestly this whole issue seems to result in debates with strong options on both sides of the fence, and clearly the benchmarks made differ seemingly due to many factors not even related to the issue.
You see the point made is not that the table is larger (since to this I/we would agree, I mean we need it to be larger to hold or larger json object)... no their point is that the table performs slower, but that is just not true. Your TEXT or MEDIUMTEXT works by pointers... and pointers are the very power of C and C++ and what gives those languages the speed we all long for in other higher level languages like PHP.
The facts!
So what facts do we have... 30 actions in the access permissions of a component is enough to push up against the current limit of varchar(5120). You see each action with a length of +-20 characters and the normal 8 groups while you set each group independently with very little inheritance, then we get 4800 characters (20x30x8=4800). To get 30 actions in a component is very easy, most components in JCB with moderate use of permissions have around 100 actions if the component has 10 views, JCB itself has 54 views which gives us 543 actions, you do the math... that is 86880 characters in WORSE case. What is worse case? Glad you asked, worse case as I explained before is when you set each group independently with very little inheritance, then you get worse case.
So in JCB we have now added the following options as the current solution.
We cannot wait for Joomla since most normal users will never have these needs, unless they use our advanced components, therefore we need to fix this with JCB. So until Joomla sets a better solution on the table this is how I think we can do this.
We added a global and per/component switch to allow you to control the outcome.
This means you have three options
I will push the code for this very soon as part of the v2.12.2 release due this week.
Just to validate that LENGTH AND CHAR_LENGTH is generally the same.
Since we are working with a json object that is normally only build-up of ISO basic Latin alphabet these two values of the string length and the char length is normally therefore the same.
Also note that when we set a column length to MEDIUMTEXT, it does not USE that much space for each row!
This is an assumption that is made by those who do not understand how smart MySql really is. Here you can read more about this in the manual. But basically the TEXT type data structure only use what it needs per row, and not bigger then the type selected. The TEXT type is 65,535 bytes ~64kb and for the MEDIUMTEXT is 16,777,215 bytes ~16MB. We have seen in a worse case a component can have more then 65535 characters, and that is why we used MEDIUMTEXT since we know MySql will only use what it needs per row and will not bloat the table, or massively impact the performance of the table.
Further more we understand that "text" and "varchar" is not the same thing.
Yet we often see that 1 byte may hold 1 character, except with multi-byte characters. Which then does indeed make the length of a string not always the same as the number of bytes we would need based on the Server/DB/Table charset being UTF-8 Unicode (utf8mb4) or something else. But all that this means is that a 50 character string (which may have multi-bytes) could need 80 bytes of space. So in our case working with the string length to detect the length required is the safe option. Since if a string can be more than 5120 characters it will not be less bytes, just more. Now even more for the benefit of all those who is following this tread we are in the assets rule column just working with a json object that is normally only build-up of ISO basic Latin alphabet. This means we very often will see the char length and the string length be the same, like seen above.
@vdm-io/jcb-core pease test the new Intelligent assets table fix, and let me know so we can push this smart update out 👍
First Test:
Second Test:
Third test:
I have tested this already and it all worked perfectly, let me know ;)
Tested successfully!