joomla/Component-Builder
joomla/Component-Builder
38
20
Fork 10
Code Issues 55 Pull Requests Projects 3 Releases Wiki Activity

Problem with Export/Import of components Field mapping #587

New Issue
Closed
opened 2020-08-13 14:00:10 +00:00 by duetnet · 10 comments
duetnet commented 2020-08-13 14:00:10 +00:00 (Migrated from github.com)
Copy Link

Steps to reproduce the issue

Export component which uses SUBFORM fields and ADMIN FIELD RELATIONS from Development Site A
Import component to Development site B.

Expected result

Field ID mappings in Development site B in SUBFORM fields and ADMIN FIELD RELATIONS correctly reflect new IDS as a result of changes in IDs during import.
Please see Additional comments below for detail explanation.

Actual result

Field IDS on Development Site B in SUBFORM fields and ADMIN FIELD RELATIONS still reflect original field IDS as per Development Site A
Please see Additional comments below for detail explanation.

System information (as much as possible)

• OS Name & Version: Win10Pro
• MySql Version: 5.5.5-10.4.13-MariaDB
• Apache Version: 2.4.41 win 64b
• PHP Version: 7.4.8
• Joomla Version: 2.9.20
• JCB Version: 2.11.4 and earlier…
• Browser: Firefox 75.0 64b

Additional comments

To simplify the problem, I will try to illustrate it by using the following example.

I use JCB do develop a NEW component on a localhost in Win 10 using Xampp on Laptop A.

It is a "clean" system with only Joomla, Akeeba, JCB Component Builder installed and the components Demo (public) and Field Type Updater (public) imported.

The NEW component has a single admin view with the following fields:

Field ID, Field Type, Name, Label, Property 'fields' used in Subform
50, Text, Txt_First, First Name,
51, Text, Txt_Last, Last Name,
52, Subform, Sf_Address, Address, 53,54,55
53, Text, Txt_Street, Street,
54, Text, Txt_Town, Town,
55, Text, Txt_Zip, Zip,

The Admin View Field Relations has the following definition:

List Field, Column Name, Join field, Area, Join Type, Set
Txt_First, Full Name, Txt_Last, View, Custom Code, [field=51],[field=50]

Sofar so good, I test component and it works fine….

Then I export component on Computer A and import into JCB installation on COMPUTER B.
Please note that Computer B has Joomla and components JCB Component Builder, Demo (public) and Field Type Updater (public) AND 2 other JCB components already installed.

The fields imported into JCB on computer B now reflect the following fields:

Field ID, Field Type, Name, Label, Property 'fields' used in Subform
95, Text, Txt_First, First Name,
96, Text, Txt_Last, Last Name,
97, Subform, Sf_Address, Address, 53,54,55 (Note that the FIELD IDS are still those from Computer A)
98, Text, Txt_Street, Street,
99, Text, Txt_Town, Town,
100, Text, Txt_Zip, Zip,

The Admin View Field Relations still reflects the Computer A field definitions:

List Field, Column Name, Join field, Area, Join Type, Set
Txt_First, Full Name, Txt_Last, View, Custom Code, [field=51],[field=50]

If I compile and run the component I have errors in the list view shows only the text '[field=51],[field=50]' in the "Full Name" column and the SUBFORM do not reflect the correct fields.

I have to manually REMAP and update:
• the field IDs for the SUBFORM ‘Sf_Address’ to now read ‘89,99,100’ and
• the Field Relations ‘Set’ in the Admin Relations View to reflect the code ‘[field=96],[field=95]’.

This is a simplistic example, but it is typical of the issues I am experiencing with my components which are in development. Subform fields and Field Relations do not change to reflect new field IDs.

According to recent discussions with Lewellyn this should not be happening, but it still is...

What I am doing, which maybe impacting this, is that I also create new FIELD TYPES based on existing JCB FIELD TYPES, mainly because I add extra XML fields to the FIELD TYPE which I need in my coding. I then use the new FIELD TYPE as a basis for some of my fields…
..for example, I predefine a CUSTOM field with additional XML field to set up filters when listing records…and then I read this XML field in the coding of the FIELD TYPE to filter data for the list. I then use this new FIELD TYPE as a bases for many different custom FIELDS.

However, I have tried the above export and import actions with just using standard JCB FIELD TYPES and the problem is still happening.

It seems to me, that the even though the JCB export/import remaps field IDS correctly for ADMIN VIEWS and their corresponding views, this remapping does not seem to happen a SUBFORM field (in the XML property 'fields') or the FIELD RELATIONS VIEW…

The only time I do not need to REMAP is if I import onto Computer B which has a JOOMLA/JCB set up identical to Computer A before I created the component…because then the ‘imported’ field IDs are identical on Computer B to those on Computer A, and no remapping is required.

I am finding this a BIG problem as I tend to test my components on a separate environment and each time I move the component (export/import) I have to manually remap all the fields on the imported environment…and this is a lot of work if one uses a many subforms and/or admin relations functionality, which I tend to do....

I can work around the admin relations view by hard coding the relevant info eg. "echo $item->txt_last . ',' . $item->txt_first", but this does not solve the issue with subforms...

So, has anyone else experienced this problem? Any suggestions as to what I might be doing incorrectly? Any ideas how I can make this export/import process simpler...

Thanks

### Steps to reproduce the issue Export component which uses SUBFORM fields and ADMIN FIELD RELATIONS from Development Site A Import component to Development site B. ### Expected result Field ID mappings in Development site B in SUBFORM fields and ADMIN FIELD RELATIONS correctly reflect new IDS as a result of changes in IDs during import. Please see **Additional comments** below for detail explanation. ### Actual result Field IDS on Development Site B in SUBFORM fields and ADMIN FIELD RELATIONS still reflect original field IDS as per Development Site A Please see **Additional comments** below for detail explanation. ### System information (as much as possible) • OS Name & Version: Win10Pro • MySql Version: 5.5.5-10.4.13-MariaDB • Apache Version: 2.4.41 win 64b • PHP Version: 7.4.8 • Joomla Version: 2.9.20 • JCB Version: 2.11.4 and earlier… • Browser: Firefox 75.0 64b ### Additional comments To simplify the problem, I will try to illustrate it by using the following example. I use JCB do develop a NEW component on a localhost in Win 10 using Xampp on Laptop A. It is a "clean" system with only Joomla, Akeeba, JCB Component Builder installed and the components Demo (public) and Field Type Updater (public) imported. The NEW component has a single admin view with the following fields: Field ID, Field Type, Name, Label, **_Property 'fields' used in Subform_** 50, Text, Txt_First, First Name, 51, Text, Txt_Last, Last Name, 52, Subform, Sf_Address, Address, **_53,54,55_** 53, Text, Txt_Street, Street, 54, Text, Txt_Town, Town, 55, Text, Txt_Zip, Zip, The **Admin View Field Relations** has the following definition: List Field, Column Name, Join field, Area, Join Type, Set Txt_First, Full Name, Txt_Last, View, Custom Code, [field=51],[field=50] Sofar so good, I test component and it works fine…. Then I export component on Computer A and import into JCB installation on COMPUTER B. **Please note** that Computer B has Joomla and components JCB Component Builder, Demo (public) and Field Type Updater (public) ****AND 2** other** JCB components already installed. The fields imported into JCB on computer B now reflect the following fields: Field ID, Field Type, Name, Label, **_Property 'fields' used in Subform_** 95, Text, Txt_First, First Name, 96, Text, Txt_Last, Last Name, 97, Subform, Sf_Address, Address, 53,54,55 (Note that the FIELD IDS are still those from Computer A) 98, Text, Txt_Street, Street, 99, Text, Txt_Town, Town, 100, Text, Txt_Zip, Zip, The Admin View Field Relations still reflects the Computer A field definitions: List Field, Column Name, Join field, Area, Join Type, **_Set_** Txt_First, Full Name, Txt_Last, View, Custom Code, **_[field=51],[field=50]_** If I compile and run the component I have errors in the list view shows only the text '[field=51],[field=50]' in the "Full Name" column and the SUBFORM do not reflect the correct fields. I have to manually **REMAP** and update: • the field IDs for the SUBFORM ‘Sf_Address’ to now read ‘89,99,100’ and • the Field Relations ‘Set’ in the Admin Relations View to reflect the code ‘[field=96],[field=95]’. This is a simplistic example, but it is typical of the issues I am experiencing with my components which are in development. Subform fields and Field Relations do not change to reflect new field IDs. According to recent discussions with Lewellyn this should not be happening, but it still is... What I am doing, which maybe impacting this, is that I also create new FIELD TYPES based on existing JCB FIELD TYPES, mainly because I add extra XML fields to the FIELD TYPE which I need in my coding. I then use the new FIELD TYPE as a basis for some of my fields… ..for example, I predefine a CUSTOM field with additional XML field to set up filters when listing records…and then I read this XML field in the coding of the FIELD TYPE to filter data for the list. I then use this new FIELD TYPE as a bases for many different custom FIELDS. **However**, I have tried the above export and import actions with just using standard JCB FIELD TYPES and the problem is still happening. It seems to me, that the even though the JCB export/import remaps field IDS correctly for ADMIN VIEWS and their corresponding views, this remapping does not seem to happen a SUBFORM field (in the XML property 'fields') or the FIELD RELATIONS VIEW… The only time I do not need to REMAP is if I import onto Computer B which has a JOOMLA/JCB set up identical to Computer A before I created the component…because then the ‘imported’ field IDs are identical on Computer B to those on Computer A, and no remapping is required. I am finding this a BIG problem as I tend to test my components on a separate environment and each time I move the component (export/import) I have to manually remap all the fields on the imported environment…and this is a lot of work if one uses a many subforms and/or admin relations functionality, which I tend to do.... I can work around the admin relations view by hard coding the relevant info eg. "echo $item->txt_last . ',' . $item->txt_first", but this does not solve the issue with subforms... So, has anyone else experienced this problem? Any suggestions as to what I might be doing incorrectly? Any ideas how I can make this export/import process simpler... Thanks
Llewellyn commented 2020-08-13 23:31:23 +00:00
Copy Link

@duetnet thank you for the clear explanation. My problem is that I have not been able to replicate this issue, since all JCB packages, like JCB itself and other are imported with the same remapping engine and it does indeed remap sub-forms, and relations correctly.

So one option, share with us a basic JCB packages that you don't mind sharing. Then I can import it into a black Joomla with a new JCB and see what happens, since the moment I can replicate the issue I can fix it. Until then it is very hard to see what goes wrong.

Lets fix this issue so you can spend your time one more important things then remapping ID's ugghh..

@duetnet thank you for the clear explanation. My problem is that I have not been able to replicate this issue, since all JCB packages, like JCB itself and other are imported with the same remapping engine and it does indeed remap sub-forms, and relations correctly. So one option, share with us a basic JCB packages that you don't mind sharing. Then I can import it into a black Joomla with a new JCB and see what happens, since the moment I can replicate the issue I can fix it. Until then it is very hard to see what goes wrong. Lets fix this issue so you can spend your time one more important things then remapping ID's ugghh..
duetnet commented 2020-08-15 12:45:25 +00:00 (Migrated from github.com)
Copy Link

@Llewellynvdm
I have spent the last 2 days trying to prove what I claimed above, by creating a component with a single view using a subform, exporting and importing and just could not find a fault with the remapping…

So I went back to my original component import/export and tried to understand what went wrong and I have a clue, but it is hard to prove other than providing you with the actual export and import field lists as per my spreadsheet…
DuetNet_Component_Export_Import_Matching.xlsx

I will try to explain (please refer to attached spread sheet).

Each PC listed starts off with Joomla standard, JCB Component Builder and components Demo (public) and Field Type Updater (public). My component is the next component created….

When I export from PC A (see worksheet Compare – columns ‘PC A Bef export’), all the field are listed in ID sequence. PC A was still using JCB v 2.11.2 and Field Updater 2.9.22 and Demo 2.0.1.

All the subforms are field type 43. There are 13 subform fields in total. The first subform appears at ID 1140 (note the XML Fields and their IDS).

Fields 23 to 1051 are JCB fields. My fields start at 1052.

Also note that export shows 3 extra fields, i.e.

  • 1024 – Name Required, Look component
  • 1049 - Privacy (Makita's) and 1050 - Terms & Conditions. The origins of these Fields I do not recall, other than that they may have been from previous JCB example components I had previously installed and removed somewhere in the past.

PC B is set up with JCB v 2.11.3 and Field Updater 2.9.22 and Demo 2.0.3.

I export the Package from PC A and then import the Package into PC B (see worksheet Compare – ‘PC B 1st import’)
Fields 23 to 1048 are JCB fields. My fields start at 1049.

Note that the extra JCB fields on PC A (1024, 1049 and 1050) are missing on PB C.

The field name sequences between PC A export and PC B import correspond from PC A field 1052 to 1343, but from PC A field IDs 1344, the field names sequence differs between PC A export and PCB import. I think this may be because of the sequence in which I had originally created the fields and also the sequence of the admin views on PC A (which I had changed at some time earlier).

With the import, it seems that the fields get sequenced in the order which they are used in the admin view. (Not sure, just a guess!!)

If you go to the first subform (PC A field 1140) you will see it is field 1137 on PC B.
NOW Note that the XML Fields on PC B are the SAME as on PC A, although the actual fields on PC B have different IDs, So the fields have not been remapped in the subforms.

  • PC A Id -> PC B Id
  • 1141 -> 1138
  • 1142 -> 1139
  • 1143 -> 1140

If you check ALL the subforms between PC A export and PC B import you will see the same issue has happened throughout. (the field references in the Admin View Relations have also not changed, just too much info to list here)

I then manually reworked/remap all the subform XML fields and field conditions in PC B and created an package export (see worksheet Compare – ‘PC B reworked & exported’)

I then set up PC C with JCB v 2.11.4 and Field Updater 2.9.22 and Demo 2.0.3.

This is where I attempted to create a single admin view as per my original example about the issue…but could not get the same effect and the trashed all my examples…

I decided to import to PC C the exported package from PC B.

The result is worksheet Compare – ‘PC C 1st import’. Because of the previous example attempts, my fields for the component imported from PC B only start at ID # 1075.

Nevertheless, you will note that all the subform fields (and the Admin View Relations) are correctly renumbered with the new numbers….so the remapping has now worked correctly!!!

So..in conclusion..

It appears that:

  1. When the Export from PC A took place, fields 1012, 1050 and 1051 were not exported
  2. On import to PC B, the fact that these fields were missing meant that the ID numbers of my component began 3 numbers earlier that the export from A
  3. Because these numbers had changed downwards, somehow (just a guess), this caused a problem with field ID pointers during the import and so the subform XML fields (and Admin Relations) are not updated.
  4. I initially thought that this may be due to me creating my own FIELD TYPES modeled on JCB standard field types, but this is not so, because when I moved from PC B to PC C, my ID numbers began at a later stage (i.e. 1075) and I was still using my own FIELD TYPES, but the remapping did not have a problem…all the subforms (and Admin Relations) were correctly remapped.

My only clues are

  • that the renumbering was affected by where the JCB fields, and my own fields began or
  • The fact that I had not used IDs in the sequence created on the admin fields, resulting in the changes in field sequence between PC A and PC B my have triggered some confusion in the renumbering process….

Sorry, this is my best attempt at trying to determine what is causing my issue.
If you wish I can forward you my entire PC A export to see if you can debug the import. Please let me know.

Thanks again for your time (and for JCB :-))

@Llewellynvdm I have spent the last 2 days trying to prove what I claimed above, by creating a component with a single view using a subform, exporting and importing and just could not find a fault with the remapping… So I went back to my original component import/export and tried to understand what went wrong and I have a clue, but it is hard to prove other than providing you with the actual export and import field lists as per my spreadsheet… [DuetNet_Component_Export_Import_Matching.xlsx](https://github.com/vdm-io/Joomla-Component-Builder/files/5078652/DuetNet_Component_Export_Import_Matching.xlsx) I will try to explain (please refer to attached spread sheet). Each PC listed starts off with Joomla standard, JCB Component Builder and components Demo (public) and Field Type Updater (public). My component is the next component created…. When I export from PC A (see worksheet Compare – columns ‘PC A Bef export’), all the field are listed in ID sequence. PC A was still using JCB v 2.11.2 and Field Updater 2.9.22 and Demo 2.0.1. All the subforms are field type 43. There are 13 subform fields in total. The first subform appears at ID 1140 (note the XML Fields and their IDS). Fields 23 to 1051 are JCB fields. My fields start at 1052. Also note that export shows 3 extra fields, i.e. - 1024 – Name Required, Look component - 1049 - Privacy (Makita's) and 1050 - Terms & Conditions. The origins of these Fields I do not recall, other than that they may have been from previous JCB example components I had previously installed and removed somewhere in the past. PC B is set up with JCB v 2.11.3 and Field Updater 2.9.22 and Demo 2.0.3. I export the Package from PC A and then import the Package into PC B (see worksheet Compare – ‘PC B 1st import’) Fields 23 to 1048 are JCB fields. My fields start at 1049. Note that the extra JCB fields on PC A (1024, 1049 and 1050) are missing on PB C. The field name sequences between PC A export and PC B import correspond from PC A field 1052 to 1343, but from PC A field IDs 1344, the field names sequence differs between PC A export and PCB import. I think this may be because of the sequence in which I had originally created the fields and also the sequence of the admin views on PC A (which I had changed at some time earlier). With the import, it seems that the fields get sequenced in the order which they are used in the admin view. (Not sure, just a guess!!) If you go to the first subform (PC A field 1140) you will see it is field 1137 on PC B. **NOW Note** that the XML Fields on PC B are the SAME as on PC A, although the actual fields on PC B have different IDs, So the fields have not been remapped in the subforms. - PC A Id -> PC B Id - 1141 -> 1138 - 1142 -> 1139 - 1143 -> 1140 If you check ALL the subforms between PC A export and PC B import you will see the same issue has happened throughout. (the field references in the Admin View Relations have also not changed, just too much info to list here) I then **manually** reworked/remap all the subform XML fields and field conditions in PC B and created an package export (see worksheet Compare – ‘PC B reworked & exported’) I then set up PC C with JCB v 2.11.4 and Field Updater 2.9.22 and Demo 2.0.3. This is where I attempted to create a single admin view as per my original example about the issue…but could not get the same effect and the trashed all my examples… I decided to import to PC C the exported package from PC B. The result is worksheet Compare – ‘PC C 1st import’. Because of the previous example attempts, my fields for the component imported from PC B only start at ID # 1075. Nevertheless, you will note that all the subform fields (and the Admin View Relations) are correctly renumbered with the new numbers….so the remapping has **now** worked correctly!!! So..in conclusion.. It appears that: 1. When the Export from PC A took place, fields 1012, 1050 and 1051 were not exported 2. On import to PC B, the fact that these fields were missing meant that the ID numbers of my component began 3 numbers earlier that the export from A 3. Because these numbers had changed downwards, somehow (just a guess), this caused a problem with field ID pointers during the import and so the subform XML fields (and Admin Relations) are not updated. 4. I initially thought that this may be due to me creating my own FIELD TYPES modeled on JCB standard field types, but this is not so, because when I moved from PC B to PC C, my ID numbers began at a later stage (i.e. 1075) and I was still using my own FIELD TYPES, but the remapping did not have a problem…all the subforms (and Admin Relations) were correctly remapped. My only clues are - that the renumbering was affected by where the JCB fields, and my own fields began or - The fact that I had not used IDs in the sequence created on the admin fields, resulting in the changes in field sequence between PC A and PC B my have triggered some confusion in the renumbering process…. - Sorry, this is my best attempt at trying to determine what is causing my issue. If you wish I can forward you my entire PC A export to see if you can debug the import. Please let me know. Thanks again for your time (and for JCB :-))
duetnet commented 2020-08-15 12:47:38 +00:00 (Migrated from github.com)
Copy Link

Sorry, accidentally close this issue..

Sorry, accidentally close this issue..
Llewellyn commented 2020-08-16 23:05:50 +00:00
Copy Link

Wow very detailed explanation!

Well from my point of view that ids starting point does not play a roll at all. I could explain how the import ID remapping works to show you that the IDs should not play a roll. But if the fields used in the subforms from PC A in the first place are not exported this could be a problem.

I remember telling you where to jump into the export and see the actual data-set before JCB locks the data. This will help you see if what fields are exported from PC A in the first place. Just as a reminder:

On /models/joomla_components.php line 1077 replace the comment with:

echo '<pre>';
var_dump($this->smartBox);
jexit();

Then look at the fields array for the ID's from your subform fields, are they being exported? Should this be the point of failure that we need not look at the importer remapping the ID's. Then this will main we have a miss on the export.

Let see what this turns up.

Wow very detailed explanation! Well from my point of view that ids starting point does not play a roll at all. I could explain how the import ID remapping works to show you that the IDs should not play a roll. But if the fields used in the subforms from PC A in the first place are not exported this could be a problem. I remember telling you where to jump into the export and see the actual data-set before JCB locks the data. This will help you see if what fields are exported from PC A in the first place. Just as a reminder: On **/models/joomla_components.php** [line 1077](https://github.com/vdm-io/Joomla-Component-Builder/blob/c32baff933a084ac08dda77f26385847f84b8216/admin/models/joomla_components.php#L1077) replace the comment with: ```php echo '<pre>'; var_dump($this->smartBox); jexit(); ``` Then look at the fields array for the ID's from your subform fields, are they being exported? Should this be the point of failure that we need not look at the importer remapping the ID's. Then this will main we have a miss on the export. Let see what this turns up.
duetnet commented 2020-08-17 07:45:28 +00:00 (Migrated from github.com)
Copy Link

Hi Llewellyn

I have done the var_dump on PC A as explained by you, herewith my findings..with reference to the spread sheet I sent on Saturday..

The var_dump does not list all the fields, only lists 126 fields (out of my total of 312 fields) , from ID 1052 to 1177 and then shows message ‘ more elements...' and then continues with ‘component_custom_Admin_views’

So in the range listed, only subform field 1140 and its subfields appear correctly as per my spreadsheet, but this is only the export. I attach that section of the dump.

I also include the export of the actual Admin view in which subform field 1140 is used, but again, var_dump seems to cut off the ‘addfields’ row so I do not see the field actually being used….

I know you mention that you do not use IDs for remapping, but I must confess, everything in this extract is linked to an original extracted ID so I am fascinated as to how you do the remapping..

Tried to read through import process and I know it happens somewhere in import_joomla_components @ line 745, or 1619 and does appear to do an id remapping here, just not sure where decipher to see how it treats subforms and field relations on import…

Hope this helps

Regards

Franz

Hi Llewellyn I have done the var_dump on PC A as explained by you, herewith my findings..with reference to the spread sheet I sent on Saturday.. The var_dump does not list all the fields, only lists 126 fields (out of my total of 312 fields) , from ID 1052 to 1177 and then shows message ‘ more elements...' and then continues with ‘component_custom_Admin_views’ So in the range listed, only subform field 1140 and its subfields appear correctly as per my spreadsheet, but this is only the export. I attach that section of the dump. I also include the export of the actual Admin view in which subform field 1140 is used, but again, var_dump seems to cut off the ‘addfields’ row so I do not see the field actually being used…. I know you mention that you do not use IDs for remapping, but I must confess, everything in this extract is linked to an original extracted ID so I am fascinated as to how you do the remapping.. Tried to read through import process and I know it happens somewhere in import_joomla_components @ line 745, or 1619 and does appear to do an id remapping here, just not sure where decipher to see how it treats subforms and field relations on import… Hope this helps Regards Franz
Llewellyn commented 2020-08-18 03:30:42 +00:00
Copy Link

The note on the browser saying more elements... means that your debugger, or is limiting the length of a string being printed out. One way around this is to instead loop over the fields array and print one field ID at a time instead, just so you can see all the values. the more elements... means there are more, they are just not being printed.

Okay lets explain in simple terms how the remapped works.

There is an array in the import class that keeps track of all ID's called $this->newID you can see this array in action in the setNewID(here) function, but it is populated all over the architecture when ever a new ID's is found. The reality is the order of the tables being updated is part of the architecture that makes this work. Then there is the way the JCB finds the fields, or any items for that matter in the the new system, you can see this search in the connection of three functions saveSmartItems(here), getLocalItem(here), and addLocalItem(here). Then we also have a delayed remapping where we can't know values head of time, and need to work back-wards as it where to in affect first create the items, and then go back after the fact and update values in various columns and rows as needed. This is done with the updateAfterAll(here) function.

So searching for the $this->newID in the import class will take you to various moments in the execution where ID's are mapped. Really simple actually, based on table names and ideal moments this $this->newID is populated.

Next we of course need to look at the updateSubformsIDs(here) and updateSubformIDs(here) which again is called from many areas/moments during the import. Specially in the prepItem(here) function which is always trigger before any item is found/updated, or created.

Then there is this value called $diverged which is used when data diverged, so you may ask, when are values seen as diverged, well good question. JCB moved data from table to other tables over the years, and this diverged method is just to move component packages from before this move, to the new locations. This works very well actually since I have a few very old packages I still move around when needed.

Then at last we have the ID re-mapper setNewID(here) that actually remaps all ID's for all fields and field types based on table, and target field, this function is also called from all over the place as needed.

This should give you enough to understand. Let me know about the full list of exported ID's.

The note on the browser saying `more elements...` means that your debugger, or is limiting the length of a string being printed out. One way around this is to instead loop over the fields array and print one field ID at a time instead, just so you can see all the values. the `more elements...` means there are more, they are just not being printed. > Okay lets explain in simple terms how the remapped works. There is an array in the [import class ](https://github.com/vdm-io/Joomla-Component-Builder/blob/staging/admin/models/import_joomla_components.php) that keeps track of all ID's called `$this->newID` you can see this array in action in the `setNewID`([here](https://github.com/vdm-io/Joomla-Component-Builder/blob/staging/admin/models/import_joomla_components.php#L1397)) function, but it is populated all over the architecture when ever a new ID's is found. The reality is the order of the tables being updated is part of the architecture that makes this work. Then there is the way the JCB finds the fields, or any items for that matter in the the new system, you can see this search in the connection of three functions `saveSmartItems`([here](https://github.com/vdm-io/Joomla-Component-Builder/blob/staging/admin/models/import_joomla_components.php#L754)), `getLocalItem`([here](https://github.com/vdm-io/Joomla-Component-Builder/blob/staging/admin/models/import_joomla_components.php#L2666)), and `addLocalItem`([here](https://github.com/vdm-io/Joomla-Component-Builder/blob/staging/admin/models/import_joomla_components.php#L2635)). Then we also have a delayed remapping where we can't know values head of time, and need to work back-wards as it where to in affect first create the items, and then go back after the fact and update values in various columns and rows as needed. This is done with the `updateAfterAll`([here](https://github.com/vdm-io/Joomla-Component-Builder/blob/staging/admin/models/import_joomla_components.php#L1063)) function. So searching for the `$this->newID` in the [import class ](https://github.com/vdm-io/Joomla-Component-Builder/blob/staging/admin/models/import_joomla_components.php) will take you to various moments in the execution where ID's are mapped. Really simple actually, based on table names and ideal moments this `$this->newID` is populated. Next we of course need to look at the `updateSubformsIDs`([here](https://github.com/vdm-io/Joomla-Component-Builder/blob/staging/admin/models/import_joomla_components.php#L1331)) and `updateSubformIDs`([here](https://github.com/vdm-io/Joomla-Component-Builder/blob/staging/admin/models/import_joomla_components.php#L1357)) which again is called from many areas/moments during the import. Specially in the `prepItem`([here](https://github.com/vdm-io/Joomla-Component-Builder/blob/staging/admin/models/import_joomla_components.php#L1561)) function which is always trigger before any item is found/updated, or created. Then there is this value called `$diverged` which is used when data diverged, so you may ask, when are values seen as diverged, well good question. JCB moved data from table to other tables over the years, and this diverged method is just to move component packages from before this move, to the new locations. This works very well actually since I have a few very old packages I still move around when needed. Then at last we have the ID re-mapper `setNewID`([here](https://github.com/vdm-io/Joomla-Component-Builder/blob/staging/admin/models/import_joomla_components.php#L1397)) that actually remaps all ID's for all fields and field types based on table, and target field, this function is also called from all over the place as needed. This should give you enough to understand. Let me know about the full list of exported ID's.
duetnet commented 2020-08-18 06:03:29 +00:00 (Migrated from github.com)
Copy Link

Thanks for detailed explanation. I will debug imporer accordingly to see if I can find where my problem occurs.FranzSent from my Samsung device

-------- Original message --------
From: Llewellynvdm notifications@github.com
Date: 2020/08/18 05:30 (GMT+02:00)
To: vdm-io/Joomla-Component-Builder Joomla-Component-Builder@noreply.github.com
Cc: Franz hfdewet@mweb.co.za, State change state_change@noreply.github.com
Subject: Re: [vdm-io/Joomla-Component-Builder] Problem with Export/Import of
  components Field mapping (#587)

The note on the browser saying more elements... means that your debugger, or is limiting the length of a string being printed out. One way around this is to instead loop over the fields array and print one field ID at a time instead, just so you can see all the values. the more elements... means there are more, they are just not being printed.

Okay lets explain in simple terms how the remapped works.

There is an array in the import class that keeps track of all ID's called $this->newID you can see this array in action in the setNewID(here) function, but it is populated all over the architecture when ever a new ID's is found. The reality is the order of the tables being updated is part of the architecture that makes this work. Then there is the way the JCB finds the fields, or any items for that matter in the the new system, you can see this search in the connection of three functions saveSmartItems(here), getLocalItem(here), and addLocalItem(here). Then we also have a delayed remapping where we can't know values head of time, and need to work back-wards as it where to in affect first create the items, and then go back after the fact and update values in various columns and rows as needed. This is done with the updateAfterAll(here) function.
So searching for the $this->newID in the import class will take you to various moments in the execution where ID's are mapped. Really simple actually, based on table names and ideal moments this $this->newID is populated.
Next we of course need to look at the updateSubformsIDs(here) and updateSubformIDs(here) which again is called from many areas/moments during the import. Specially in the prepItem(here) function which is always trigger before any item is found/updated, or created.
Then there is this value called $diverged which is used when data diverged, so you may ask, when are values seen as diverged, well good question. JCB moved data from table to other tables over the years, and this diverged method is just to move component packages from before this move, to the new locations. This works very well actually since I have a few very old packages I still move around when needed.
Then at last we have the ID re-mapper setNewID(here) that actually remaps all ID's for all fields and field types based on table, and target field, this function is also called from all over the place as needed.
This should give you enough to understand. Let me know about the full list of exported ID's.

—You are receiving this because you modified the open/close state.Reply to this email directly, view it on GitHub, or unsubscribe.
[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/vdm-io/Joomla-Component-Builder/issues/587#issuecomment-675228429",
"url": "https://github.com/vdm-io/Joomla-Component-Builder/issues/587#issuecomment-675228429",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]

Thanks for detailed explanation. I will debug imporer accordingly to see if I can find where my problem occurs.FranzSent from my Samsung device -------- Original message -------- From: Llewellynvdm <notifications@github.com> Date: 2020/08/18 05:30 (GMT+02:00) To: vdm-io/Joomla-Component-Builder <Joomla-Component-Builder@noreply.github.com> Cc: Franz <hfdewet@mweb.co.za>, State change <state_change@noreply.github.com> Subject: Re: [vdm-io/Joomla-Component-Builder] Problem with Export/Import of &nbsp; components Field mapping (#587) The note on the browser saying more elements... means that your debugger, or is limiting the length of a string being printed out. One way around this is to instead loop over the fields array and print one field ID at a time instead, just so you can see all the values. the more elements... means there are more, they are just not being printed. Okay lets explain in simple terms how the remapped works. There is an array in the import class that keeps track of all ID's called $this->newID you can see this array in action in the setNewID(here) function, but it is populated all over the architecture when ever a new ID's is found. The reality is the order of the tables being updated is part of the architecture that makes this work. Then there is the way the JCB finds the fields, or any items for that matter in the the new system, you can see this search in the connection of three functions saveSmartItems(here), getLocalItem(here), and addLocalItem(here). Then we also have a delayed remapping where we can't know values head of time, and need to work back-wards as it where to in affect first create the items, and then go back after the fact and update values in various columns and rows as needed. This is done with the updateAfterAll(here) function. So searching for the $this->newID in the import class will take you to various moments in the execution where ID's are mapped. Really simple actually, based on table names and ideal moments this $this->newID is populated. Next we of course need to look at the updateSubformsIDs(here) and updateSubformIDs(here) which again is called from many areas/moments during the import. Specially in the prepItem(here) function which is always trigger before any item is found/updated, or created. Then there is this value called $diverged which is used when data diverged, so you may ask, when are values seen as diverged, well good question. JCB moved data from table to other tables over the years, and this diverged method is just to move component packages from before this move, to the new locations. This works very well actually since I have a few very old packages I still move around when needed. Then at last we have the ID re-mapper setNewID(here) that actually remaps all ID's for all fields and field types based on table, and target field, this function is also called from all over the place as needed. This should give you enough to understand. Let me know about the full list of exported ID's. —You are receiving this because you modified the open/close state.Reply to this email directly, view it on GitHub, or unsubscribe. [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/vdm-io/Joomla-Component-Builder/issues/587#issuecomment-675228429", "url": "https://github.com/vdm-io/Joomla-Component-Builder/issues/587#issuecomment-675228429", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]
duetnet commented 2020-08-30 08:56:13 +00:00 (Migrated from github.com)
Copy Link

I have just upgraded to JCB V2.11.4
Dont know what you may have changed, but I ran an export from V2.11.2, ran an import into V2.11.4 and mapping worked perfectly!!!
Tried to debug on the older version, but just kept getting lost in the debugging and gave up as it had taken up too much of my time already...was quicker to manually remap 👍

But, thanks anyway for your all your feedback, it did clarify things for me and was very much appreciated..

I have just upgraded to JCB V2.11.4 Dont know what you may have changed, but I ran an export from V2.11.2, ran an import into V2.11.4 and mapping worked perfectly!!! Tried to debug on the older version, but just kept getting lost in the debugging and gave up as it had taken up too much of my time already...was quicker to manually remap 👍 But, thanks anyway for your all your feedback, it did clarify things for me and was very much appreciated..
duetnet commented 2020-09-17 11:18:37 +00:00 (Migrated from github.com)
Copy Link

Hi Llewellyn

Sorry to reopen this issue, I experienced the problem again and I think I may now have a handle on what is happening.

First clue....

  • I moved the component to a new dev environment twice without a problem, but the 3rd time the subform mapping went askew again.
  • I realised I had merely added a new field as a component config parameter.
  • So on the export system, the new field was listed at the end of the existing field numbers, i.e. field ID# 1378 out of 331 fields (my component fields ran from 1049 to 1378).
  • On the import to the new system this field ID had moved to the top of the list as the last of 3 config fields , i.e. ID# 1051, every other field in the list above this field was moved up 1 ID number, eg. 1052 now became 1053 and 1377 became 1378.
  • So, every field number listed in each subform was not updated and still retained the mapping of the export system.

Second clue..

Although I am not sure how the admin view sequence is exported and imported, this may also be an issue.
I have noticed that if I change the sequence of admin views in my exporting component's config, then the import sequence in the imported admin view list changes, however this sequence of the ID#s in import system is not necessarily the sequence in which they appear in the component config, but it has changed. (Simple example, I add a new admin view, placed it 3rd in the list of admin views, or I move an admin view up or down in the list of component admin views)

Nevertheless, when I now look at the sequence in which fields are imported, it appears that they are imported in the following sequence

  • First - component config fields
  • then - per admin view
    • by admin view id# sequence (i.e. the id# sequence allocated in the importing site)
    • by field id# of fields configured for that admin view (so if a field is used in multiple admin views, it appears in the list together with fields the first admin view where it is used - again I am not sure of the field sequence within the admin view as I cannot yet see the pattern)

If I change the admin view sequence, this appears to impact the subform mapping....

Approach used to prove this...

  • I had created a clean (System A) joomla installation with only akeeba and jcb installed, and the field updater component.
  • Created an akeeba backup A of the system A
  • Imported my development component, exported the field list to excel and checked all field sequences were correct. This component uses 10 different subforms in various admin views.
  • Added a field "TESTPARAM" and added it to the component config and reexported to see new parameter a the end of the excel list.
  • Exported the JCB component (Export A)
  • Set up a new separate joomla installation (System B), and restored backup A to this system.
  • Imported JCB component Export A into System B
  • Extracted the field list into excel and compared with System A extract.
  • TESTPARAM had moved to the top of the list to appear as the last config parameter. All other fields had moved up 1 ID#. All subform fields were out by 1 ID#
  • Example Subform field in System A shows field sequence 1156,1157,1155, the same sequence is shown in System B, but it should now read 1157, 1158, 1156.
  • I have to check and then manually update each subform to reflect the correct field ID#s

I use the same approach when changing admin view sequence in the component and sometimes this affects the subform but not every time and I have not yet established a pattern....will try to pin it down.

Hi Llewellyn Sorry to reopen this issue, I experienced the problem again and I think I may now have a handle on what is happening. ### First clue.... - I moved the component to a new dev environment twice without a problem, but the 3rd time the subform mapping went askew again. - I realised I had merely added a new field as a component config parameter. - So on the export system, the new field was listed at the end of the existing field numbers, i.e. field ID# 1378 out of 331 fields (my component fields ran from 1049 to 1378). - On the import to the new system this field ID had moved to the top of the list as the last of 3 config fields , i.e. ID# 1051, every other field in the list above this field was moved up 1 ID number, eg. 1052 now became 1053 and 1377 became 1378. - So, every field number listed in each subform was not updated and still retained the mapping of the export system. ### Second clue.. Although I am not sure how the admin view sequence is exported and imported, this may also be an issue. I have noticed that if I change the sequence of admin views in my exporting component's config, then the import sequence in the imported admin view list changes, however this sequence of the ID#s in import system is not necessarily the sequence in which they appear in the component config, but it has changed. (Simple example, I add a new admin view, placed it 3rd in the list of admin views, or I move an admin view up or down in the list of component admin views) Nevertheless, when I now look at the sequence in which fields are imported, it appears that they are imported in the following sequence - First - component config fields - then - per admin view - - by admin view id# sequence (i.e. the id# sequence allocated in the importing site) - - by field id# of fields configured for that admin view (so if a field is used in multiple admin views, it appears in the list together with fields the first admin view where it is used - again I am not sure of the field sequence within the admin view as I cannot yet see the pattern) If I change the admin view sequence, this appears to impact the subform mapping.... ### Approach used to prove this... - I had created a clean (System A) joomla installation with only akeeba and jcb installed, and the field updater component. - Created an akeeba backup A of the system A - Imported my development component, exported the field list to excel and checked all field sequences were correct. This component uses 10 different subforms in various admin views. - Added a field "TESTPARAM" and added it to the component config and reexported to see new parameter a the end of the excel list. - Exported the JCB component (Export A) - Set up a new separate joomla installation (System B), and restored backup A to this system. - Imported JCB component Export A into System B - Extracted the field list into excel and compared with System A extract. - TESTPARAM had moved to the top of the list to appear as the last config parameter. All other fields had moved up 1 ID#. All subform fields were out by 1 ID# - Example Subform field in System A shows field sequence 1156,1157,1155, the same sequence is shown in System B, but it should now read 1157, 1158, 1156. - I have to check and then manually update each subform to reflect the correct field ID#s I use the same approach when changing admin view sequence in the component and sometimes this affects the subform but not every time and I have not yet established a pattern....will try to pin it down.
Llewellyn commented 2021-10-11 23:26:17 +00:00
Copy Link

After listening on high speed the last JUG... I realize I have not touched on this issue again in a long time, besides that I have in fact implemented the GUID import option in v2.12.11 for all tables that has GUID field.

Would be interesting to see if this improvement to this area has helped.

After listening on high speed the last JUG... I realize I have not touched on this issue again in a long time, besides that I have [in fact implemented](https://github.com/vdm-io/Joomla-Component-Builder/blob/staging/admin/models/import_joomla_components.php#L2680) the **GUID** import option in v2.12.11 for all _tables_ that has GUID field. Would be interesting to see if this improvement to this area has helped.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
Beta

Issues related to the a beta release

  
Bug

   
Documentation

   
enhancement

   
help wanted

   
invalid

   
joomla

Joomla related changes

  
question

   
URGENT

   
wontfix
No Label
Beta
Bug
Documentation
enhancement
help wanted
invalid
joomla
question
URGENT
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: joomla/Component-Builder#587
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?