Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

Upgrade QB8->QB10 with extremely large database. #4492

duclong.ng ·

I am having a problem with upgrade from QB8 to QB10. I using mysql for QB service.
I used ".../qb10/bin/upgrade.sh .../qb8" to upgrade.
Everything was quite easy when I tested with a small amount of data. And it runs for 2 hours with data about 4GB. According to what I observed in the log, the data will be export and import again. so, the process is quite long time.

With the official service, we have more than 100GB of data. This process estimates that it takes too much time and is likely to have an error.
So is there any other way for us to do this process faster and safer?

  • Create more schools that are missing with SQL command?
  • replies 10
  • views 233
  • stars 0
robinshen ADMIN ·

I guess there are many many build records in the database. You may consider cleaning up obsolete records with build clean up policy to make the database smaller.

duclong.ng ·

Is that the only way? We have scheduled for the old records to be deleted automatically. And here is the remaining data.

duclong.ng ·
2023-05-08 18:19:46,021 INFO  >>> 2023-05-08 18:19:46,021 WARN  OutOfMemory error encountered. Reduce batch size and retry...
2023-05-08 18:19:46,023 INFO  >>> 2023-05-08 18:19:46,022 INFO  Loading table rows (342001->343000) from database...
2023-05-08 18:19:50,562 INFO  >>> 2023-05-08 18:19:50,561 WARN  OutOfMemory error encountered. Reduce batch size and retry...
2023-05-08 18:19:50,562 INFO  >>> 2023-05-08 18:19:50,562 INFO  Loading table rows (342001->342500) from database...
2023-05-08 18:19:51,564 INFO  >>> 2023-05-08 18:19:51,564 WARN  OutOfMemory error encountered. Reduce batch size and retry...
2023-05-08 18:19:51,564 INFO  >>> 2023-05-08 18:19:51,564 INFO  Loading table rows (342001->342250) from database...
2023-05-08 18:19:52,296 INFO  >>> 2023-05-08 18:19:52,296 WARN  OutOfMemory error encountered. Reduce batch size and retry...
2023-05-08 18:19:52,296 INFO  >>> 2023-05-08 18:19:52,296 INFO  Loading table rows (342001->342125) from database...
2023-05-08 18:19:52,349 INFO  >>> 2023-05-08 18:19:52,349 WARN  OutOfMemory error encountered. Reduce batch size and retry...
2023-05-08 18:19:52,349 INFO  >>> 2023-05-08 18:19:52,349 INFO  Loading table rows (342001->342062) from database...
2023-05-08 18:19:52,365 INFO  >>> 2023-05-08 18:19:52,365 INFO  Stopping QuickBuild server...
2023-05-08 18:19:52,385 INFO  >>> 2023-05-08 18:19:52,385 INFO  Stopping plugins...
2023-05-08 18:19:52,389 INFO  >>> 2023-05-08 18:19:52,389 INFO  Stopping task scheduler...
2023-05-08 18:19:52,389 INFO  >>> 2023-05-08 18:19:52,389 INFO  Closing session factory...
2023-05-08 18:19:52,478 INFO  >>> 2023-05-08 18:19:52,476 WARN  FAILED net.bull.javamelody.MonitoringFilter-1140e7a4: java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-05-08 18:19:52,478 INFO  >>>  java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-05-08 18:19:52,482 INFO  >>> 2023-05-08 18:19:52,481 WARN  FAILED org.eclipse.jetty.servlet.ServletHandler@b73433: java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-05-08 18:19:52,482 INFO  >>>  java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-05-08 18:19:52,482 INFO  >>> 2023-05-08 18:19:52,482 WARN  FAILED org.eclipse.jetty.server.session.SessionHandler@1bf14704: java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-05-08 18:19:52,482 INFO  >>>  java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-05-08 18:19:52,482 INFO  >>> 2023-05-08 18:19:52,482 WARN  FAILED o.e.j.s.ServletContextHandler@2be21396{/,null,UNAVAILABLE}: java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-05-08 18:19:52,482 INFO  >>>  java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-05-08 18:19:52,483 INFO  >>> 2023-05-08 18:19:52,483 WARN  FAILED org.eclipse.jetty.server.handler.HandlerCollection@124dac75[o.e.j.s.ServletContextHandler@2be21396{/,null,UNAVAILABLE}]: java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-05-08 18:19:52,483 INFO  >>>  java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-05-08 18:19:52,483 INFO  >>> 2023-05-08 18:19:52,483 WARN  FAILED org.eclipse.jetty.server.Server@315105f: java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-05-08 18:19:52,483 INFO  >>>  java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-05-08 18:19:52,483 ERROR  >>> Exception in thread "main" java.lang.reflect.InvocationTargetException
2023-05-08 18:19:52,484 ERROR  >>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-05-08 18:19:52,484 ERROR  >>>      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-05-08 18:19:52,484 ERROR  >>>      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-05-08 18:19:52,484 ERROR  >>>      at java.lang.reflect.Method.invoke(Method.java:498)
2023-05-08 18:19:52,484 ERROR  >>>      at com.pmease.quickbuild.bootstrap.Bootstrap.shutdown(Bootstrap.java:216)
2023-05-08 18:19:52,484 ERROR  >>>      at com.pmease.quickbuild.bootstrap.Bootstrap.boot(Bootstrap.java:659)
2023-05-08 18:19:52,484 ERROR  >>>      at com.pmease.quickbuild.bootstrap.Bootstrap.main(Bootstrap.java:169)
2023-05-08 18:19:52,484 ERROR  >>> Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
2023-05-08 18:19:52,484 ERROR  >>>      at com.pmease.quickbuild.bootstrap.BootstrapUtils.wrapAsUnchecked(BootstrapUtils.java:56)
2023-05-08 18:19:52,484 ERROR  >>>      at com.pmease.quickbuild.equinoxadapter.EquinoxAdapter.shutdown(EquinoxAdapter.java:284)
2023-05-08 18:19:52,484 ERROR  >>>      ... 7 more
2023-05-08 18:19:52,484 ERROR  >>> Caused by: java.lang.reflect.InvocationTargetException
2023-05-08 18:19:52,484 ERROR  >>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-05-08 18:19:52,484 ERROR  >>>      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2023-05-08 18:19:52,484 ERROR  >>>      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-05-08 18:19:52,484 ERROR  >>>      at java.lang.reflect.Method.invoke(Method.java:498)
2023-05-08 18:19:52,484 ERROR  >>>      at com.pmease.quickbuild.equinoxadapter.EquinoxAdapter.shutdown(EquinoxAdapter.java:282)
2023-05-08 18:19:52,484 ERROR  >>>      ... 7 more
2023-05-08 18:19:52,484 ERROR  >>> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Hello@robinshen,
I had a problem when Upgrade QB8 -> QB10 with a database only 13GB. Do you have any advice?

robinshen ADMIN ·

How much memory you've assigned to QB server? You may check it at conf/wrapper.conf for property wrapper.java.maxmemory.

duclong.ng ·

The server's memory is quite large, I think it is due to the limit placed in the conf/wrapper.conf file. I will try to change it.
Thanks.

duclong.ng ·

Hello@robinshen,
It seems that the upgrade process has successfully .
But the names of my users written in Korean have been replaced by "????". The same thing happens to configurations using the description in Korean.
Is this normal or abnormal?

robinshen ADMIN ·

This is abnormal. Seems that your database is not using UTF8 encoding. You may check names in the original backup (just unzip and open users.xml) to see if names are displayed correctly.

duclong.ng ·

Our data is stored in UTF8-BIN format. Does this cause errors?

duclong.ng ·

I see that the name displayed in the users.xml file is in accordance with the format.

duclong.ng ·

The problem did not recur when I tried to try again. Thanks!