Tanti Technology

My photo
Bangalore, karnataka, India
Multi-platform UNIX systems consultant and administrator in mutualized and virtualized environments I have 4.5+ years experience in AIX system Administration field. This site will be helpful for system administrator in their day to day activities.Your comments on posts are welcome.This blog is all about IBM AIX Unix flavour. This blog will be used by System admins who will be using AIX in their work life. It can also be used for those newbies who want to get certifications in AIX Administration. This blog will be updated frequently to help the system admins and other new learners. DISCLAIMER: Please note that blog owner takes no responsibility of any kind for any type of data loss or damage by trying any of the command/method mentioned in this blog. You may use the commands/method/scripts on your own responsibility. If you find something useful, a comment would be appreciated to let other viewers also know that the solution/method work(ed) for you.

Wednesday, 11 December 2013

Migrating OS in AIX 5.2, 5.3 and 6.1

This document covers migration preparation information relating to AIX 5.2, 5.3, and 6.1. 

  • Information regarding version 5 and version 6 installation
  • Recommendations directly prior to migration
  • Making sure your system is ready to migrate
  • Walk-thru of the Base System Install menus for version 5
  • Walk-thru of the Base System Install menus for version 6
  • What migration will do to your system
  • Post migration checks
  • FAQ

Information regarding version 5 and version 6 installation

  • Version 5 incorporates electronic license acceptance. At the first reboot, after an install, you will be prompted to view/accept your licenses before you cancontinue to use your system.
  • In AIX Version 6.1, a separate Software Maintenance Agreement (SWMA) acceptance window displays during installation immediately after the license acceptance window. The response to the SWMA acceptance (accept or decline) is stored on the system, and either response allows the installation to proceed, unlike license acceptance which requires an accept to proceed.
  • NIM masters/servers which are to serve version 5 or version 6 resources should be upgraded first. A NIM master/server must be at the same level or later than the software in any resources being served
  • Any migration will require more space. If you have no free partitions in the root volume group, or all file systems are near full, it would be a good idea to add another disk to the rootvg. Alternatively you can install a mksysb of the system to a larger disk before running the migration. Generally, if you have 600 MB or more available, that should be sufficient for the majority of migrations within 5.x.
  • AIX Version 6.1 requires a minimum total of 2.2 GB of physical disk space for the same set of installed filesets due to increased library sizes and additional function.
  • Be sure any additional applications you are currently running are supported in the version of AIX you are migrating to, and if they need to be upgraded before, or after the migration. Obtain the latest versions required for running at your new level.
For more complete information on your release, it is highly recommended you review the Release Notes.

5.2 Release Notes :-http://publib16.boulder.ibm.com/pseries/en_US/infocenter/base/52relnotes.htm 
5.3 Release Notes :- http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.resources/53relnotes.htm 
6.1 Release Notes :- http://publib.boulder.ibm.com/infocenter/systems/scope/aix/index.jsp?topic=/com.ibm.aix.resources/RELNOTES/610_base_relnotes.htm 

 NOTE: -The latest version of the media should be used to do the migration. The latest version will always be shipped when you order the media. If you have an older version of the media and would like to obtain the latest version, you can order it at the following web site: 
https://www-05.ibm.com/servers/eserver/ess/OpenServlet.wss  

 If assistance is required registering for the site, call Software Delivery (1-800-879-2755 opt2 then opt2 again for the U.S.). They will requirethe machine model and serial number of a machine licensed to run theAIX version you are ordering. Outside the U.S., contact your local support center.

Recommendations directly prior to migration


There are things that you can do to speed your migration along and avoid potential problems. While some may and some may not be official requirements to migrate your system, they are recommended and can prevent situations that may cause the system to hang and a restore to be necessary.


Break mirroring on the root volume group before the migration begins. To determine if the system is mirrored, run:

# lsvg -l rootvg
LV NAMETYPELPsPPsPVsLV STATEMOUNT POINT
hd5boot242closed/syncdn/a
hd6paging961922open/syncdn/a
hd8jfslog122open/syncdn/a
hd4jfs362open/syncd/
hd2jfs1563122open/syncd/usr
hd9varjfs362open/syncd/var
hd3jfs38762open/syncd/tmp
hd1jfs122open/syncd/home
hd10optjfs11222open/syncd/opt

*NOTE: If the pp's are twice the number of lp's, then that lv is mirrored.  

Update the system FW to the latest level available on the download site.

Firmware can be found here,  i have updated a separate procedure in blog for firmware update/upgrade
http://www14.software.ibm.com/webapp/set2/firmware/gjsn   


You can check the firmware level on CHRP architecture systems at AIX 5.1 and above with the ‘lsmcode’ command.
If you have older microcode, the system can fail to boot from the media, or could hang on reboot after the migration. If your FW level does not support the level of AIX, you could find that attempting to boot from the CD could corrupt your nvram, which could prevent the system from booting until the nvram battery has been drained. It is highly recommended to be at the latest FW before any migration.


To reduce the time needed to boot the system into maintenance mode you may wish to disconnect any external storage, if possible. The non-root volume groups can be imported and varied on after the migration 


Backup the system

Create a mksysb or sysback backup of the system so that you can restore to the previous AIX level if there were to be a problem with the migration. If you have an available drive, you may also want to consider doing an alt_disk_install clone of rootvg as an additional precaution. Refer to the AIX documentation on mksysb and alt_disk_install for your version if you need assistance.

If upgrading from AIX 5.1 or 5.2 use the
http://publib16.boulder.ibm.com/pseries/en_US/infocenter/base/ 

If upgrading from AIX 5.3 use the
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp 

Making sure your system is ready to migrate 


Make sure your system does not have existing installation issues before migrating. There is a pre_migration script that can be run from the Install CD to check out the system. All output from the pre_migration script is saved in the /home/pre_migration. directory. To run the pre_migration script, mount the Install CD:

# mount -v cdrfs -o ro /dev/cd0 /mnt 


Either copy the script to your system or run it from the CD. The location of the script is /mnt/usr/lpp/bos/pre_migration. The pre_migration script can be run multiple times. If you have questions on the output of the pre_migration script, check with your IBM Support Center. Here are some of the checks the script runs


  1. System platform check.
    Only CHRP systems are supported in AIX 5.2 and above. Run the following command on lower level systems to check the platform type.
    # lscfg | grep Architecture
  2. Required hardware if migrating to AIX 6.1
    Only 64-bit Common Hardware Reference Platform (CHRP) machines running selected PowerPC 970, POWER4, POWER5, and POWER6 processors that implement the POWER architecture Platform Requirements (PAPR) are supported.
    To see if you have a supported machine, log into the machine as the root user, and run the following command:
    # prtconf | grep 'Processor Type'
     
  3. Verify the major number for /dev/rootvg and /dev/hd5 is 10.
     
    # ls -al /dev/rootvg
    The output should be similar to:
    crw-rw----   1 root     system       10,  0 Jan 29 09:44 /dev/rootvg
     
    # ls -al /dev/hd5
    The output should be similar to:
    brw-rw----   1 root     system       10,  1 Feb 22 16:00 /dev/hd5
     
  4. *If the major number is not 10 on your system, contact your support personnel for assistance.
  5. Check boot logical volume size.
    The hd5 logical volume holds your boot image. There are a few checks to make against the boot lv before starting your update_all process. First, find out which disk contains your hd5 logical volume.
     
    # lslv -m hd5
    hd5:N/A
    LP   PP1  PV1
    0001 0001 hdisk0
     
  6. The listing under the “PV1” column indicates which disk hd5 is located. Any other entries under “PV2”...etc, simply represent mirrored copies.
    Next, verify your boot image can be successfully recreated using the hdisk# from above and by using /dev/ipldevice.
    # bosboot -ad /dev/hdisk0
    bosboot: Boot image is 35795 512 byte blocks.
    
    # bosboot -ad /dev/ipldevice
    bosboot: Boot image is 35795 512 byte blocks.

    If either of these two commands fail for any reason, please call the support center for resolution before proceeding.
    Finally, with the continual introduction of new devices and hardware the boot images are growing larger. You will want to make sure your hd5 logical volume has enough free and contiguous space to hold the boot image. Currently the recommended allocated space to have for hd5 is 32MB. This is of concern in environments with smaller disk sizes, and should be checked before running an update.
    # lsvg -l rootvg |grep hd5
    hd5 boot 1 1 1 closed/syncd N/A
    # lsvg rootvg |grep SIZE
    VG STATE: active PP SIZE: 32 megabyte(s)

    This information tells us that hd5 is 1 partition, and that the partition size is 32meg. This should be enough to contain the boot image. Smaller partition sizes will require more partitions to be allocated.

    Your hd5 partitions also do need to be contiguous partitions. Check this by running the following command :
    # lspv -M hdisk0 |grep hd5
    hdisk0:1 hd5:1
    hdisk0:2 hd5:2
    hdisk0:3 hd5:3

    You can see in this example that the hd5 logical volume covers the first 3 partitions on the disk and they are all contiguous. If your partitions are not contiguous, or are not covered on the first partitions of the disk please call the software support center for assistance with correcting this.
    Again, you may only have 1 partition that is large enough to handle the boot image, or you may have multiple smaller partitions. Either is fine.
  7. Check if Kerberos is being used. This is particular to 5.2 and above, because the libraries needed for Kerberos are now shared, and need to be installed separately from the Expansion pack. The BOS install process will check if Kerberos is enabled, and will set the default for installing the Kerberos software bundle to yes, and prompt you for the Expansion Pack (if doing a CD install).
  8. Memory check Verify you have the minimum required memory for your target operating system level by running the following command :
     
    # bootinfo -r
    AIX 5.2 : 128mbAIX 5.3 : 256mbAIX 6.1 : 512m
  9. A list of system configuration files that will not be merged to the latest level, but will rather, be saved in /tmp/bos, will be created by the pre_migration script, and can be viewed in the Migration Confirmation Menu.
  10. The pre_migration script will generate a list of system configuration files being merged, and will save these files in /home/pre_migration./saved_configuration_files. 
  11. ‘lppchk’ commands should be run to check system consistency:
    # lppchk -v
    # lppchk -c
    # lppchk -f
    # lppchk -l

    (If executed, the pre_migration script will run these commands for you).
  12. TCB
    If Trusted Computing Base (TCB) is enabled, the ‘tcbck -n tree’ command is run and the output saved for comparison purposes after the migration.
  13. bos.net.ipsec.keymgt

    If bos.net.ipsec.keymgt is installed, a check is made for the /etc/ipsec/inet/DB directory, and if it does not exist, it is recommended you do a force install of bos.net.ipsec.keymgt at the same level as currently installed on the system. This fileset will fail to migrate successfully if this directory is missing.
  14. No migration option
    It has been reported that on some systems, migration is not offered as an installation option as expected. To help ensure migration will be an option, the system will verify that you are currently at a previous version or release of AIX. To use the same method the BOS installation code uses enter the following:
    # lslv -m hd5
    hd5:N/A
    LP   PP1  PV1
    0001 0001 hdisk0
    # /usr/lpp/bosinst/blvset -d /dev/hdisk0 -g level

    (The output from the command above needs to match your current version and release.)

    Another cause of not receiving a migration option could be due to the number of disks your system is presented during the bootup process. The system will only scan the first 50 disks it encounters before presenting the BOS menu selection screens. If it does not encounter your rootvg disk(s) during that scan your installation option may be set to "New and Complete Overwrite". If this is the case select the option, "55 More Disk Options". Taking this option will allow you to scan through all disks on the system. Once you've gone through all of the disks your system should recognize your rootvg drive(s). The option for "Migration" should be selected for you at this time.
  15. Required users and groups

    Verify that these user and groups are unchanged. The /etc/passwd file should have the following entries. It is also important that the User IDs (UIDs), which is the third colon-separated field, are not duplicated for other users.
    root:!:0:0::/:/bin/ksh
       daemon:!:1:1::/etc:
       bin:!:2:2::/bin:
       sys:!:3:3::/usr/sys:
       adm:!:4:4::/var/adm:
       uucp:!:5:5::/usr/lib/uucp:
       guest:!:100:100::/home/guest:
       nobody:!:4294967294:4294967294::/:
       lpd:!:9:4294967294::/:

    The /etc/group file should have these entries. It is acceptable to have more users added to a group (comma separated additions to the last field as seen with group "sys" in this example). As with users, the Group IDs (GID's), which is the third colon-separated field, should not be duplicated for other groups.
    system:!:0:root
       staff:!:1:
       bin:!:2:root,bin
       sys:!:3:root,bin,sys
       adm:!:4:bin,adm
       uucp:!:5:uucp
       mail:!:6:
       security:!:7:root
       cron:!:8:root
       printq:!:9:
       audit:!:10:root
       ecs:!:28:
       nobody:!:4294967294:nobody,lpd
       usr:!:100:guest
       perf:!:20:
       shutdown:!:21:

    (Additional users and groups are fine.)
  16. Scheduled Jobs

    Jobs that are scheduled during the migration time should be removed or rescheduled until after migration time. It is recommended that all at jobs be removed by saving off the /var/spool/cron/atjobs file, and then removing /var/spool/cron/atjobs. You can verify that no jobs are scheduled by running the following command:
    # at -l

    Save the following files if you have edited them:
    /usr/adm/cron/at.allow
       /usr/adm/cron/at.deny
       /usr/adm/cron/cron.allow
       /usr/adm/cron/cron.deny
  17. Logical volume mapping
    Verify that your system-created file systems and logical volumes in the rootvg are as follows:


    # lsvg -l rootvg

    LV NAMETYPELPsPPsPVsLV STATEMOUNT POINT
    hd5boot222closed/syncdn/a
    hd6paging96962open/syncdn/a
    hd8jfslog112open/syncdn/a
    hd4jfs332open/syncd/
    hd2jfs1561562open/syncd/usr
    hd9varjfs332open/syncd/var
    hd3jfs38382open/syncd/tmp
    hd1jfs112open/syncd/home
    hd10optjfs11112open/syncd/opt



    What we are verifying here is that your "LV NAME" entries and "MOUNT POINT" entries match the entries listed above. A missing filesystem, or a filesystem not mounted over the proper logical volume could cause you to not receive a migration option.

    *NOTE: The numbers for LPs, PPs and PVs will vary and are of no importance to this check.
  18. Location code verification
    You may want to record the location codes of your rootvg disks before the migration, as sometimes in maintenance mode the hdisk numbers can be different than what they are in normal mode. The location code will always remain the same.
    # lsdev -Cc disk

Walk-thru of the Base System Install menus for version 5

Welcome to Base Operating System
Installation and Maintenance
Type the number of your choice and press Enter.  Choice is indicated by >>>.
>>> 1 Start Install Now with Default Settings
 2 Change/Show Installation Settings and Install
 3 Start Maintenance Mode for System Recovery
 88  Help ?
 99  Previous Menu
>>> Choice [1]: 2

It is recommended you always choose Option 2 to confirm the type of install that will occur.
Installation and Settings
Either type 0 and press Enter to install with current settings, or type the number of the setting you want to change and press Enter.
 1  System Settings:
      Method of Installation.............Migration              
      Disk Where You Want to Install.....hdisk0
 2  Primary Language Environment Settings (AFTER Install):
      Cultural Convention................English (United States)
      Language ..........................English (United States)
      Keyboard ..........................English (United States)
      Keyboard Type......................Default
 3  More Options  (Desktop, Security, Kernel, Software, ...)
>>> 0  Install with the current settings listed above.
                    +-----------------------------------------------------
 88  Help ?         |    WARNING: Base Operating System Installation will
 99  Previous Menu  |    destroy or impair recovery of SOME data on the
                    |    destination disk hdisk0.
>>> Choice [0]:

From here, you confirm that the method of installation is migration, and the proper disks are selected. It is possible that disks could get renumbered during the boot into Maintenance Mode. You can confirm that the correct disk is selected using the location code information obtained from item #16 listed in the checks above.

If everything looks correct, you will select 0 to continue.


You can look at option number 3, 'More options', if you want to review these choices.
Install Options
1.  Enable Trusted Computing Base.................................... No
2.  Import User Volume Groups........................................ Yes
3.  Enable System Backups to install any system...................... Yes
  (Installs all devices and kernels)
4.  Remove Java 1.1.8 Software....................................... No

>>> 0  Install with the current settings listed above.
 88  Help ?
 99  Previous Menu
>>> Choice [0]:

Set TCB to Yes, if it is enabled on the system before the migration.

You now maintain the desktop previously on the system and migrate it to the latest level. Since file systems are not recreated during a migration, the JFS2 and 64-bit kernel selections have been removed in this case, and the code determines which kernel to link based on the system type, and if JFS or JFS2 file systems are in use.

Walk-thru of the Base System Install menus for version 6

Type the number of your choice and press Enter.  Choice is indicated by >>>.
>>> 1 Start Install Now with Default Settings
 2 Change/Show Installation Settings and Install
 3 Start Maintenance Mode for System Recovery
 4 Configure Network Disks (iSCSI)
 88  Help ?
 99  Previous Menu
>>> Choice [1]:

With AIX6, you can install the Base Operating System to an iSCSI disk.
To configure an iSCSI disk for Base Operating System use, you must supply several parameters before beginning the installation. Gather the following parameters:

Adapter Name
Name of network adapter used for iSCSI. For iSCSI TOE adapters, this field is formatted ics#, where # is a number. For the iSCSI SW Initiator, this field is the Ethernet interface name and is formatted en#, where # is a number.
IP Address of Adapter
IP address that is assigned to the adapter specified by Adapter Name.
IP Address of Gateway
IP address of the gateway that is used by the adapter specified by Adapter Name.
Subnet Mask
Subnet mask that is assigned to the adapter specified by Adapter Name.
iSCSI Target Name
Name that is configured for the iSCSI Target.
iSCSI Initiator Name
Initiator name that is configured for the iSCSI Target.
Port Number
Port Number that is configured for the iSCSI Target.
IP Address of Target
IP Address that is configured for the iSCSI Target.

*Note: Consult your iSCSI vendor's documentation for more information.


It is recommended you always choose Option 2 to confirm the type of install that will occur.
Installation and Settings
Either type 0 and press Enter to install with current settings, or type the number of the setting you want to change and press Enter.
 1  System Settings:
      Method of Installation.............Migration
      Disk Where You Want to Install.....hdisk0
 2  Primary Language Environment Settings (AFTER Install):
      Cultural Convention................  English (United States)
     Language ..........................  English (United States)
     Keyboard ..........................  English (United States)
 3  Security Model.......................Default
 4  More Options  (Software install options)
>>> 0  Install with the current settings listed above.
                    +-----------------------------------------------------
 88  Help ?         |    WARNING: Base Operating System Installation will
 99  Previous Menu  |    destroy or impair recovery of SOME data on the
                    |    destination disk hdisk0.
>>> Choice [0]:

If everything looks correct, you will select 0 to continue. You can look at option number 4, 'More options', if you want to review these choices.
Install Options
1.  Enable System Backups to install any system...................... Yes
 (Installs all devices)
2.  Import User Volume Groups........................................ Yes
3.  Remove Java 1.1.8 Software....................................... No

What migration will do to your system

If the boot logical volume (hd5) is too small, and there is not sufficient space to increase it, then the migration will halt with no changes to the system. If the boot logical volume is too small, but there is space to increase it, the migration will do so and continue. Remember that you must have free contiguous partitions for the boot logical volume to be created properly.

The Migration Confirmation Menu comes next. You will be offered choices to:
  1. See which base system configuration files will be stored in /tmp/bos and not automatically merged into the system.
  2. See which filesets will be removed from the system and will not be replaced with new filesets.
  3. See which directories will have the current contents removed during the migration installation.
An option to cancel the migration is also available from this Menu. If you are doing a non-prompted migration, this Menu will not be shown.

The contents of /tmp are removed to make room for migration processes and for rebuilding the boot image at the end of the migration. Migration first saves system configuration files and then restores the files from the bos image (bos.rte.* subsystems or filesets).

After the files are restored, migration performs the operations on the system configuration files, whether it be merging the new data into the old file, using the new file and saving the old file or vice versa. The filesets listed for removal are then removed.

Afterwards, the remaining filesets on the system are migrated up to the latest level available on the media. Special file merge situations are handled after this, a new boot image is built and the system reboots.

Post migration checks

After the migration, you will want to check that your migration was successful. There will be a post_migration script in the directory /usr/lpp/bos. The post_migration script will provide data on the success of the migration. All output will be saved in /home/post_migration. directory.

Here are some of the commands in the post_migration script that you can run to determine the success of your migration.

* The lppchk -v command will check the consistency of the software. If some dependency is missing it will be reported here. If you do get any output, then run lppchk -m3 -v to get more details on the condition.

* Other flags can be used with lppchk, but the -v flag gives the most important post migration data. The other flags are:

lppchk -l
lppchk -f
lppchk -c

In all cases no output is best.

* If you migrated a TCB enabled system with TCB turned on, then run tcbck -n tree to check for TCB errors. If you ran this before the migration you can compare the output from before and after. If not, check your errors against some known conditions documented in the Release Notes for the release to which you just migrated. There are documented steps to clean up the error conditions.

* Check the output of the installs during the migration. The log of the migration is saved in /var/adm/ras/devinst.log. This log is concatenated to during the migration, so the initial output is from the previous install. Find where the installation of the version you migrated to begins, and search the summaries for failed or canceled installations. If you find one, search backwards in the log until you find the source of the error. Canceled installations usually are due to missing requisites not being on the media. If you migrated using CD media, continue searching the log for the fileset, as it may have successfully installed later in the migration process. If you migrated using NIM, there may have been filesets you needed that were missing in the lpp_source.

* Filesets that did not migrate can now be migrated using a software source containing the software and running a smitty update_all.

Read the migration sections in the Release Notes. There are special instructions for some situations that might occur after the migration, and methods to correct them.


FAQ


This FAQ will be updated with the most common questions we receive concerning migration upgrades. If you have any suggestions/questions feel free to submit them and we may add them in. This section is not intended for problem resolution questions - simply common short answer issues that hopefully will save you some time by not having to open a PMR for a quick question.
  1. How long will the migration take ?
    That all depends on the size of the rootvg. A conservative average to recommend blocking off time would be this :
    |normal reboot time| + |2-3 hours| + |normal reboot time|
    You may also want to take into account the possibility that you may have to recover from a system backup. You never know when the extreme (such as a power outage during the upgrade) can happen. It is much better to block off too much time and be finished early than to not reserve enough time for the upgrade.
  2. Will I need to reboot the system when the migration is complete ?
    No, the migration process will reboot on its own and set the boot device to the rootvg disk containing hd5. No interaction from you is required to do this.
  3. What about my non-rootvg volume groups ?
    Non-rootvg volume groups are not affected by the migration.
  4. My current rootvg filesystems are JFS filesystems but I’d like to change these to JFS2 during the migration. Is there an option for this ?
    At this time there is not a way to do this by any means (supported or unsupported).
  5. Will product X at level Y.Z work with my new operating system level ?
    Some products may only be certified up to a certain operating system level while other products may require an update. The best thing to do would be to contact product X's support center. If it is an IBM product feel free to contact our support center and open a PMR requesting to speak to the product X team. Any 3rd party products should be cleared by their support before upgrading.
  6. I have 8 volumes in my CD set. What are they for, and will I need them all ?
    You will likely not need all 8 volumes of media. The install process will prompt you for the volumes it requires as the installation proceeds. Below is a brief description of each volume :
    Volume 1

    Contains the minimal Base Operating System (BOS) software installed on every system, as well as all devices and both kernels, and the English message catalogs. To install a system with only "Volume 1" from the BOS menu, set Desktop to NONE, change the default for Graphics bundle to No, and change the default for System Management Client bundle to No.

    Volume 2

    Contains software to install the Graphics bundle, and to set the Desktop to CDE. System management software, X11 software and Java software are on this CD and on Volume 3.

    Volume 3

    Contains system management software, printer software, as well as additional CSM, RSCT, Fortran, and Java software.

    Volume 4

    Contains pieces of Open_GL, PEX_PHIGS, LDAP, and DB2® software that comes with AIX.

    Volume 5

    Contains software for installing AIX in other languages. Messages, help text, and locales for German, French, Italian and Japanese are included.
    Volume 6

    Contains software for installing AIX in other languages. Messages, help text, and locales for Catalan, Czech, Spanish, Hungarian, Polish, Brazilian Portuguese, Russian, and Slovakian are included.
    Volume 7

    Contains software for installing AIX in other languages. Messages, help text, and locales for Korean, Simplified Chinese, and Traditional Chinese are included.
    Volume 8

    Contains software to support bidirectional and complex text languages, including Arabic, Hebrew, Thai, Vietnamese, Hindi, Tamil, Telugu, Gujarati, Marathi, Kannada, Malayalam, Welch, Maltese, Azerbaijani-Latin, Urdu (India), and Urdu (Pakistan). Chinese language support for Hong Kong and Singapore is also included 

No comments:

Post a Comment